Roadmap, future directions and organization

[JeanKossaifi]

Hi all!

As the project grows it might be handy to have a common platform to easily exchange and chat. Gitter isn't the best for that so I'm thinking of creating a Slack, if everybody is happy with that.

I also created https://github.com/tensorly/lab for quickly trying new things, for instance quickly iterating on the API of the object oriented interface @MarieRoald @yngvem or new optimization routines @cohenjer @caglayantuna -- this has the advantage of being pip-installable for quick experimentation on unstable functionalities, without messing up with the main package.

To get started on a roadmap, I think these are outstanding issues:

New features

• More decompositions:
  • BTD @juliagusak
  • Tensor-Ring @merajhashemi ✅ PR Tensor Ring implementation #229
  • CMTF @IsabellLehmann and @aarmey PR CMTF-ALS #293
• Generalize tensor regression to tensor outputs (Tensor regression with y of arbitrary order #51 )
• Add support for tensor classification (e.g. CP regression method: logistic & multinomial logistic regression #273 )
• (hand-optimized) gradient of some functions (e.g. CP reconstruction error) @aarmey and @JeanKossaifi
• Serialization of models @yngvem PR WIP: Serialisation lab#4
• Visualization submodule - rather than just examples, we could have some general visualization tools

CI (Continuous Integration)

• Refactor the tests so they are more reliable, some tend to occasionally fail (e.g. Issue with test_svd #287 Issue with tucker rank validation #288) - Tracked in Tests to fix #310
• Update Conda build and have it automatically done on release in Github worflows

Enhancements of existing functions

• Better support for missing values (e.g. Improving handling of missing values #198, @aarmey )

API Improvement

• More consistent tensor product support ( tensordot vs tensor_dot vs contract vs inner  #250 ) - PR Adding Batched Tensor Dot + Simplifying API #309
• Better SVD interface (eigsh for all backends #251, Bug when n_eigenvecs is None in truncated_svd #233, A clearer interface to SVD options #227)
• Refine the API:
  • Backend VS main functions (we may want to allow backends to define some higher level functions, e.g. Reduce memory allocation for partial_svd  #249)
  • Factorized tensors: OO interface, general support for batching (How to use the functions in batch-wise way? #199)
  • Tensor decomposition objects, better support/integration with transform and pipelines (e.g. Question: repeatable transform() step #172 ). I am iterating in tensorly-torch with a more feature rich version
• Sparse tensors: add support for more backends, should the sparse support be in a separate package (e.g. tensorly-sparse) or moved to a main submodule (e.g. tensorly.sparse)? @asmeurer
• Formalize the level of granularity we want between optimization functions, decompositions, etc (e.g. repeating ourselves vs function and class proliferation) (Optimization submodule #80 )

We can put together a roadmap and collect thoughts/feedback on the project organization and API!

[caglayantuna]

HI @JeanKossaifi , thanks for the detailed roadmap. I think, it is necessary to have such a list for Tensorly developers. Slack works for me. Maybe, we could also inform each other about what we are working on to avoid time wasting. Thanks!

[cohenjer]

Great !
Slack works for me as well.
This roadmap looks excellent :) One decomposition that we handle kind of on the side right now is tensor trains and tensor networks in general, but supporting these properly would require a significant amount of work.
Also about having higher-end backend functions, one function that is kind of ``in the middle'' is the MTTKRP. Several research papers focus on how to compute it efficiently, but I don't think it is (and will be soon) implemented in the back-ends. A python implementation of it would be difficult since state-of-the-art MTTKRPs heavily use fine-tuned memory allocation and such (I am no expert ^^).

[JeanKossaifi]

I completely agree with both -- tensor networks should definitely be on the list and I've been adding more routines to manipulate TT/TTM. Allowing for various implementations is the goal of the tenalg backend that I hope to extend to allow for more optimized implementations.

[MarieRoald]

Thanks for such an excellent overview, @JeanKossaifi! I agree with @caglayantuna and @cohenjer that this list is very useful, and the https://github.com/tensorly/lab is a good idea! Slack works for me as well.
I have a lot on my plate this summer, but I really look forward to working more on this when my schedule clears up :)

[aarmey]

This list is really useful @JeanKossaifi—there are many nice improvements here that I wasn't aware of. I really like the idea of a Slack. The Julia community one has been extremely helpful.

Would it be helpful to have the CMTF decomposition merged? I don't want to step on toes, but we're using this quite a bit, and having a tested reference implementation would be nice. I may be able to find someone in the lab to help with edits from the last review.

One longer-term thought... I know brute-optimization would add unwanted dependencies, but would it be helpful to think about adding common derivatives (e.g., the derivative of error between CP and a tensor)? This is another case of something we're already using and could allow a separate Tensorly package for brute optimization, or different line search routines, for example.

[aarmey]

@IsabellLehmann great! I should be able to add that to all backends. At least in my experience ALS has converged much faster than our implementation of CMTF-OPT. Your implementation looks pretty similar to what I have. However, we do have an implementation of CMTF-OPT here as the missing value pattern was showing up in the factors with ALS in one project.

@JeanKossaifi how would you advise we both access to a PR?

[yngvem]

@JeanKossaifi how would you advise we both access to a PR?

I would add @IsabellLehmann's fork as a remote, fetch her CMTF_ALS branch and add a pull request into her CMTF_ALS branch. When it's accepted, it will automatically be added to the PR into tensorly main. @JeanKossaifi do you have any other approach?

[JeanKossaifi]

This sounds great @yngvem @aarmey, I think it may indeed be the easiest to first finish this PR so we're all on the same page as some of the features are reused and we'd want to have the same API for all CMTF versions. Do we need to explicitly allow others to add to the PR?

[aarmey]

I think the three approaches are:

1. @JeanKossaifi can provide write permissions on Tensorly (optionally locking main and other branches for admins/maintainers only)
2. @IsabellLehmann can provide write permissions to her fork.
3. I can make a PR on @IsabellLehmann's fork.

I'll proceed with 3. It might help to think about the other options one day.

[IsabellLehmann]

Hi @aarmey. Thank you :) I have invited you as a collaborator of my fork.

[JeanKossaifi]

Hi all, I am slowly catching up with all the tasks!

I have created a Slack for the TensorLy project: Join here

Looking forward to seeing you there!

In the meantime, feel free to edit the top message or suggest anything else for the roadmap and happy summer!