Multivariate time-series

[moghadas76]

Univariate pre-training addresses different variate numbers of datasets during pre-training. A multivariate extension is crucial for the future improvement of Sundial, including:
Architecture: Recent works proposed new attention mechanisms (e.g., Moirai, Timer-XL) for intra-/inter-variate modeling, which can be seamlessly incorporated into Sundial. Multivariate extension on the flow-matching network (e.g., from MLP to iTransformer) is applicable to make the post-merging on univariate representations.
Post-training: Another roadmap is univariate pre-training and multivariate fine-tuning. Similar to GPT-3, a univariate pre-trained TSFM is a start-point model. We will explore multivariate prompting (e.g., special variate token) to instruct TSFMs on downstream tasks.

How to do that in your code?