[Question] How do you fine-tune LLaVA-NeXT on video data?

[DrVictorBenjamin]

Question

I have a collection of videos and annotations. How do I fine-tune one of the LLaVA-NeXT models? I see the instructions for how to do so with traditional LLaVA but the directions for LLaVA-NeXT with video data are unclear. Thank you very much

[DrVictorBenjamin]

Ay after spending some time digging around, I came across this tutorial in case anyone else is searching for an answer: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LLaVA-NeXT-Video/Fine_tune_LLaVa_NeXT_Video_with_HFTrainer.ipynb

I haven't tried it yet but I will

[anjaligupta1104]

Did you try it and have any success? I'm also curious about how this applies to LLaVA-Video and any documentation you found about the data format.

[DrVictorBenjamin]

I didn't try the guide yet, got distracted with another project. I may try it in the next week. If you get to it first, let me know!

[anjaligupta1104]

Sure thing!