Question about LLaVA-NeXT-Video fine-tuning example

[DrVictorBenjamin]

Hello,

I tried following your example shown here https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LLaVA-NeXT-Video/Fine_tune_LLaVa_NeXT_Video_with_HFTrainer.ipynb

Without changing a line of code, the tutorial currently emits the following error: ValueError: Video features and video tokens do not match: tokens: 1004, features 4608

I google'd around but didn't find if anyone has identified a clean solution for this yet. Wondering if you've had any luck

Thanks

[NielsRogge]

Pinging @zucchini-nlp here

[zucchini-nlp]

These scripts are very old, unfortunately I don't have much bandwidth to update them 😢 @DrVictorBenjamin are you using the latest release v4.50.0?

Would it also be possible to get inputs used for the processor call (can be also dummy videos, as long as input shapes match)?

[DrVictorBenjamin]

Hi @zucchini-nlp thanks for the response

I followed the tutorial exactly how it is, even using the sample datasets included (I chose the streaming route to acquire the example dataset)

I am indeed running the latest versions of every package involved as of 2 weeks ago, when I setup my environment to test the sample script

I did see one tip online where someone recommended changing the maxlength variable to a higher number - I tried 5000 but ran into the same issue

I'm a bit lost about your request regarding getting inputs used for the processor call, apologies about that

[DrVictorBenjamin]

BTW I just checked some debug output I wrote in my last run of the code, it seems to fail near the end on this line:

"run_inference(example["pixel_values_videos"], model)"

[zucchini-nlp]

Oh right, we definitely need to change the max length. If you don't know how long the inputs will be, you can also remove truncation=True, max_length=MAX_LENGTH here and just put padding=True so it pads to max length possible

def collate_fn(example, path):
    ...
    batch = processor(
        text=prompt,
        videos=video_clip,
        truncation=True,
        max_length=MAX_LENGTH,
        return_tensors="pt"
    )

    return batch

I meant to print the input ids after the dataset was processed and mapped. Specifically, you can see how many placeholders the batch has and how many frames the video has. If it matches at this point, then problem is either in modeling or trainer, otherwise you need a higher max length indeed.

tokens_in_text = (input_ids == self.config.videos_token_index).sum(-1)
pixel_values_videos.shape # each video frame is mapped to 144 tokens, 8 frames -> 1152 tokens
input_ids.shape 

[NielsRogge]

@zucchini-nlp should we make any updates to the notebook?

[zucchini-nlp]

yes, I will do so. It is rather lower priority among other tasks, feel free if you have more bandwidth :)

[DrVictorBenjamin]

Thanks to you both. I will try the padding=True change tomorrow and will report back here in case anyone else stumbles on this thread searching for a solution. If that doesn't work I'll try the other code you provided. I assume it should be inputted just before the "Load model" section in your tutorial?

[DrVictorBenjamin]

Hi @zucchini-nlp unfortunately commenting out max_length and setting padding = true has resulted in the same error

Would appreciate greatly if you can take a look sometime at the tutorial, but understood if you can't. May be worth pulling offline in the mean time so others don't follow it and get stuck as well

[zucchini-nlp]

Hm interesting, I will give it a look later. Since the dataset is hue to download, can you try to print one train example after collate_fn is mapped, like:

sample["input_ids"].shape, sample["pixel_values_videos"].shape, (sample["input_ids"] == model.config.video_token_index).sum(-1)

If one dataset point is correctly aligned in tokens, then smth happens later in the run, in modeling code or trainer. Otherwise collate is not correct for transformers==4.50.0

[DrVictorBenjamin]

Sorry I tried to get your example code snippet you provided to run but was unable to retrieve meaningful output for diagnosing the problem. Curious what version of transformers you think may work? I can try older packages.

[zucchini-nlp]

Btw, we will be raising errors from now on whenever inputs get truncated too much. That will be in the next release