[Usage] AssertionError Applying Delta to Get LLaVA Weights: edit_head.query not in base model

[TainanKyle]

Describe the issue

Issue:

Hi,

I followed your instructions to get the LLaVA weights by applying the delta, but I encountered the error AssertionError: edit_head.query not in base model.

Command:

python3 -m llava.model.apply_delta \
    --base /path/to/llama-7b \
    --target /output/path/to/LLaVA-7B-v0 \
    --delta liuhaotian/LLaVA-Lightning-7B-delta-v1-1

Log:

'vae.encoder.down_blocks.2.resnets.1.norm1.bias', 'unet.up_blocks.2.resnets.0.conv1.weight', 'unet.up_blocks.3.attentions.2.transformer_blocks.0.ff.net.0.proj.weight', 'unet.mid_block.resnets.1.conv1.weight', 'edit_head.mapper.decoder.layers.3.norm1.bias', 'vae.decoder.up_blocks.2.resnets.2.norm1.weight', 'unet.down_blocks.2.resnets.1.norm1.bias', 'unet.up_blocks.3.attentions.2.transformer_blocks.0.attn1.to_out.0.weight', 'unet.down_blocks.2.attentions.1.transformer_blocks.0.attn1.to_q.weight', 'unet.up_blocks.3.attentions.0.transformer_blocks.0.ff.net.2.weight', 'unet.up_blocks.1.resnets.1.conv1.bias', 'vae.decoder.mid_block.resnets.0.conv1.bias', 'vae.decoder.conv_in.bias', 'unet.down_blocks.0.attentions.0.transformer_blocks.0.attn1.to_out.0.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
/home/gpl_homee/miniconda3/envs/mgie/lib/python3.10/site-packages/huggingface_hub/file_download.py:795: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
generation_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 137/137 [00:00<00:00, 437kB/s]
tokenizer_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 833/833 [00:00<00:00, 2.80MB/s]
tokenizer.model: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500k/500k [00:00<00:00, 4.85MB/s]
added_tokens.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88.0/88.0 [00:00<00:00, 317kB/s]
special_tokens_map.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 97.0/97.0 [00:00<00:00, 346kB/s]
Applying delta
Applying delta:  23%|████████████████████████████████                                                                                                         | 325/1388 [00:10<00:34, 30.67it/s]
Traceback (most recent call last):
  File "/home/gpl_homee/miniconda3/envs/mgie/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/gpl_homee/miniconda3/envs/mgie/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/gpl_homee/indoor_scene/ml-mgie/LLaVA/llava/model/apply_delta.py", line 48, in <module>
    apply_delta(args.base_model_path, args.target_model_path, args.delta_path)
  File "/home/gpl_homee/indoor_scene/ml-mgie/LLaVA/llava/model/apply_delta.py", line 25, in apply_delta
    assert name in ['model.mm_projector.weight', 'model.mm_projector.bias'], f'{name} not in base model'
AssertionError: edit_head.query not in base model

I am using llava 0.2.0. Could you please advise how to resolve this issue?

Thanks!

[CaglarTaha]

Update LLaVA code to latest main branch for delta compatibility

• Switched from v0.2.0 to the main branch to ensure compatibility with the LLaVA-Lightning delta,
  which requires support for the new edit_head layers and other architecture changes.

• Updated installation instructions: now use pip install -e . to match the latest dependencies.

• Updated delta application command:

  python3 -m llava.model.apply_delta
  --base /path/to/llama-7b
  --target /output/path/to/LLaVA-7B-v0
  --delta liuhaotian/LLaVA-Lightning-7B-delta-v1-1

• Reminder: Confirm that your base model is LLaMA-7B (and not LLaVA pretrained weights) and that
  the provided paths for --base and --target are correct and writable.

• For troubleshooting, refer to the LLaVA-Lightning Documentation.

This should resolve assertion failures caused by mismatched versions.