Add support for Qwen MoE (Qwen2MoeForCausalLM)

[maziyarpanahi]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Feature Description

Please provide a detailed written description of what you were trying to do, and what you expected llama.cpp to do as an enhancement.

Qwen1.5-MoE-A2.7B, a small MoE model with only 2.7 billion activated parameters yet matching the performance of state-of-the-art 7B models like Mistral 7B and Qwen1.5-7B.

Compared to Qwen1.5-7B, which contains 6.5 billion non-embedding parameters, Qwen1.5-MoE-A2.7B contains only 2.0 billion non-embedding parameters, approximately one-third of Qwen1.5-7B’s size. Notably, it achieves a 75% decrease in training expenses and accelerates inference speed by a factor of 1.74, offering substantial improvements in resource utilization without compromising performance.

we have incorporated 4 shared experts to be always activated alongside 60 routing experts with 4 to be activated

https://qwenlm.github.io/blog/qwen-moe/

Motivation

Please provide a detailed written description of reasons why this feature is necessary and how it is useful to llama.cpp users.

Not sure if it's safe to use --skip-unknown here:

model.embed_tokens.weight                        -> token_embd.weight                        | BF16   | [151936, 2048]
model.layers.0.input_layernorm.weight            -> blk.0.attn_norm.weight                   | BF16   | [2048]
Traceback (most recent call last):
  File "/llama.cpp/convert.py", line 1486, in <module>
    main()
  File "/llama.cpp/convert.py", line 1472, in main
    model   = convert_model_names(model, params, args.skip_unknown)
  File "/llama.cpp/convert.py", line 1217, in convert_model_names
    raise Exception(f"Unexpected tensor name: {name}. Use --skip-unknown to ignore it (e.g. LLaVA)")
Exception: Unexpected tensor name: model.layers.0.mlp.experts.0.down_proj.weight. Use --skip-unknown to ignore it (e.g. LLaVA)

• https://huggingface.co/Qwen/Qwen1.5-MoE-A2.7B
• https://huggingface.co/Qwen/Qwen1.5-MoE-A2.7B-Chat

Possible Implementation

If you have an idea as to how it can be implemented, please write a detailed description. Feel free to give links to external sources or share visuals that might be helpful to understand the details better.

Haven't looked at it yet, but could be more similar to Mixtral than DBRX

[maziyarpanahi]

I tried --skip-unkown, as expected, it didn't work:

llama_model_load: error loading model: create_tensor: tensor 'blk.0.ffn_gate.weight' not found
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model 'quantized/Qwen/Qwen1.5-MoE-A2.7B/Qwen1.5-MoE-A2.7B.fp16.gguf'
main: error: unable to load model

[DisOOM]

@maziyarpanahi (#6074) This PR has not been merged yet

[maziyarpanahi]

@maziyarpanahi (#6074) This PR has not been merged yet

Thanks a lot! I missed the moe part, just saw Qwen2 and assumed it's different :)

[maziyarpanahi]

I would like to also leave this PR on the record since it's about MoE and Qwen models: #6453

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.