What exactly is the base attn donor?

[icsy7867]

Hello! Sorry for the potentially stupid question, but I am having trouble with the documentation and understanding what exactly this is doing when making a clowncar MOE.

Looking at:
https://github.com/arcee-ai/mergekit/blob/main/docs/moe.md

I originally thought that if I included a model with the "Base model" and 2 experts, this should come out to about 36B parameters (I am trying out a merge with 12B Mistral Nemo models).

However, this actually ended up around 24B parameters. So what exactly is the base model donating in this case? Should the base model always be included as one of the experts?

Thank you!