Skip to content

Integrate mme5 models #2081

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Integrate mme5 models #2081

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
481d1a2
Integrate mme5 models
Aradhye2002 Feb 17, 2025
6c6c0a3
Add mme5 folder
Aradhye2002 Feb 17, 2025
36a1673
Merge branch 'embeddings-benchmark:main' into integrate-mme5
Aradhye2002 Feb 17, 2025
0aef67c
Merge branch 'embeddings-benchmark:main' into integrate-mme5
Aradhye2002 Feb 24, 2025
044581f
Merge branch 'embeddings-benchmark:main' into integrate-mme5
Aradhye2002 Mar 2, 2025
79085a8
change to sentence transformer
Aradhye2002 Mar 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions mteb/models/mme5_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
from __future__ import annotations

from functools import partial

from mteb.model_meta import ModelMeta, sentence_transformers_loader

mme5_model = ModelMeta(
loader=partial(
sentence_transformers_loader,
model_name="intfloat/mmE5-mllama-11b-instruct",
revision="cbb328b9bf9ff5362c852c3166931903226d46f1",
),
Comment on lines +8 to +12
Copy link
Member

@Samoed Samoed Mar 2, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you try to run it? It hard to tell if it will mach with authors implementation, because they were benchmarking on MMEB #1709

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which benchmark would you like me to run it on? MMEB is not implemented currently it seems.

Copy link
Member

@Samoed Samoed Mar 2, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just on some task to make sure that it working. Later we can check implementation (you can create issue for it)

name="intfloat/mmE5-mllama-11b-instruct",
languages=["eng_Latn"],
revision="cbb328b9bf9ff5362c852c3166931903226d46f1",
release_date="2025-02-16",
modalities=["image", "text"],
n_parameters=10_600_000_000,
memory_usage_mb=20300,
max_tokens=8192,
embed_dim=7680,
license="mit",
open_weights=True,
public_training_code=None,
public_training_data="https://huggingface.co/datasets/intfloat/mmE5-synthetic",
framework=["Sentence Transformers", "PyTorch"],
reference="https://huggingface.co/intfloat/mmE5-mllama-11b-instruct",
similarity_fn_name="cosine",
use_instructions=True,
training_datasets=None,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
training_datasets=None,
training_datasets={
# Synthetic
},
adapted_from="meta-llama/Llama-3.2-11B-Vision",

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, pydantic throws an error with this since training_datasets needs to be a string or None. What do you think should be done?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thats strange, because it should be dict

mteb/mteb/model_meta.py

Line 115 in 587892d

training_datasets: dict[str, list[str]] | None

)
2 changes: 2 additions & 0 deletions mteb/models/overview.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@
linq_models,
llm2vec_models,
misc_models,
mme5_models,
moco_models,
model2vec_models,
moka_models,
Expand Down Expand Up @@ -108,6 +109,7 @@
linq_models,
llm2vec_models,
misc_models,
mme5_models,
model2vec_models,
moka_models,
moco_models,
Expand Down