Aggregate results of a custom model

[oroojlooy]

I have four questions.

1. Is there any automated way to aggregate results of a custom model to be used in a local leader-board?
   Right now in the result I have the results of validation and test chunks, for different languages. Was not sure if I only need to report the average stats among all languages, and assuming only for text. Can you please clarify?

2. hen I download the result table via mteb.load_results(tasks=tasks), it gives me bunch of warnings like:

MTOPDomainClassification: Missing subsets {'en'} for split test
MassiveIntentClassification: Missing subsets {'en'} for split test
MassiveScenarioClassification: Missing subsets {'en'} for split test
MassiveIntentClassification: Missing subsets {'en'} for split test

Is this expected? If I want an apple to apple comparison, I assume I need to remove en results of test chunk for these datasets for my custom model as well, right?

3. What are the metrics reported for each task-type/dataset? For example, for classification tasks, do you report F1 score or accuracy? And a similar question for other tasks like retrieval, summarization, etc.
   From the paper I found the following, but wanted to confirm:

{"BitextMining": "F1", "Classification": "accuracy", "Clustering": "v_measure",
                     "PairClassification": "cosine_ap", "Reranking": "map", "Retrieval": "ndcg_at_10",
                     "STS": "cosine_spearman", "Summarization": "cosine_spearman"}

4. The results of the different models which are reported in the leaderboard for each dataset are different than reported results that can be downloaded viaresults = mteb.load_results(tasks=tasks). For example, for STS17 dataset, "google/gemini-embedding-exp-03-07" the leaderboard has 88.57 and the mteb.load_results(tasks=tasks) has 91.6. Any idea?

[Samoed]

1. Currently you can't, but we have PR for this CLI Tool for results dataframe on leaderboard #2454
2. Yes, this is expected, because some models could have missing results
3. You can look target metric in TaskMetadata.main_score for each task
4. Can you please specify which benchmark you are referring to?

[oroojlooy]

Thanks for the provided information.
4. I am using all benchmark datasets on MTEB(eng, v2), but not just english, all languages.

[KennethEnevoldsen]

It is possible to load the results into a dataframe:

You can see the documentation for this here:
https://github.com/embeddings-benchmark/mteb/blob/main/docs/usage/results.md

[oroojlooy]

I know it is possible to load the dataset into a data frame. The issue is the results that I get from results = mteb.load_results(tasks=tasks) is different than the result which is available in the benchmark in https://huggingface.co/spaces/mteb/leaderboard
Like the example that I provided above.
Any idea why they are different?

[KennethEnevoldsen]

Hmm, shouldn't be - Can you please provide the exact code you run to get the scores?

[Samoed]

I think the leaderboard might display only selected splits or subsets of tasks. However, when you use mteb.load_results (assuming you ran tasks = mteb.get_tasks([task_names])), it loads all splits and subsets.

[oroojlooy]

I am using

tasks = mteb.get_benchmark("MTEB(eng, v2)").tasks
task_df = tasks.to_dataframe(properties=["name", "type", "domains"])
results = mteb.load_results(tasks=tasks)
df = results.to_dataframe(format="wide")

to get the results.

One thing that might be the root cause of that is the result-filtering on language on the leaderboard is not work correctly. Basically, it may not care that I set language=eng, and as a result it is reporting the results of all languages.

Is there any way to get the list of tasks for all languages with mteb.get_benchmark()?

[oroojlooy]

@Samoed So, what you are saying is the df = results.to_dataframe() is the most accurate version, right?

[Samoed]

@Samoed So, what you are saying is the tasks = mteb.get_tasks([task_names])) is the most accurate version, right?

No, I'm just saying that there can be difference on how you've taken tasks.

One thing that might be the root cause of that is the result-filtering on language on the leaderboard is not work correctly. Basically, it may not care that I set language=eng, and as a result it is reporting the results of all languages.

Can you create tables to compare how scores are different?

Is there any way to get the list of tasks for all languages with mteb.get_benchmark()?

I'm not sure what you want. To get tasks directly, you can use mteb.get_tasks. In benchmarks, list of tasks is predefined with splits and subsets and can't be changed

[oroojlooy]

Here is the issue. I get the benchmark results using:

tasks = mteb.get_benchmark("MTEB(eng, v2)").tasks
results = mteb.load_results(tasks=tasks)
df = results.to_dataframe(format="wide")

which apparently only gets the results of eng language, and it looks like there is no other way to get the results of datasets on MTEB v2 on other languages.

Then, I have the result of a custom model obtained on all languages of MTEB v2, by calling:

    model = MyCustomModel()
    tasks = ["ArguAna", "ArXivHierarchicalClusteringP2P", "ArXivHierarchicalClusteringS2S", "AskUbuntuDupQuestions",
             "BIOSSES", "Banking77Classification", "BiorxivClusteringP2P.v2", "CQADupstackGamingRetrieval",
             "CQADupstackUnixRetrieval", "ClimateFEVERHardNegatives", "FEVERHardNegatives", "FiQA2018",
             "HotpotQAHardNegatives", "ImdbClassification", "MTOPDomainClassification", "MassiveIntentClassification",
             "MassiveScenarioClassification", "MedrxivClusteringP2P.v2", "MedrxivClusteringS2S.v2",
             "MindSmallReranking", "SCIDOCS", "SICK-R", "STS12", "STS13", "STS14", "STS15", "STSBenchmark",
             "SprintDuplicateQuestions", "StackExchangeClustering.v2", "StackExchangeClusteringP2P.v2", "TRECCOVID",
             "Touche2020Retrieval.v3", "ToxicConversationsClassification", "TweetSentimentExtractionClassification",
             "TwentyNewsgroupsClustering.v2", "TwitterSemEval2015", "TwitterURLCorpus", "SummEvalSummarization.v2",
             "AmazonCounterfactualClassification", "STS17", "STS22.v2"]
    for cnt, task in tqdm(enumerate(tasks), total=len(tasks)):
        evaluation = mteb.MTEB(tasks=[task])
        results = evaluation.run(model, verbosity=3)

This ends in getting the results of all datasets in benchmark MTEB v2 on all possible languages. For example, STS17 is available on 11 languages, so the results are also available on 17 languages.
To compare the results of my custom model, I need to compare its results to benchmarks when all languages are contributing to the results, not just english language.

So, now what I need is how to get the final metric value datasets on MTEB v2 for all languages.
Does this make sense?

[oroojlooy]

Also, the result of some datasets in "MTEB(eng, v2)" are not available in the leaderboard. For example, AskUbuntuDupQuestions is not in the leaderboard, and not in
https://github.com/embeddings-benchmark/results/tree/main/results
as well. Do you store the results of those datasets somewhere else?

[Samoed]

which apparently only gets the results of eng language, and it looks like there is no other way to get the results of datasets on MTEB v2 on other languages.

Yes, this is because MTEB(eng, v2) is engish only benchmark.

So, now what I need is how to get the final metric value datasets on MTEB v2 for all languages.
Does this make sense?

You can evaluate on MTEB(Multilingual, v1) if you want to evaluate on mltiple languages.

AskUbuntuDupQuestions is not in the leaderboard

It's presented on the leaderboard

[oroojlooy]

Thanks for pointing out to MTEB(Multilingual, v1). This should be helpful.

It's presented on the leaderboard

I still do not see it there. Any idea?
Also as I mentioned it is not available in https://github.com/embeddings-benchmark/results/tree/main/results

One more question, for some of datasets, I get a max-token error from my embedding model. What is the common practice for long tokens? truncation?
Is there any way to introduce the max-token size of the custom model in the CustomModel?

[Samoed]

One more question, for some of datasets, I get a max-token error from my embedding model. What is the common practice for long tokens? truncation?

Yes. You should fix your tokenizer, but it's hard to tell without implementation