CLI Tool for results dataframe on leaderboard

[ayush1298]

This PR is related to addition of CLI Tool for inspection and creation of dataframe of results as suggested in issue #2174 .
closes #2174 .

To use it command is:

mteb create-table  --results <results_path> --models model1 model2 --benchmark benchmark_name1 benchmark_name2 --aggregation-level <aggregation-level > --output <output_file>

1. results_path: Path to the results folder. Works with local results repo or the official MTEB results repository
2. Compare multiple models side by side across different tasks.
3. benchmark_name: Name of the benchmark (optional)
4. Three aggregation levels:

• subset: Results for each subset within each split for each task
• split: Results aggregated over subsets for each split for each task
• task: Results aggregated over subsets and splits for each task

5. Multiple output formats: CSV, Excel, or Markdown (based on file extension)
6. output_path: Path to save the output tables

Sample example to run these is for openai/text-embedding-3-large, openai/text-embedding-3-small models is using following command:

mteb create-table --results https://github.com/embeddings-benchmark/results \
                 --models "openai/text-embedding-3-large" "openai/text-embedding-3-small" \
                 --benchmark "MTEB(Multilingual, v1)" \
                 --aggregation-level subset \
                 --output comparison_table4.csv

Based on the aggregation, we get different results. As full csv file is very large, I am sharing below results of these 2 models on AmazonCounterfactualClassification task having test and validation splits and 4 subsets: de,en,en-ext,ja, so as to verify that aggregation is actually working.

subset aggregation level

split aggregation level

task aggregation level

Code Quality

• Code Formatted: Format the code using make lint to maintain consistent style.

Documentation

• Updated Documentation: Add or update documentation to reflect the changes introduced in this PR.

Testing

• New Tests Added: Write tests to cover new functionality. Validate with make test-with-coverage.
• Tests Passed: Run tests locally using make test or make test-with-coverage to ensure no existing functionality is broken.

[Samoed]

Great addition!

[ayush1298]

Update PR description with CLI command to run, sample example and results and their verifications

[Samoed]

Great! Can you create test for this like test_create_meta_from_existing?

mteb/tests/test_cli.py

Lines 137 to 138 in 43adb0c

	def test_create_meta_from_existing(existing_readme_name: str, gold_readme_name: str):
	"""Test create_meta function directly as well as through the command line interface"""

[Samoed]

I tried to run this command, but it is not working with local dirs. I tried to run with mteb create-table --results v2/ --output example.md, but it tried to load all results from results repo
v2.zip

[ayush1298]

I tried to run this command, but it is not working with local dirs. I tried to run with mteb create-table --results v2/ --output example.md, but it tried to load all results from results repo v2.zip

With local dir, there is some issue ig. As I have also got the same thing. I will try to find it out. You can try with github url as given in PR description. That was working as expected.

[ayush1298]

@Samoed, Its working now with local path as well. Also, in tests. for testing it with local results repo, should I do results of both these models under tests/results?

[ayush1298]

@KennethEnevoldsen Can you please review this PR?

[Samoed]

Great work!

[KennethEnevoldsen]

Thanks for this PR.

Currently it seems like we haven't really added a meaningful interface in python.

I think that there is a few use cases that we can see:

1. I have run a model and want to see how well it compared to another model
2. I make a PR to embeddings-benchmark/results and want to create a markdown table of performances

There is probably more, but I think it is worth spending some more time on getting this one right.

Feel free to just outline how you think it might look like in a test.

[KennethEnevoldsen]

I would love an additional test that doesn't go through the CLI.

[ayush1298]

@KennethEnevoldsen, without CLI how to test, because we are mainly dependent on output file that CLI is producing.

[Samoed]

Currently it seems like we haven't really added a meaningful interface in python.

I've created a table with create_comparison_table, and it worked fine. Do you want to create a table from a dict?

I make a PR to embeddings-benchmark/results and want to create a markdown table of performances

I think this table can be created using the current CLI. Just the model name needs to be specified.

I have run a model and want to see how well it compared to another model

You want to compare results from different directories?

[KennethEnevoldsen]

But there is no documentation on the python interface and no tests for functionality

You want to compare results from different directories?

Taking a user perspective: I just evaluated my model (let's say a fine-tune of e5) and now want to compare the it with the original model. I don't care what directory something is in.

[Samoed]

Taking a user perspective: I just evaluated my model (let's say a fine-tune of e5) and now want to compare the it with the original model. I don't care what directory something is in.

I think this can be done with additional command (or flag to current command) to compare current results with leaderboard or tasks from folder, but maybe this can be done in other PR

[ayush1298]

2. I make a PR to embeddings-benchmark/results and want to create a markdown table of performances

Are you saying that PR is not merged and user wants to check results before merging?

But there is no documentation on the python interface and no tests for functionality

Are you saying about adding documentation related to this, I have added how to use it at:

mteb/mteb/cli.py

Lines 79 to 89 in 89db9ac

	## Creating Comparison Tables
	To create comparison tables between models based on various aggregation levels (task, split, or subset), use the `mteb create-table` command. For example:
	```bash
	mteb create-table --results results/ \
	--models "intfloat/multilingual-e5-small" "intfloat/multilingual-e5-base" \
	--benchmark "MTEB(eng, v1)" \
	--aggregation-level task \
	--output comparison_table.csv
	"""

. And shared some of results in PR description. Also, if you want me to add this in readme.md or in separate file, then I will add that at end once we have finalized with rest of things.

Taking a user perspective: I just evaluated my model (let's say a fine-tune of e5) and now want to compare the it with the original model. I don't care what directory something is in.

I am assuming that if fine-tune e5 is also run and results are produced for it, but they are not uploaded to results repo. Then, results will be saved under mteb/results. So, either user can add this as results path or else as @Samoed suggested we can set an additional argument something which tells if model results are present in results repo or not. We can set default to true, and if user passes this as false, then we will set results path as mteb/results instead of default results repo. By these I think we can also avoid results argument which takes results_path by setting its default to results repo (Github URL). 1 more thing that can be done is in load_results or maybe in separate function, we should check directly if results folder is present locally(Don't know how to do that exactly). Because, if we can do that, then if it exists then we will not clone results repo again. And if it not exists, then in same function, we will add functionality to clone the repo directly.
Now, user just have to give name of the models, and can get the tables.

[KennethEnevoldsen]

Hmm, I will skip many of the comments here. This is not to be rude, but rather to focus on what I think is the main problem.

While I agree that this PR targets the Issue, it does it in a very direct way. This is a problem because then the CLI dictates the Python interface. This makes it inflexible and hard to test (we can only test the input/output).

Here is a few example of why I don't think create_comparison_table is a good python interface:

• if I already have a mteb.load_results(...) I would have to recreate that to get to a dataframe
• it is a different interface that load_results and it is not compatible (e.g. exclude_meta is forced)
• If I want to check something in the results before I make the table that is impossible
• If something is missing in the table, I have no idea why/when it was filtered out

So here is an alternative suggestion:

models = ... # desired models
tasks = mteb.get_benchmark("MTEB(eng, v2)")

benchmark_results = benchmark.load_results(models = models, tasks = tasks)
df = benchmark_results.to_dataframe(aggregate = "task")

This might not be perfect, but I think it is definitely true. more flexible than the current implementation.

[ayush1298]

@KennethEnevoldsen, sorry for having too long comments previously.

if I already have a mteb.load_results(...) I would have to recreate that to get to a dataframe

Using implementation that you suggested, still user has to run mteb.load_results(...) atleast 1 time. And in current implementation using below code,

results_path = Path(results_repo)
    if results_path.exists() and results_path.is_dir():
        logger.info(f"Using local results repository at {results_path}")
        return results_path

if it run it once, then it will use that only and not load again when we use create_comparison_table again.

If I want to check something in the results before I make the table that is impossible

What kind of checking you are referring here(any example), I think our main aim behind this was, just to provide easy way of comparing results.

If something is missing in the table, I have no idea why/when it was filtered out

Am I missing something here, because in whole code we are not doing any specific filtering.

[Samoed]

I've used this command for creating tables (thanks for this feature!) in #2415 and I think there can be some improvements:

1. Add flag for transposing tables, because when I have 2-3 tasks and 5 models, it would be simpler to have tasks as headers.
2. load_results is expecting all data in results folder, but I can have different names and this is a bit confusing to have custom_name/results just to correctly create table

   mteb/mteb/load_results/load_results.py

   Lines 114 to 115 in 89db9ac

   	repo_directory = download_of_results(results_repo, download_latest=download_latest)
   	model_paths = [p for p in (repo_directory / "results").glob("*") if p.is_dir()]

[KennethEnevoldsen]

Using implementation that you suggested...

if it run it once, then it will use that only and not load again

Hmm, but yes, but if I already have it in memory, then I wouldn't. E.g.:

# analysis script
results = mteb.load_results(...)

# Some checks and analyses (e.g. task X is included, all tasks are run etc.)
# potentially custom filertering e.g. mteb version higher than X

# create table
create_comparison_table(...)
# requires respecification of argument
# does not guarantee that the results are the same as what is checked
# has to reload the results

Just to clarify, I think this is a good change, but I do want to standardize the way we convert the results object to tables (it should be the same code for the leaderboard, python API and CLI). So while it is good I don't think it is quite there yet.

If the suggested changes are unclear (which I can see that they might be). I can also write a suggested solution, and we can discuss it from there (but that will not be before Thursday)

[ayush1298]

@KennethEnevoldsen I understand some of the things, but it will be good if you can write suggested solution whenever you become free after thursday. I will work on this then.

[KennethEnevoldsen]

Perfect, I have set a reminder :)

[ayush1298]

@KennethEnevoldsen After merge of #2542, Is their any work left that is to be done in this particular PR as I have saw that you have added Prerequisite for https://github.com/embeddings-benchmark/mteb/pull/2454 in the description of that PR.
Sorry for replying after many days, I was busy because of exams.

[Samoed]

1. I think you can solve these problems firstly CLI Tool for results dataframe on leaderboard #2454 (comment)
2. And these comments left unresolved CLI Tool for results dataframe on leaderboard #2454 (review)

[ayush1298]

1. I think you can solve these problems firstly CLI Tool for results dataframe on leaderboard #2454 (comment)
2. And these comments left unresolved CLI Tool for results dataframe on leaderboard #2454 (review)

aren't all these gets solved after #2542? As kenneth wants instead of CLI, that should be a function like to_dataframe() which he implemented

[Samoed]

No, they're not solved. #2542 is added some functions to simplify working with BenchmarkResults and not address these issues

[ayush1298]

No, they're not solved. #2542 is added some functions to simplify working with BenchmarkResults and not address these issues

So, will still want these CLI Tool, right?
And using those updated functionality from #2542, we want to simplify these CLI tool correct?
Should I continue in this PR only, or close and create new one?

[Samoed]

So, will still want these CLI Tool, right?

Yes, and also python API

And using those updated functionality from #2542, we want to simplify these CLI tool correct?

Yes

Should I continue in this PR only, or close and create new one?

You can continue this PR

[KennethEnevoldsen]

Yes as @Samoed says, the PR that I made only adds a lot of dataframe utilities required for this PR.

You would have to refactor this PR to rely on the dataframe utilities (to ensure that we consistently do it the same way)

[KennethEnevoldsen]

@ayush1298 just pinging you here

[ayush1298]

@ayush1298 just pinging you here

Sorry, forgot about this one. Will start working on this again on weekends.

[KennethEnevoldsen]

@ayush1298 seems like we never finished up this PR, do you still have time to work on it?