Leaderboard: Local reproducibility

[Pringled]

Hi!

I'm trying to reproduce the leaderboard tables locally, but running into some issues. Version: mteb==1.34.4. I'm using the following code:

from mteb.load_results import load_results
from mteb.leaderboard.table import scores_to_tables

benchmark_results = load_results(
    results_repo="https://github.com/Pringled/mteb-results",
    only_main_score=True
)

scores_long = benchmark_results.get_scores(format="long") 
summary_df, per_task_df = scores_to_tables(scores_long=scores_long)
print(summary_df)
print(per_task_df)

Where the results repo is a fork of https://github.com/embeddings-benchmark/results containing a subset of the existing model results.

I get the following error:

mteb/leaderboard/table.py", line 132, in scores_to_tables
    per_task = data.pivot(index="model_name", columns="task_name", values="score")

raise ValueError("Index contains duplicate entries, cannot reshape")
ValueError: Index contains duplicate entries, cannot reshape

I could not find a simple script in the docs to do this, and from looking at the new leaderboard code, this looks like what's happening on the live leaderboard. Is there an extra step I'm missing? Thanks in advance!

[KennethEnevoldsen]

I believe there are a few calls made on the benchmark_results object (e.g. join_revisions) that need to happen before you can make it a table (otherwise you will get duplicate model entries).

Note that you can also just download the table from the leaderboard, which you can also run locally using:

from mteb.leaderboard import demo

demo.launch()

That way to get the exact processing of the leaderboard (though I agree that it is not quite as convenient).

[Pringled]

Thanks for the quick response @KennethEnevoldsen! I'm mainly interested in viewing the leaderboard scores of models I'm developing before actually submitting them to MTEB, so viewing the existing leaderboard doesn't help there unfortunately. I will have a look at the calls that need to be made.

Would you be interested in having this functionality in MTEB at some point, for example, a single function that can be used to reproduce the MTEB leaderboard tables from benchmark_results locally?

[KennethEnevoldsen]

Ahh sorry, I was unclear. You can run the leaderboard using the local results repository, by setting the environment variable

import os
path_to_my_results = ...
os.environ["MTEB_CACHE"] = str(path_to_my_results)

so for viewing you should be all good.

[KennethEnevoldsen]

Would you be interested in having this functionality in MTEB at some point, for example, a single function that can be used to reproduce the MTEB leaderboard tables from benchmark_results locally?

Would absolutely love that. Would welcome a PR on it.

[x-tabdeveloping]

You should try:

benchmark_results = benchmark_results.join_revisions()

[Pringled]

Thanks @KennethEnevoldsen and @x-tabdeveloping! I've managed to figure it out. I've created a PR as well: #2015.