Replies: 1 comment 2 replies
-
|
We're checking all splits datasets results |
Beta Was this translation helpful? Give feedback.
-
|
So just to expand: If a model is run on a task for only English, but not all languages, then it will be filtered out and not included in the leaderboard. |
Beta Was this translation helpful? Give feedback.
-
|
If a task appears on two benchmarks, it can have different languages specified (and thus the score will be different). We could write all of these out:
However, currently, the average is made across tasks though. |
Beta Was this translation helpful? Give feedback.
-
The calculation logic of MTEB(Multilingual) seems to involve many multilingual datasets, where the main score for each dataset is the average of the results across various languages (considering only the languages that the model has been evaluated on). I am quite puzzled about how it is ensured that all the models currently on the leaderboard have been evaluated on all the languages. Are these results now comparable?
Beta Was this translation helpful? Give feedback.
All reactions