Proposal: Building an Arabic Leaderboard for MTEB

[haytamdon]

Summary

We propose creating a section of MTEB for the the Arabic language MTEB-AR. We aim to add the Standard Arabic language in the languages taken in consideration MTEB by integrating tasks from MTEB

Motivation

Arabic is one of the most common language in the world with millions of speakers and a lot of literature developed in arabic throughout time. We aim to:

• Encourage the development of embedding models in the arabic language and the different variants & dialects of the language
• Provide a benchmark that allows for fair comparison of the Arabic models

Request for Feedback

I would appreciate input from the maintainers and community on:

• Recommended datasets or tasks that should be prioritized for Arabic.
• Suggestions for additional tasks or metrics relevant to Arabic embeddings.

FInal Note

Looking forward to your feedback and insights!

[haytamdon]

@imenelydiaker I think you're currently working on the topic if you have inputs it would greatly appreciated thank you

[imenelydiaker]

Hi @haytamdon, thank you for the interest in our project! We're always welcoming contributions! 🎉

When building MMTEB, the community helped us build 28 tasks for Arabic (cf screenshot below). Clearly, we're missing some tasks to make an Arabic benchmark, it would be nice to add more.

Also, I came across this paper: https://arxiv.org/abs/2411.01192, they say the code will be released along with the models. Maybe it's worth it to contact the authors and help them integrate their work into MTEB. They have models and tasks, once everything is implemented we can think about adding an Arabic tab to the current leaderboard.

[haytamdon]

Thank you for your response, but I was thinking of create an MTEB-AR for the arabic language itself to evaluate it independantly not specifically in MMTEB

[Samoed]

Also issue about this paper #1408

[imenelydiaker]

Thank you for your response, but I was thinking of create an MTEB-AR for the arabic language itself to evaluate it independantly not specifically in MMTEB

Actually MMTEB is a collection of benchmarks, it's a v2 of MTEB where we added support for more languages. I just referenced our paper to let you know that there have been some work on Arabic but not enough for an Arabic benchmark.

MTEB-AR would be a benchmark we'd create out of a collection of tasks like we did here for the German benchmark:

mteb/mteb/benchmarks/benchmarks.py

Line 551 in 762f729

MTEB_DEU = Benchmark(

. We agree that you'd like to build something like this?

If that's the case then the paper and the issue @Samoed mentioned can help us build it. We just need to contact the authors and help them integrate their work and we'll have the benchmark for Arabic 😊

Do you have other NLP tasks you'd like to evaluate Arabic tasks on? Other than the ones present in MTEB (classification, retrieval, reranking, etc.)?