Building a Hindi Leaderboard.

[SaileshP97]

Proposal: Building a Hindi Leaderboard for MTEB by Creating Hindi-Specific Datasets and Tasks

Summary

This proposal outlines the idea of creating a Hindi-specific leaderboard within the Massive Text Embedding Benchmark (MTEB) framework. The objective is to design and integrate Hindi language datasets and tasks derived from existing MTEB data. By focusing on Hindi language benchmarks, this project aims to improve the evaluation ecosystem for models generating embeddings specifically for Hindi NLP tasks.

Motivation

Hindi is one of the top five most spoken languages globally, with over 600 million speakers. Despite its widespread use, the performance of text embedding models on Hindi-specific tasks is often underrepresented in multilingual benchmarks. Building a dedicated Hindi leaderboard within MTEB will:

• Enable fair comparison of models on Hindi tasks, helping researchers focus on performance improvements for the language.
• Encourage the development of Hindi-centric embedding models and language-specific optimizations.
• Provide a comprehensive benchmark to evaluate Hindi embedding quality across multiple tasks.

Request for Feedback

I would appreciate input from the maintainers and community on:

• Recommended datasets or specific MTEB tasks that should be prioritized for Hindi.
• Suggestions for additional tasks or metrics relevant to Hindi embeddings.
• Collaboration opportunities to align this proposal with MTEB’s long-term roadmap.

Looking forward to your feedback and insights!

[SaileshP97]

Hi @KennethEnevoldsen,

I discussed this idea with Niklas, and he was quite excited about it. He suggested I tag you, so I wanted to clarify a few points before my team and I begin working on the project:

Is it possible to utilize existing multilingual datasets, extract the Hindi data, and use it for a different task?
Once the tasks are created, do we need to evaluate every model listed on the MTEB leaderboard and rank them on a Hindi-specific leaderboard?
Is there any documentation or reference material we can consult for guidance?

[KennethEnevoldsen]

This sounds like a great idea, feel free to loop both me and @x-tabdeveloping in as well.

The process should be fairly easy and we can help you.

Generally, the process is as follows:

• select a set of tasks (@x-tabdeveloping outlines this below). However, do also take a look at indic language benchmark tasks:

mteb/mteb/benchmarks/benchmarks.py

Line 916 in 950f050

MTEB_INDIC = Benchmark(

• (optional) add any tasks that are missing
• add the tasks to a benchmark in benchmarks.py
• run any model of interest (you do not need to run them all) - we have a bunch of model already implemented which you can fetch using mteb.get_model(name)
• (optional): write up a paper or a technical report on the results

[SaileshP97]

Hi Kenneth,
Thanks for your response! We have a meeting scheduled with Marton tomorrow at 10 am CET. I’ll forward the meeting invite shortly. Please feel free to join us.

[x-tabdeveloping]

@SaileshP97 Yup, this is totally possible. You can obtain a list of all hindi tasks by calling mteb.get_tasks(languages=["hin"]) the you can go through all tasks that contain Hindi. You can curate these to your liking and even restrict tasks to only use Hindi splits in the dataset, then add a custom Benchmark object in benchmarks.py. If you need help, we can take a meeting sometime and I can walk you through it.

[SaileshP97]

Thanks Marton. I have sent you a mail. Lets meet and discuss this.

[SaileshP97]

Hi @KennethEnevoldsen and @x-tabdeveloping , How can I generate the default.jsonl file for data_overall and data_tasks in Leaderboard/boards_data/language. Is there any code to rank the models?

[Samoed]

They are autogenerated if you submit your scores to results repo and recreate EXTERNAL_MODELS_RESULTS.json, but I think you should look to new leaderboard version. There results will be automatically added if you submit benchmark to main repo

[SaileshP97]

Thanks @Samoed

[x-tabdeveloping]

I agree, we would probably prefer if you focused on the new leaderboard, since we intend to release it in the near future (couple of weeks at most).