evaluation script for MMTEB

[lifu-tu]

Hi, recently, I found your MMTEB work. The benchmark looks good. I would like to evaluate some models on this benchmark. Can you share a sample script for the evaluation? Thanks!

[Samoed]

You can find example scripts in readme

mteb/README.md

Lines 41 to 97 in 4d23c6c

	* Using a Python script:
	```python
	import mteb
	from sentence_transformers import SentenceTransformer
	# Define the sentence-transformers model name
	model_name = "average_word_embeddings_komninos"
	model = mteb.get_model(model_name) # if the model is not implemented in MTEB it will be eq. to SentenceTransformer(model_name)
	tasks = mteb.get_tasks(tasks=["Banking77Classification"])
	evaluation = mteb.MTEB(tasks=tasks)
	results = evaluation.run(model, output_folder=f"results/{model_name}")
	```
	<details>
	<summary> Running SentenceTransformer model with prompts </summary>
	Prompts can be passed to the SentenceTransformer model using the `prompts` parameter. The following code shows how to use prompts with SentenceTransformer:
	```python
	from sentence_transformers import SentenceTransformer
	model = SentenceTransformer("average_word_embeddings_komninos", prompts={"query": "Query:", "passage": "Passage:"})
	evaluation = mteb.MTEB(tasks=tasks)
	```
	In prompts the key can be:
	1. Prompt types (`passage`, `query`) - they will be used in reranking and retrieval tasks
	2. Task type - these prompts will be used in all tasks of the given type
	1. `BitextMining`
	2. `Classification`
	3. `MultilabelClassification`
	4. `Clustering`
	5. `PairClassification`
	6. `Reranking`
	7. `Retrieval`
	8. `STS`
	9. `Summarization`
	10. `InstructionRetrieval`
	3. Pair of task type and prompt type like `Retrival-query` - these prompts will be used in all classification tasks
	4. Task name - these prompts will be used in the specific task
	5. Pair of task name and prompt type like `NFCorpus-query` - these prompts will be used in the specific task
	</details>
	* Using CLI
	```bash
	mteb available_tasks
	mteb run -m sentence-transformers/all-MiniLM-L6-v2 \
	-t Banking77Classification \
	--verbosity 3
	# if nothing is specified default to saving the results in the results/{model_name} folder
	```

[lifu-tu]

Thanks for the quick response. Actually, I am looking for the evaluation script for the whole MMTEB (the benchmark, not one task from MTEB). It looks the pointed file only includes the evaluation of a task in MTEB.

[Samoed]

Similarly you can run

import mteb
benchmark = mteb.get_benchmark("MTEB(Multilingual, v1)")
evaluation = mteb.MTEB(tasks=benchmark)
evaluation.run(model)

[lifu-tu]

The run of the evaluation is done. And all the results are saved in a folder for one model.

One question: is there a script in this repo to aggregated these results? For example, the results of each category and the average over all tasks? Thanks!

[Samoed]

Not really. You can try to reuse some parts from this script https://github.com/embeddings-benchmark/mteb/blob/main/scripts/make_leaderboard.py

[KennethEnevoldsen]

Added an issue on this #2174 (multiple people have asked)

[lifu-tu]

@KennethEnevoldsen One follow-up question: in the MMTEB leaderboard, there are several models need detailed instructions for each task (e,g, f'Instruct: {task_description}\nQuery: {query}' for [multilingual-e5-large-instruct]).

Could you show some suggestions how to do MMEB evaluation on these models? Thanks!

[Samoed]

If mean how to run on MMTEB, then code for running the same

import mteb
benchmark = mteb.get_benchmark("MTEB(Multilingual, v1)")
evaluation = mteb.MTEB(tasks=benchmark)
model = mteb.get_model("intfloat/multilingual-e5-large-instruct")
evaluation.run(model)

[lifu-tu]

Thanks for the quick reply.

Just double confirm: is the detailed instruction per task in MTEB(Multilingual, v1) added during the evaluation? If it is, could you point to the details instructions per task during evaluation?

[Samoed]

instruction per task in MTEB(Multilingual, v1) added during the evaluation?

Yes

If it is, could you point to the details instructions per task during evaluation?

You can find instructions in the TaskMetadata in the prompt field. If prompt is not specified, then abstaskprompt will be used from AbsTask class