What evaluators and AbsTasks can MIEB reuse / create common ones?

[isaac-chung]

From #1337 (comment). Originally, I see mainly a few main options:

1. Some tasks in MIEB directly reuse MTEB classes. This is not possible.
2. Some tasks in MTEB directly reuse MIEB classes. e.g. Any2AnyRetrievalEvaluator?
3. Status quo: We create a separate modality folder for modality specific classes
4. Combine: we create classes that can be used across MIEB and MTEB. Some require little modifications, e.g. STSEvaluator and VisualSTSEvaluator share pretty much everything below the line logger.info("Evaluating..."). The rest seem to need some modifications, i.e. unify the modalities from which embeddings are generated.

I'm leaning forward option 4 and would like to lay out how to potentially write classes that accommodate both mieb and mteb tasks. Then consider the extensibility in anticipation of MOEB modalities, e.g. audio.

This may be closely related to the encoder interface discussion, so maybe that's something we should establish first?

Tagging @gowitheflow-1998 @imenelydiaker @KennethEnevoldsen @Muennighoff @x-tabdeveloping @Samoed @orionw

[Samoed]

I think we should create a common abstract task for each task type to separate them by modality. For example, AbsClassification could define a evaluator property and an evaluate method, since all classification tasks likely use a train/test split. Then, specific task classes like TextClassification and ImageClassification would only need to implement _evaluate and _calculate_descriptive_metrics. I think this can be easily done for now in v2 as step one for unifying.

If some evaluators can be fully merged and reused—like Any2AnyRetrievalEvaluator—we could keep just one, but I haven’t checked its code yet, so this is just speculation, but I don't think we can totally unify all of them, because it might be hard to maintain.

[isaac-chung]

Let's check the code first. Ideally I'd like to not have too many layers of abstraction to make it easier to understand.

In my head, e.g. a common AbsTaskAnyClassification could support both text and image columns, AND any other modalities that we would introduce, like audio.

Note that it is also possible for datasets to mix modalities, like image+text or image+audio.

Also any thoughts on discussing the encoder interface first?

[KennethEnevoldsen]

In my head, e.g. a common AbsTaskClassification could support both text and image columns AND any other modalities that we would introduce, like audio.

Yea, this is where I would want to end up.

Also any thoughts on discussing the encoder interface first?

Yep, could imagine that once it is settled that would make the rest quite simple to adapt

[isaac-chung]

Re: encoder interface, this might be a good place to start from.