---
license: apache-2.0
library_name: peft
pipeline_tag: feature-extraction
base_model:
- Qwen/Qwen2-VL-2B-Instruct
- Qwen/Qwen2-VL-7B-Instruct
- Qwen/Qwen3.5-2B
tags:
- multimodal-embedding
- universal-embedding
- image-text-retrieval
- contrastive-learning
- peft
- dora
- b3
- mmeb
- arxiv:2610.15035
---

# UME-Recipes: What Really Works for Universal Multimodal Embeddings?

[**Paper**](https://arxiv.org/abs/2610.15035) · [**Code and Recipes**](https://github.com/alibaba-multimodal-industrial-ai/UME-Recipes) · [**Alibaba Multimodal & Industrial AI Team**](https://huggingface.co/alibaba-multimodal-industrial-ai)

This repository provides the released models for **What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies**.

We systematically study representative contrastive training strategies under aligned backbones, training data, optimization budgets, and MMEB evaluation.

---

## Key findings

- **Controlled comparison:** Many sophisticated strategies provide only marginal gains over a properly aligned VLM2Vec baseline, and some degrade overall performance.
- **Gradient distortion:** Upweighting high-similarity negatives can hurt downstream generalization.
- **Distribution illusion:** A wider positive–negative similarity gap does not necessarily yield better retrieval performance.
- **Synergistic training:** Combining hard-negative mining, false-negative mitigation, and hard-batch construction achieves the strongest result, improving average MMEB performance by **6.0%** with Qwen2-VL-2B.

The paper also introduces an **MLLM-as-a-Judge** diagnostic framework that categorizes top-1 retrieval outcomes into labeled positives, false negatives, and hard or easy negative errors.

---

## Released models

We release the best-performing model for each evaluated backbone under our controlled training and evaluation setting. Across Qwen2-VL-2B, Qwen2-VL-7B, and Qwen3.5-2B, the strongest configuration is obtained with the B3 training recipe.

| Model | Base model | Training recipe | Adapter |
|---|---|---|---|
| [UME-B3-Qwen2-VL-2B](https://huggingface.co/alibaba-multimodal-industrial-ai/UME-Recipes/tree/main/UME-B3-Qwen2-VL-2B) | [Qwen2-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct) | B3 | DoRA, rank 16 |
| [UME-B3-Qwen2-VL-7B](https://huggingface.co/alibaba-multimodal-industrial-ai/UME-Recipes/tree/main/UME-B3-Qwen2-VL-7B) | [Qwen2-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct) | B3 | DoRA, rank 16 |
| [UME-B3-Qwen3.5-2B](https://huggingface.co/alibaba-multimodal-industrial-ai/UME-Recipes/tree/main/UME-B3-Qwen3.5-2B) | [Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) | B3 | DoRA, rank 16 |

The released artifacts are **PEFT DoRA adapters**, not standalone copies of their base models. Use them with the corresponding Qwen backbone and the model implementation in UME-Recipes. All models use last-token pooling followed by L2 normalization and were trained with an image pixel range of 4,096–2,257,920.

## MMEB results

Scores are mean Precision@1 (%) on the corresponding MMEB datasets. Overall is the average across all 36 image-task datasets.

| Released model | Image Classification (10) | Visual Question Answering (10) | Image-level Retrieval (12) | Visual Grounding (4) | Overall (36) |
|---|---:|---:|---:|---:|---:|
| UME-B3-Qwen2-VL-2B | 67.3 | 63.0 | 71.4 | 81.4 | **69.0** |
| UME-B3-Qwen2-VL-7B | 70.6 | 69.1 | 75.1 | 85.6 | **73.3** |
| UME-B3-Qwen3.5-2B | 66.9 | 67.2 | 70.0 | 86.5 | **70.2** |

---

## Download

Download one released adapter with the Hugging Face CLI:

```bash
hf download alibaba-multimodal-industrial-ai/UME-Recipes \
  --include "UME-B3-Qwen2-VL-2B/*" \
  --local-dir ./released-models
```

Replace `UME-B3-Qwen2-VL-2B` with another model directory from the table when needed.

## Evaluation

Install the official repository and run its MMEB evaluation scripts:

```bash
git clone https://github.com/alibaba-multimodal-industrial-ai/UME-Recipes.git
cd UME-Recipes
python -m pip install -e .

bash scripts/eval_all_qwen2_vl.sh \
  /path/to/Qwen2-VL-2B-Instruct \
  /path/to/released-models/UME-B3-Qwen2-VL-2B \
  ./exp/evaluation/UME-B3-Qwen2-VL-2B
```

For Qwen3.5, use `scripts/eval_all_qwen3_5.sh`. See the [GitHub repository](https://github.com/alibaba-multimodal-industrial-ai/UME-Recipes) for environment setup, training recipes, data layout, and complete evaluation instructions.

---

## License

The released adapters are provided under the Apache License 2.0. The corresponding base models, datasets, and evaluation resources remain subject to their original licenses and terms.

---

## Citation

If you find this work useful, please cite:

```bibtex
@article{li2026what,
  title={What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies},
  author={Li, Duo and Pan, Haojun and Yang, Zuhao and Wang, Xintong and Huo, Chengfu and Lu, Shijian and Ding, Liang},
  journal={arXiv preprint arXiv:2610.15035},
  year={2026}
}
```