UME-Recipes: What Really Works for Universal Multimodal Embeddings?

Paper · Code and Recipes · Alibaba Multimodal & Industrial AI Team

This repository provides the released models for What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies.

We systematically study representative contrastive training strategies under aligned backbones, training data, optimization budgets, and MMEB evaluation.

---

Key findings

• Controlled comparison: Many sophisticated strategies provide only marginal gains over a properly aligned VLM2Vec baseline, and some degrade overall performance.
• Gradient distortion: Upweighting high-similarity negatives can hurt downstream generalization.
• Distribution illusion: A wider positive–negative similarity gap does not necessarily yield better retrieval performance.
• Synergistic training: Combining hard-negative mining, false-negative mitigation, and hard-batch construction achieves the strongest result, improving average MMEB performance by 6.0% with Qwen2-VL-2B.

The paper also introduces an MLLM-as-a-Judge diagnostic framework that categorizes top-1 retrieval outcomes into labeled positives, false negatives, and hard or easy negative errors.

---

Released models

We release the best-performing model for each evaluated backbone under our controlled training and evaluation setting. Across Qwen2-VL-2B, Qwen2-VL-7B, and Qwen3.5-2B, the strongest configuration is obtained with the B3 training recipe.

Model	Base model	Training recipe	Adapter
UME-B3-Qwen2-VL-2B	Qwen2-VL-2B-Instruct	B3	DoRA, rank 16
UME-B3-Qwen2-VL-7B	Qwen2-VL-7B-Instruct	B3	DoRA, rank 16
UME-B3-Qwen3.5-2B	Qwen3.5-2B	B3	DoRA, rank 16

The released artifacts are PEFT DoRA adapters, not standalone copies of their base models. Use them with the corresponding Qwen backbone and the model implementation in UME-Recipes. All models use last-token pooling followed by L2 normalization and were trained with an image pixel range of 4,096–2,257,920.

MMEB results

Scores are mean Precision@1 (%) on the corresponding MMEB datasets. Overall is the average across all 36 image-task datasets.

Released model	Image Classification (10)	Visual Question Answering (10)	Image-level Retrieval (12)	Visual Grounding (4)	Overall (36)
UME-B3-Qwen2-VL-2B	67.3	63.0	71.4	81.4	69.0
UME-B3-Qwen2-VL-7B	70.6	69.1	75.1	85.6	73.3
UME-B3-Qwen3.5-2B	66.9	67.2	70.0	86.5	70.2

---

Download

Download one released adapter with the Hugging Face CLI:

hf download alibaba-multimodal-industrial-ai/UME-Recipes \
  --include "UME-B3-Qwen2-VL-2B/*" \
  --local-dir ./released-models

Replace UME-B3-Qwen2-VL-2B with another model directory from the table when needed.

Evaluation

Install the official repository and run its MMEB evaluation scripts:

git clone https://github.com/alibaba-multimodal-industrial-ai/UME-Recipes.git
cd UME-Recipes
python -m pip install -e .

bash scripts/eval_all_qwen2_vl.sh \
  /path/to/Qwen2-VL-2B-Instruct \
  /path/to/released-models/UME-B3-Qwen2-VL-2B \
  ./exp/evaluation/UME-B3-Qwen2-VL-2B

For Qwen3.5, use scripts/eval_all_qwen3_5.sh. See the GitHub repository for environment setup, training recipes, data layout, and complete evaluation instructions.

---

License

The released adapters are provided under the Apache License 2.0. The corresponding base models, datasets, and evaluation resources remain subject to their original licenses and terms.

---

Citation

If you find this work useful, please cite:

@misc{li2026reallyworksuniversalmultimodal,
  title={What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies},
  author={Duo Li and Haojun Pan and Zuhao Yang and Xintong Wang and Chengfu Huo and Shijian Lu and Liang Ding},
  year={2026},
  eprint={2610.15035},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2610.15035}
}