TakTak/qwen3-vl-8b-mathmm-tradeoff-research-lora

Qwen3-VL-8B Math-MM Trade-off Research LoRA

Important: research artifact, not a general upgrade

This adapter is released as a category-dependent trade-off research artifact. It improves one Korean image-math evaluation set while significantly regressing on two others. It is not recommended as a drop-in replacement for the Base model and must not be described as a general math specialist.

Its purpose in this release is to make the observed trade-off reproducible and to support research on explicit Base/adapter selection rules.

Model details

• Base model: Qwen/Qwen3-VL-8B-Thinking
• Base revision: 92f3c4b4feadd3a016ef468d103bb5f58b2a2c6b
• Image-trace teacher: Qwen/Qwen3-VL-32B-Thinking
• Teacher revision: 7edd10ffd1196091948fb245ff63e406ccb2d4d1
• Adapter: PEFT LoRA, rank 32, alpha 64, dropout 0.05
• Target modules: q/k/v/o and gate/up/down projections
• Adapter weight SHA-256: 5ecf27b144369a260fe5f1aa49bfa095c27f879b9302dbcf266ee2721b4d11bb

Training construction

The training mixture combined Korean text-math solutions with answer-verified multimodal teacher traces.

1. Korean text-math rows were prepared from ChuGyouk/AI-MO-NuminaMath-CoT-Ko. Rows required an extractable boxed answer and a minimum solution length. A deterministic 12,000-row subset was selected.
2. The 32B teacher generated Korean reasoning traces for MMK12 image-math rows. Only 568 teacher outputs scored correct and containing an image were retained.
3. The 568 multimodal rows were repeated twice before a deterministic 1% validation split, yielding 13,005 training rows and 131 validation rows.

The final training mixture contained 11,883 text-only rows and 1,122 single-image rows. Source rows and images are not redistributed.

Training procedure

• Epochs: 2
• Learning rate: 1e-4
• Precision: BF16
• Per-device batch size: 1
• Gradient accumulation: 8
• Maximum sequence length: 4,096
• Maximum image long side: 896 pixels
• Seed: 1234
• Hardware: 8 GPUs
• Completed steps: 408/408
• Recorded training loss: 0.4021

Canonical paired evaluation

Base and adapter arms used identical inputs, prompts, greedy decoding, 4,096-token effective output budget, scorer, and UID-aligned paired tests. All three comparisons survived family-wise correction. Answer type below means the expected final-answer form (for example number, expression, or text); it is not a school subject label.

Evaluation set	N	Base	Adapter	Delta	95% paired bootstrap CI	Exact McNemar p	Data-overlap status
AIHub15	500	65.4%	76.4%	+11.0pp	[+6.6, +15.4]pp	2.06e-06	Leakage-clean on four audited axes
MMK12 evaluation set	300	59.0%	40.3%	-18.7pp	[-25.0, -12.3]pp	2.28e-08	Not held out for this adapter: 118/300 rows overlap training; clean-row delta remains -18.7pp
AIHub17	500	77.0%	66.8%	-10.2pp	[-14.6, -5.8]pp	1.07e-04	Leakage-clean on four audited axes

The AIHub15 gain is concentrated in expression- and text-answer rows, while the regressions are concentrated in number-answer rows. This pattern is why all three results must be reported together.

Rule-based selection pilot

A pre-registered CPU-only pilot used existing scored rows and selected among Base, the primary Distill32 adapter, and this adapter using only evaluation set identity and answer type. On a deterministic held-out half (n=614), the rule kept Base for number-answer rows and used this adapter only in selected AIHub15 expression/text cells.

• Always Base: 66.4%
• Rule-selected arm: 71.5%
• Difference: +5.05pp, 95% CI [+2.6, +7.7]pp
• Exact McNemar p: 1.17e-04

This is a retrospective viability pilot, not a learned router, deployed ensemble, or proof of automatic routing performance.

Intended use

• Reproduce the measured category-dependent trade-off.
• Study answer-type-aware Base/adapter selection with a safe Base fallback.
• Analyze multimodal teacher-trace plus Korean text-math mixtures.

Limitations and prohibited interpretations

• Do not present this adapter as a general improvement over Base.
• Do not omit the MMK12 and AIHub17 regressions when citing the AIHub15 gain.
• Do not call the MMK12 set held out for this adapter.
• Do not claim a completed specialist, learned router, deployed ensemble, or broad Korean math/science generalization.
• The rule pilot uses known evaluation-set identity and answer type; a real pre-inference router was not trained or validated.
• Safety, fairness, and high-stakes educational use were not evaluated.

Loading

from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

base_id = "Qwen/Qwen3-VL-8B-Thinking"
base_revision = "92f3c4b4feadd3a016ef468d103bb5f58b2a2c6b"
adapter_id = "TakTak/qwen3-vl-8b-mathmm-tradeoff-research-lora"

processor = AutoProcessor.from_pretrained(base_id, revision=base_revision)
base = AutoModelForImageTextToText.from_pretrained(
    base_id,
    revision=base_revision,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base, adapter_id)

License and upstream sources

The adapter is released under CC BY-NC 4.0. The Korean text-math source is CC-BY-NC-4.0. The pinned Qwen models are Apache-2.0, and the pinned MMK12 source snapshot contains an Apache-2.0 license file. No raw data, images, teacher completions, evaluation rows, predictions, or scored row-level outputs are included.

• Base: https://huggingface.co/Qwen/Qwen3-VL-8B-Thinking
• Teacher: https://huggingface.co/Qwen/Qwen3-VL-32B-Thinking
• Text-math source: https://huggingface.co/datasets/ChuGyouk/AI-MO-NuminaMath-CoT-Ko
• Image-math source: https://huggingface.co/datasets/FanqingM/MMK12