Qwen3.8-Flash-Next-NVFP4-lean

The nvidia/Qwen3.8-Flash-Next-NVFP4 checkpoint with an overthinking-marker logit penalty baked into the weights (method of arXiv:2606.00206). No server flags, no per-request args, no client loop — serve it exactly like the upstream model and it thinks measurably less while scoring at parity or better. Built and measured on a DGX Spark (vLLM, TP1 + PLE offload) by Olyno / MiaAI-Lab, Sep 2026.

Drop-in: vllm serve /path/to/Qwen3.8-Flash-Next-NVFP4-lean --served-model-name qwen3.8-lean (plus your usual engine args). Architecture, NVFP4 quantization, PLE, MTP head, tokenizer and chat template are untouched; only 49 lm_head rows differ.

Headline numbers (baked vs base checkpoint)

Greedy, seed-fixed, strict \boxed scoring; OT% = overthinking-error rate (wrong answer whose gold already appeared in its own trace); trunc% = runs hitting the 16k cap.

Dataset	Accuracy (base → lean)	Mean thinking tokens (base → lean)	OT% (base → lean)	Trunc% (base → lean)
GPQA-Diamond (n=198)	74.2 → 82.8 (+8.6 pp, McNemar p=.002)	8597 → 7220 (−16.0%)	24.7 → 14.6	30.8 → 21.2
MATH-500 (n=500)	88.8 → 88.4 (−0.4 pp, n.s., p=.73)	1728 → 1562 (−9.6%)	3.0 → 3.4	1.2 → 1.4
GSM8K (n=100)	96.0 → 97.0 (+1.0 pp, n.s.)	414 → 462 (+11.6%)*	3.0 → 1.0	0.0 → 0.0

* known GSM8K length wart of the shipped bake; with the optional Chain-of-Draft prompt file the combo lands at 96.0 acc = base at −20.8% tokens (see Composition).

Optional prompt-plane overlay (baked weights × Chain-of-Draft 8-shot, chat_template_cod.jinja in the serving repo — arXiv:2502.18600):

Combo (weights+CoD prompt)	Accuracy	Δ tokens
GPQA-Diamond	86.4% (+12.2 pp vs base, p<.001)	−33.6%
MATH-500	88.7 (= base)	−31.5%
GSM8K	96.0 (= base)	−20.8%

What was done, in order

1. Runtime λ sweep first, bake second. Paper Eq. 1 (z_v ← z_v − λ on marker logits) applied as per-request logit_bias on the stock checkpoint: base + λ∈{0.5,1,2,4} × {math500, gsm8k100, gpqa200}, 15 runs, greedy, matched-pair McNemar stats.
2. λ = 2.0 selected by a pre-registered rule — accuracy-neutral (Δacc ≥ −1.0 pp per dataset) → max ΣCoT reduction → prefer smallest λ. (Mean Δacc +2.76 pp, mean Δtok −10.5%.)
3. Calibration harvest: a CALIB serve with a dump hook on compute_logits captured 428,957 real decode hidden-state rows (final-norm outputs, 3×40 problems).
4. The weight edit (bake_lm_head.py): ParallelLMHead is bias=False, so the per-step penalty reduces to a row shift. A minimum-norm ridge least-squares delta d = −λ·(G + tεI)⁻¹·E[h], G = E[h hᵀ], fitted on the harvested rows, guarantees mean(h·d) = −λ on real decode states: achieved mean shift −2.0 nats, bf16-truthful spread σ = 0.198. Shard surgery rewrote only the 49 marker rows of lm_head.weight (in model-00009-of-00010.safetensors, bf16 unquantized head); every other byte of every tensor is bit-identical to the NVIDIA snapshot (per-file verification: SHA256MANIFEST.json).
5. generation_config.json carries logit_bias {"<id>": −2.0} — exact on HF Transformers stacks; vLLM ignores that key, so on vLLM the baked rows are the operative change (no double-penalty).
6. Verification gates: static fit check + all-tensors-bit-identical check; two-dump logit equivalence (base-server-with-bias ≡ baked-server-without, PENALTY-EQUIVALENCE PASS); full 3-dataset re-benchmark of the baked folder; 14/14 acceptance gates, 34/34 packaging hashes (PACKAGING.sha256).

The marker set

marker_token_map.json holds the token ids. 50 candidates from arXiv:2606.00206 Table 8 (hedging/revisal unigrams: revisit, whether, however, hmm, reconsider, …), resolved against this model's tokenizer (vocab 248,044): 46/50 are single tokens, 3 adapted to their leading-space form, recheck dropped (multi-token). The penalty lands on all 49; a data-grounded refinement (enrichment of marker TF in overthinking-error vs correct traces, n=754) found 37 surfaces keeping 89.1% doc coverage — it tied on math/gsm but lost on GPQA (79.8 vs 82.8), so the shipped map is the original 49 (MAP-CHOICE=original). λ, not the token list, sets the operating point: λ=1.0 under-suppresses, λ=2.0 is the sweet spot, λ=4.0 hits a coherence cliff (GPQA failures at 2×λ* were all 16k-cap rambles).

Context: the full method matrix this bake survived

The bake is the shipped arm of a 28-arm study (STUDY_REPORT.md). Arms, in brief: P1 = this marker penalty (logit plane → weight bake, arXiv:2606.00206) · P2 = Chain-of-Draft prompt (arXiv:2502.18600) · P3 = DEER confidence-gated exit (arXiv:2504.15895) · P4 = REFRAIN redundancy-gated stop (ACL 2026 long.1256). Cross-arm findings that motivated shipping P1 as a weight artifact:

• Every inference-time gate (DEER, REFRAIN, single-shot) either paid tokens without accuracy or collapsed on hard sets (REFRAIN math −18.2 pp). Weight-level and config-level suppression generalizes; sampling-gate suppression does not — single-arm and pair-level evidence.
• The runtime penalty is not equivalent to the bake: at λ=4 applied at sampling time GPQA derails; the baked fold keeps its official +8.6 pp. The bake calibrates the penalty on the contexts where markers actually fire.
• The win lives on hard sets: token cuts are universal (−9.6% to −62% across arms), accuracy gains concentrate on GPQA. Base-checkpoint diagnostic: wrong answers run 3.0× (math), 5.8× (GSM8K), 2.5× (GPQA) longer than correct ones; the GPQA wrong-answer median sits at the 16k generation cap.

Limits disclosed: the +11.6% GSM8K token wart of the bake (neutralized by the CoD overlay); MCQ ot_proxy inflation on GPQA (strict committed-answer measure 5/51, reported in the study); no claim exceeds its arm's dataset scale.

File inventory / repo structure

Everything the upstream folder had, unchanged in name and bytes, plus study provenance:

README.md                    this card (packaged README superseded; its sha256 e56b4bd5… lives in PACKAGING.sha256)
config.json                  architecture, unmodified
generation_config.json       + logit_bias {marker_id: -2.0}  (HF stacks; vLLM ignores)
hf_quant_config.json         NVFP4 config, unmodified
model-0000{1..10}-of-00010.safetensors   ~84 GiB weights — ONLY 09 differs from upstream (49 lm_head rows)
model-fp8-mtp-ple.safetensors            FP8 MTP draft head + PLE side file (unmodified)
model.safetensors.index.json tokenizer.json tokenizer_config.json vocab.json merges.txt
chat_template.jinja          stock NVIDIA template (CoD variant lives in the serving repo)
preprocessor_config.json  processor_config.json  video_preprocessor_config.json
overthinking_bake.json       bake manifest: λ, 49 ids, target/achieved shift, calib rows
marker_token_map.json        marker surface ↔ id map + adaptation notes
STUDY_REPORT.md              full 28-arm matrix, methodology, verdicts
benchmark_final_table.md     machine-generated final table
bake_selection_analysis.json per-arm selection stats
lean_serving.patch           single-/dual-DGX-Spark serving recipes (MiaAI-Lab repo)
SHA256MANIFEST.json          per-tensor sha256 of every file except the edited lm_head rows
PACKAGING.sha256             sha256 of every packaged file (see Verify below)
.gitattributes               LFS tracking

Verify: sha256sum -c --ignore-missing PACKAGING.sha256 — every present file passes except README.md (this card). Four packaged entries are not published and report as missing: the old README.md (sha256 e56b4bd5…, superseded by this card), MODEL_CARD.md, REPRODUCTION.md and EXPERIMENT_LOG.md. For tensor-level provenance use SHA256MANIFEST.json: it names every upstream tensor bit-identical, with the 49 marker rows of lm_head.weight as the declared sole delta.

License / intended use

Same license and usage restrictions as the upstream nvidia/Qwen3.8-Flash-Next-NVFP4 (NVIDIA Open Model License + Qwen Community License 1.0); this repo redistributes the checkpoint with a 49-row delta and research artifacts. Intended use: reasoning-heavy math/science served on vLLM-compatible stacks. No safety behavior was removed or added; safety/refusal behavior is whatever the upstream checkpoint exhibits. The lab scripts and raw per-arm logs behind this artifact are not part of this repo; the published STUDY_REPORT.md carries their full result record.