Qwen3.8-Flash-Next-FP4-Local-NVFP4 (TP=8, vision-enabled)

A lossless, representation-only re-packaging of nvidia/Qwen3.8-Flash-Next-FP4-Local, produced so the checkpoint can be served with tensor parallelism (TP=8) on eight GPUs, including the vision tower (image-text-to-text). No weights were re-quantized: the FP4 (E2M1) values are byte-identical to the source checkpoint.

What was changed (and why)

The source checkpoint stores its routed-expert scale factors in NVIDIA's new lossless FP4-CSF compressed format, its text projections in MXFP8, and its vision-encoder projections in MXFP8 (attention) and W4A16_NVFP4 (MLP/merger). With the shipped kernel, several of these can only be decoded on a tensor-parallel path that requires each per-rank slice to satisfy a kernel alignment constraint, which this model violates at TP=8. Purely representational conversions were applied to remove those constraints:

1. FP4-CSF expert scales → plain NVFP4 block scales (E4M3, block size 16). The compressed scale planes were decompressed back to their original per-16 block scales. This is the reversible inverse of the lossless CSF compression: the decoded scale values are identical to the source, so the effective weights are unchanged. Because plain NVFP4 uses block size 16, the per-rank expert width 640 / 8 = 80 (80 % 16 == 0) now works at TP=8.
2. Text MXFP8 projections → BF16 (exact). 384 text-layer MXFP8 projections were expanded to their exactly-representable BF16 values (max_abs_error = 0 against the float32 reconstruction). MXFP8 has a 32-block input-width constraint that 80 violates at TP=8; the BF16 expansion removes it.
3. Vision-encoder projections → BF16 (exact). All 110 vision quantization projections were expanded to BF16: 54 MXFP8 (attention qkv/proj, input width 1152 / 8 = 144, not a multiple of 32) and 56 W4A16_NVFP4 (MLP/merger, input width 4304 / 8 = 538, not a multiple of 16 → input feature size is not a multiple of 16). MXFP8 is reversible (max_abs_error = 0); the NVFP4 conversion was verified against the per-block max. This makes the vision tower loadable at TP=8, which previously failed. (In a prior text-only release these projections were left quantized, so --language-model-only was required.)

Everything else is untouched: routed-expert FP4 weights, the MTP head, the PLE / n-gram embedding table (CPU-offloaded as in the source), the rest of the vision tower, tokenizer, and the original scalar weight_scale_2 / input_scale calibration values.

Machine-readable conversion records: adaptation.json (steps 1–2) and adaptation-vision.json (step 3).

Note on "lossless": the conversions are lossless in value (the numbers the kernels multiply are the same), but the storage precision of the converted tensors changes (compressed→plain scales; MXFP8/NVFP4→BF16). They are not byte-wise invertible, and the FP4-CSF bytes themselves are not recoverable from this repo.

Accuracy

Because the converted tensors hold the same values as the source, this checkpoint is expected to match nvidia/Qwen3.8-Flash-Next-FP4-Local at the same precision. Independent, task-level evaluation has not been run for this re-packaged copy; only the exactness of each conversion step was verified (max_abs_error = 0 for every converted tensor, with per-block max checks for the NVFP4 vision projections).

Performance (reference, TP=8, vision enabled)

Measured on this re-packaged checkpoint with the configuration below. Numbers are informal single-run figures on one specific machine and are not a guarantee.

Hardware: 8 x NVIDIA RTX PRO 2000 Blackwell (16 GB each). Serving: vLLM (community local-inference-lab Blackwell build), --tensor-parallel-size 8 --enable-expert-parallel, MTP speculative decoding (num_speculative_tokens=5), FP8 KV cache, --max-model-len 262144, --gpu-memory-utilization 0.95, chunked prefill.

Metric	Result
Single-stream decode (256 output tokens, code prompt)	~116 tok/s
Aggregate decode, 1 concurrent (128 tok each)	~103 tok/s
Aggregate decode, 2 concurrent	~158 tok/s
Aggregate decode, 4 concurrent	~257 tok/s
Aggregate decode, 8 concurrent	~457 tok/s
Single prefill, ~3,200-token prompt (warm)	~3,460 tok/s
KV cache (256K context)	263,637 tokens (1.01x)

Prefill note: earlier figures of ~6,700 tok/s were measured with a benchmark that reused an identical prompt and therefore benefited from prefix-cache hits. With a unique prompt per request (prefix cache defeated) the honest single-request prefill is ~3,460 tok/s at ~3,200 tokens. num_speculative_tokens=5 favors single-stream latency; 4 is marginally better for high-concurrency aggregate.

Usage (vLLM, TP=8)

Requires a vLLM build that reads NVIDIA ModelOpt FP4 checkpoints (e.g. the community local-inference-lab Blackwell images). Example, 8 GPUs, vision enabled, MTP speculative decoding, FP8 KV cache:

vllm serve <path-or-repo> \
  --served-model-name qwen38-flash-next-fp4 \
  --tensor-parallel-size 8 --enable-expert-parallel \
  --quantization modelopt \
  --max-model-len 262144 --max-num-seqs 8 \
  --gpu-memory-utilization 0.95 \
  --enable-chunked-prefill --max-num-batched-tokens 2048 \
  --kv-cache-dtype fp8 \
  --trust-remote-code \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}'

Add --language-model-only to serve text-only (skips the vision tower).

Architecture, context length (262,144 native), modalities and all model properties are inherited from the source model card.

Attribution and License

This is a derivative of nvidia/Qwen3.8-Flash-Next-FP4-Local, which is governed by the NVIDIA Open Model Agreement; the underlying base model Qwen/Qwen3.8-Flash-Next is governed by the Qwen Community License 1.0. The source model card, license and use terms apply to this re-packaged copy as well. NVIDIA and Qwen do not endorse or warrant this derivative.