RESULTS.md

Complete experiment results

Bold entries in the summary tables mark the best measured value per column. All four real-data candidates use CR2/window8192 exact-budget prefill. Earlier smoke results use a different policy and are kept in a separate section.

Training summary

Candidate	Train BCE ↓	Validation BCE ↓	Validation accuracy ↑
32K epoch20	0.2659	0.4210	81.32%
64K epoch10 (released)	0.2976	0.4205	81.40%
64K epoch20	0.2723	0.4429	81.25%
64K epoch10 + 10 fresh	0.3343	0.3937	82.59%

Classifier validation contexts differ between 32K and 64K. Serving comparisons below share the same inputs.

---

Nanbeige context/depth comparison

All candidates use exact-budget CR2, window 8192 and the same primitive-ai FP8 model. The shared validation set has six real-data 64K contexts and 31 scored decode predictions per context. Final holdouts remain sealed.

Candidate	BF16 mean KL	BF16 agreement	FP8 mean KL	FP8 agreement	Worst KL BF16 / FP8
32K epoch20	.015638	96.24%	.030867	95.16%	.242 / 1.162
64K epoch10	.015563	97.85%	.022474	95.70%	.221 / .193
64K epoch20	.015922	96.77%	.020007	95.70%	.290 / .305

All candidates recovered every target token on all four synthetic retrieval cases under teacher forcing, with both KV dtypes. At 128K, BF16 decode was 16.79/16.81/16.87 tok/s and FP8 decode 20.33/20.39/20.47 tok/s in table order. Exact-budget live KV is 13.813 GiB BF16 or 6.906 GiB FP8 at 128K.

Selection: 64K epoch10, based on higher BF16 prediction agreement and lower worst per-token KL for both KV dtypes. Epoch20 has lower mean FP8 KL but a higher maximum. No final-set result was used in selection.

Raw results: artifacts/nanbeige-real/{validation,retrieval,perf}-*.json and artifacts/nanbeige-real64/epoch{10,20}-{validation,retrieval,perf}-*.json. Selected raw checkpoint: artifacts/nanbeige-real64/training/epoch-10.pt. Selected calibrated export: artifacts/nanbeige-real64/epoch10-export/final/.

---

Nanbeige progressive fresh-data results

The continuation completed ten epochs from the selected 64K epoch-10 raw checkpoint. Each epoch used 24 fresh 64K streams plus 24 replay streams; validation stayed fixed. Total fresh tokens: 15,728,640. Total training tokens processed across these epochs: 31,457,280. Capture plus fitting took about 3 hours 26 minutes. All ten completion records and source manifests were verified; fresh exact-text document hashes do not overlap earlier corpora or other refresh epochs.

Fresh activations were deleted after each successful epoch, after preserving source selections, shard hashes, metrics and checkpoints. Original replay captures remain. Fresh source token files and provenance remain under artifacts/nanbeige-progressive/epoch-*/. The continuation began with weights from epoch10 and fresh optimizer state; subsequent refresh epochs carried optimizer state forward. Each epoch's export was calibrated on that epoch's training rows only.

Heldout classification BCE changed from 0.420536 before continuation to 0.393684 at continuation epoch10; accuracy changed from 81.4026% to 82.5859%.

Shared development set

Six 64K source categories, 31 decode predictions per case. All comparisons use the same FP8 model weights and dense BF16 KV reference. KL is in nats.

Candidate	KV dtype	Mean KL	Top-1 agreement	Maximum per-token KL
Selected 64K epoch10	BF16	.015563	97.85%	.2212
Fresh continuation epoch10	BF16	.016123	96.24%	.1937
Selected 64K epoch10	FP8	.022474	95.70%	.1925
Fresh continuation epoch10	FP8	.027148	97.31%	.8485

Both candidates recover every target token in all four synthetic retrieval cases (32K/128K, early/late needle), with both KV dtypes under teacher forcing.

Previously sealed final set

Both candidate checkpoints were frozen before opening the six final 64K streams. No checkpoint or policy was changed using final results. The final set has 186 scored decode predictions. This is a bounded teacher-forced comparison, not free-generation task accuracy.

Configuration	Mean KL	Top-1 agreement	Maximum per-token KL
Dense BF16 KV reference	0	100%	0
Dense FP8 KV	.081011	94.09%	8.9380
Selected baseline, DMS BF16	.023749	94.09%	.6982
Continuation, DMS BF16	.029896	93.55%	1.6364
Selected baseline, DMS FP8	.062752	91.94%	4.6649
Continuation, DMS FP8	.040502	90.32%	.8821

The continuation reduced heldout classifier loss. Serving metrics are mixed: final BF16 mean KL increased and prediction agreement decreased; final FP8 mean and maximum KL decreased while prediction agreement decreased. These results do not support replacing the selected baseline with the continuation as an across-metric improvement. Both artifacts are retained; neither is labelled universally quality-qualified.

Continuation performance

Batch 1, eager, one warmup and two measured trials, 32 output tokens. Same performance prompt and exact-budget CR2/window8192 policy as the baseline.

Context	KV dtype	Live KV GiB	Peak allocated GiB	Decode tok/s
32K	BF16	4.063	44.30	30.47
32K	FP8	2.031	42.26	31.84
128K	BF16	13.813	72.94	16.80
128K	FP8	6.906	66.02	20.35

Artifacts

• Baseline: artifacts/nanbeige-real64/epoch10-export/final/.
• Continuation: artifacts/nanbeige-progressive/epoch-10/training/final/.
• Per-epoch source and capture manifests, resume checkpoints and classifier histories: artifacts/nanbeige-progressive/epoch-*/.
• Development quality/performance: artifacts/nanbeige-progressive/{validation,retrieval,perf}-*.json.
• Final-set results and reference logits: artifacts/nanbeige-final/.
• Driver: training/progressive_sidecar.py; data builder: training/prepare_real_data.py; capture: training/capture_real_sidecar.py.

Training was at 32K and 64K. 128K serving was tested, but no 128K training run was performed. Capture uses query stride16 and 1024 sampled eligible rows per layer/stream, recorded in manifests. Source separation checks exact text hashes; it is not a near-duplicate decontamination claim.

---

Nanbeige long-context DMS memory and KV quality

Measured 2026-10-26 on RTX PRO 6000 Blackwell 96 GB GPUs in Mamba fastdms. All runs use primitive-ai/Nanbeige4.3-33B-FP8 revision 0140c157bc44507929a742c220865b0870052552. DMS uses the two-update smoke sidecar at artifacts/nanbeige-smoke/trained512/dms_metadata.json (window 64).

Result: memory savings are real, but this DMS sidecar fails long-context retrieval with either BF16 or FP8 KV. Do not use it as a quality-qualified policy.

Performance and memory

Batch 1, eager execution, 32 output tokens; one warmup and two measured trials. Median speeds below use a repeated explanatory-text prompt. Live KV payload is measured immediately after prefill, before freeing the request. Total peak is PyTorch allocated memory during generation; it includes temporary activations but excludes model-loading transients and CUDA allocator reserves.

Input tokens	DMS KV dtype	Live KV GiB	Dense same-dtype KV GiB (calculated)	KV saved GiB	Total peak GiB	Prefill tok/s	Decode tok/s
32,768	BF16	2.031	6.500	4.469	42.27	4,140	31.67
32,768	FP8	1.015	3.250	2.235	41.25	3,909	32.37
131,072	BF16	8.686	26.000	17.314	67.81	1,990	19.24
131,072	FP8	4.343	13.000	8.657	63.46	1,993	21.09

Retention is 31.24% at 32K and 33.41% at 128K, identical for the two KV dtypes on this prompt. Actual compact arena allocations are slightly larger than live payload: 2.040/1.020 GiB at 32K and 8.695/4.348 GiB at 128K (BF16/FP8). DMS FP8 KV saves 21.657 GiB at 128K relative to dense BF16 KV payload. These are prompt-dependent retention measurements, not a guaranteed compression ratio. The configured CR2 training label target does not constrain inference to exactly 2x retention compression.

The first dense BF16 quality reference exhausted memory at 128K while converting FP8 linear inputs, despite its model and cache fitting. _fp8_linear now bounds activation conversion to 4,096-row chunks. Per-token scaling is unchanged. The updated path passes a long-row regression and the full 46-test suite. Performance was rerun after this fix: use chunked-dms-*.json, not the earlier unprefixed artifacts. The earlier DMS 128K peaks were 79.08/74.73 GiB; the final measurements above supersede them.

Quality comparison

All four variants use the same FP8 weights and exact same token inputs. Dense BF16 KV is the reference. Tests consist of three synthetic, training-disjoint 512-token contexts (science, code, Japanese) with fixed continuations and four retrieval contexts (32K/128K, answer at 10%/80% of the context). The retrieval answer is the six-digit code 739281.

The harness feeds the same target token after each prediction: these are teacher-forced metrics, not free-generation task scores. Prefill is dense in all variants, so the first prediction is excluded from aggregate KL and agreement below. Only subsequent predictions exercise retained/quantized KV. The short-context aggregate has 79 decode positions; retrieval has 24.

KL divergence measures changes in the whole next-token distribution (lower is closer to reference); top-1 agreement measures identical most-likely tokens. Neither alone proves general answer quality. Aggregates are weighted by scored decode positions, not by case.

KV variation	Short-context mean KL	Short-context top-1 agreement	Retrieval mean KL	Retrieval top-1 agreement	Retrieval cases with every target token correct
Dense BF16	0	100%	0	100%	4/4
Dense FP8	0.00827	92.4%	0.000192	100%	4/4
DMS BF16	0.09164	82.3%	1.4481	62.5%	0/4
DMS FP8	0.08718	87.3%	1.5092	62.5%	0/4

DMS retrieval target-token accuracy, including the initial dense prediction, was 57.1% for 32K/10% and 71.4% for the other three cases, with either KV dtype. Dense variants achieved 100% on all four. DMS retrieval target negative log-likelihood rose to 1.14–1.40 nats/token versus 0.008–0.028 for the reference. No predetermined statistical quality threshold was used; failing exact recovery of the code in all four cases is sufficient to reject this smoke sidecar for this retrieval use. Small differences between DMS dtypes on the short suite do not establish that FP8 improves quality.

The near-identical retrieval failures in DMS BF16 and DMS FP8 implicate eviction, not FP8 cache storage alone. This does not isolate poor training from transfer of a BF16-trained sidecar onto FP8 weights. Dense FP8 looks promising on this bounded suite but is not certified across tasks or longer generations.

Reproduction and artifacts

Performance command (repeat with both KV dtypes and both lengths):

PYTHONNOUSERSITE=1 CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=4 FASTDMS_DIST_PORT=2360 \
mamba run -n fastdms python -m benchmarks.bench_nanbeige_precision \
  --model primitive-ai/Nanbeige4.3-33B-FP8 \
  --metadata artifacts/nanbeige-smoke/trained512/dms_metadata.json \
  --kv-dtype bfloat16 --lengths 131072 --decode 32 --repeats 2 \
  --output artifacts/nanbeige-long/chunked-dms-bfloat16-131072.json

Quality reference:

PYTHONNOUSERSITE=1 CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=4 FASTDMS_DIST_PORT=2360 \
mamba run -n fastdms python -m benchmarks.check_nanbeige_kv_quality \
  --model primitive-ai/Nanbeige4.3-33B-FP8 \
  --reference artifacts/nanbeige-quality/reference --write-reference \
  --output artifacts/nanbeige-quality/dense-bfloat16.json

For candidates omit --write-reference, select --kv-dtype float8_e4m3fn for FP8 KV, and add the sidecar --metadata argument for DMS. Use a different output file for each configuration. References store float32 logits and a SHA-256 of the prompt/target token IDs; candidates reject input identity mismatches.

Raw performance: artifacts/nanbeige-long/chunked-dms-*.json. Raw quality: artifacts/nanbeige-quality/*.json, seven cases per variant. The quality suite completed 28 case/configuration combinations. GPUs 2/3 ran unrelated workloads; clocks were not locked. Small speed differences should not be treated as hardware-isolated measurements.

---

Nanbeige BF16 / FP8 speed and memory comparison

Measured 2026-10-26 on NVIDIA RTX PRO 6000 Blackwell 96 GB GPUs, using the fastdms Mamba environment (PyTorch 2.11.0+cu130). All eight configurations completed. The regression suite passed 45 tests.

Results at 4,096 input tokens

Batch size 1, 64 output tokens, eager execution, prefix cache cleared before each trial. Speeds are medians of three trials after one warmup. Prefill includes the first generated token; decode measures the remaining 63. Memory is the maximum PyTorch allocated peak across those trials, excluding model-load transients.

Model weights	Retention	KV storage	Prefill tok/s	Decode tok/s	Peak allocated GiB
BF16	Dense	BF16	4,421	23.29	63.64
BF16	Dense	FP8	4,245	21.18	63.21
BF16	DMS	BF16	4,197	22.88	63.06
BF16	DMS	FP8	4,094	22.97	62.93
primitive-ai FP8	Dense	BF16	6,949	39.58	35.50
primitive-ai FP8	Dense	FP8	6,836	29.42	35.07
primitive-ai FP8	DMS	BF16	6,686	37.08	34.92
primitive-ai FP8	DMS	FP8	6,571	35.94	34.80

With dense BF16 KV, FP8 weights improve prefill by 1.57x and decode by 1.70x; peak allocated memory falls by 44.2%. Resident model tensors, including model buffers, decrease from 61.94 to 33.34 GiB. Embeddings and the output head remain BF16 in the primitive-ai checkpoint.

DMS is not faster at this length: with FP8 weights/BF16 KV it reduces peak memory by about 0.58 GiB but lowers decode speed by 6.3%. The dense FP8 KV path converts cache blocks to BF16 for FlashAttention on every decode step, explaining why storage savings do not imply a speedup. Compact DMS attention consumes FP8 KV without that dense conversion path.

Results at 512 input tokens

Model weights	Retention	KV storage	Prefill tok/s	Decode tok/s	Peak allocated GiB
BF16	Dense	BF16	4,523	23.63	62.96
BF16	Dense	FP8	4,817	21.24	62.53
BF16	DMS	BF16	3,814	23.14	62.17
BF16	DMS	FP8	3,647	23.13	62.14
primitive-ai FP8	Dense	BF16	7,186	40.74	34.50
primitive-ai FP8	Dense	FP8	7,149	29.85	34.07
primitive-ai FP8	DMS	BF16	5,338	36.47	33.71
primitive-ai FP8	DMS	FP8	5,303	36.05	33.68

Dense cache capacity is fixed at the larger benchmark limit (4,096 + 64 tokens, rounded to 256-token pages), including for the 512-token row. DMS uses lazy per-layer allocation. These are allocated-memory comparisons, not comparisons of live KV bytes alone. Raw results also include reserved and device-used bytes; those are not interchangeable with allocated memory.

Reproduction

The implementation is benchmarks/bench_nanbeige_precision.py. Run each row in a fresh process. Use distinct FASTDMS_DIST_PORT values for simultaneous runs. The benchmark disables compact CUDA graphs and the fused greedy generation loop so prefill/decode timing is split consistently.

mamba activate fastdms
export PYTHONNOUSERSITE=1 OMP_NUM_THREADS=4
CUDA_VISIBLE_DEVICES=0 FASTDMS_DIST_PORT=2340 \
python -m benchmarks.bench_nanbeige_precision \
    --model primitive-ai/Nanbeige4.3-33B-FP8 \
    --kv-dtype bfloat16 \
    --output artifacts/nanbeige-precision/fp8-dense-bfloat16.json

For BF16 weights use --model Nanbeige/Nanbeige4.3-33B. For FP8 KV use --kv-dtype float8_e4m3fn. For DMS add --metadata artifacts/nanbeige-smoke/trained512/dms_metadata.json. Defaults are --lengths 512 4096 --decode 64 --repeats 3.

Exact local model revisions:

• BF16: c1093c501cc6460427c0ac634db7223d0450c0ee.
• primitive-ai FP8: 0140c157bc44507929a742c220865b0870052552.

Raw results: artifacts/nanbeige-precision/{bf16,fp8}-{dense,dms}-{bfloat16,float8_e4m3fn}.json. BF16 ran on GPU 1, FP8 on GPU 0, concurrently. Both are the same GPU model; clocks/power were not locked. Other workloads occupied GPUs 2 and 3. Treat small percentage differences as provisional, not hardware-isolated performance claims.

Scope and limitations

The FP8 loader preserves the checkpoint's channel-wise E4M3 weights and scales and uses dynamic per-token FP8 activations. It does not dequantize/requantize the checkpoint. Tests cover packed QKV values, scales, linear outputs and avoiding accidental re-quantization. A prefix-cache scheduling defect discovered during benchmark preparation was fixed and regression-tested.

These runs measure inference, not FP8 training. The DMS sidecar received only two smoke-training updates on repository text. The prompt is repeated explanatory text, not a task-quality suite. No claim is made about retained answer quality, long-context accuracy, quantized-model sidecar transfer quality, CUDA-graph throughput, concurrent serving, or maximum supported context. The sidecar's fingerprint binds the BF16 base, not the FP8 checkpoint. Do not promote this sidecar on speed/memory evidence alone.

For the measured workloads, FP8 weights with dense BF16 KV provide the best speed/memory tradeoff. DMS and FP8 KV should be reconsidered at longer contexts only after the sidecar passes quality checks.