# Complete experiment results

**Bold entries in the summary tables mark the best measured value per column.** All four real-data candidates use CR2/window8192 exact-budget prefill. Earlier smoke results use a different policy and are kept in a separate section.

## Training summary

| Candidate | Train BCE ↓ | Validation BCE ↓ | Validation accuracy ↑ |
|---|---:|---:|---:|
|32K epoch20|**0.2659**|0.4210|81.32%|
|64K epoch10 (released)|0.2976|0.4205|81.40%|
|64K epoch20|0.2723|0.4429|81.25%|
|64K epoch10 + 10 fresh|0.3343|**0.3937**|**82.59%**|

Classifier validation contexts differ between 32K and 64K. Serving comparisons below share the same inputs.


---

# Nanbeige context/depth comparison

All candidates use exact-budget CR2, window 8192 and the same primitive-ai FP8
model. The shared validation set has six real-data 64K contexts and 31 scored
decode predictions per context. Final holdouts remain sealed.

| Candidate | BF16 mean KL | BF16 agreement | FP8 mean KL | FP8 agreement | Worst KL BF16 / FP8 |
| --- | ---: | ---: | ---: | ---: | ---: |
| 32K epoch20 | .015638 | 96.24% | .030867 | 95.16% | .242 / 1.162 |
| 64K epoch10 | .015563 | 97.85% | .022474 | 95.70% | .221 / .193 |
| 64K epoch20 | .015922 | 96.77% | .020007 | 95.70% | .290 / .305 |

All candidates recovered every target token on all four synthetic retrieval
cases under teacher forcing, with both KV dtypes. At 128K, BF16 decode was
16.79/16.81/16.87 tok/s and FP8 decode 20.33/20.39/20.47 tok/s in table order.
Exact-budget live KV is 13.813 GiB BF16 or 6.906 GiB FP8 at 128K.

Selection: 64K epoch10, based on higher BF16 prediction agreement and lower
worst per-token KL for both KV dtypes. Epoch20 has lower mean FP8 KL but a
higher maximum. No final-set result was used in selection.

Raw results: `artifacts/nanbeige-real/{validation,retrieval,perf}-*.json` and
`artifacts/nanbeige-real64/epoch{10,20}-{validation,retrieval,perf}-*.json`.
Selected raw checkpoint: `artifacts/nanbeige-real64/training/epoch-10.pt`.
Selected calibrated export: `artifacts/nanbeige-real64/epoch10-export/final/`.

---

# Nanbeige progressive fresh-data results

The continuation completed ten epochs from the selected 64K epoch-10 raw
checkpoint. Each epoch used 24 fresh 64K streams plus 24 replay streams;
validation stayed fixed. Total fresh tokens: 15,728,640. Total training tokens
processed across these epochs: 31,457,280. Capture plus fitting took about
3 hours 26 minutes. All ten completion records and source manifests were
verified; fresh exact-text document hashes do not overlap earlier corpora or
other refresh epochs.

Fresh activations were deleted after each successful epoch, after preserving
source selections, shard hashes, metrics and checkpoints. Original replay
captures remain. Fresh source token files and provenance remain under
`artifacts/nanbeige-progressive/epoch-*/`. The continuation began with weights
from epoch10 and fresh optimizer state; subsequent refresh epochs carried
optimizer state forward. Each epoch's export was calibrated on that epoch's
training rows only.

Heldout classification BCE changed from 0.420536 before continuation to
0.393684 at continuation epoch10; accuracy changed from 81.4026% to 82.5859%.

## Shared development set

Six 64K source categories, 31 decode predictions per case. All comparisons use
the same FP8 model weights and dense BF16 KV reference. KL is in nats.

| Candidate | KV dtype | Mean KL | Top-1 agreement | Maximum per-token KL |
| --- | --- | ---: | ---: | ---: |
| Selected 64K epoch10 | BF16 | .015563 | 97.85% | .2212 |
| Fresh continuation epoch10 | BF16 | .016123 | 96.24% | .1937 |
| Selected 64K epoch10 | FP8 | .022474 | 95.70% | .1925 |
| Fresh continuation epoch10 | FP8 | .027148 | 97.31% | .8485 |

Both candidates recover every target token in all four synthetic retrieval
cases (32K/128K, early/late needle), with both KV dtypes under teacher forcing.

## Previously sealed final set

Both candidate checkpoints were frozen before opening the six final 64K
streams. No checkpoint or policy was changed using final results. The final
set has 186 scored decode predictions. This is a bounded teacher-forced
comparison, not free-generation task accuracy.

| Configuration | Mean KL | Top-1 agreement | Maximum per-token KL |
| --- | ---: | ---: | ---: |
| Dense BF16 KV reference | 0 | 100% | 0 |
| Dense FP8 KV | .081011 | 94.09% | 8.9380 |
| Selected baseline, DMS BF16 | .023749 | 94.09% | .6982 |
| Continuation, DMS BF16 | .029896 | 93.55% | 1.6364 |
| Selected baseline, DMS FP8 | .062752 | 91.94% | 4.6649 |
| Continuation, DMS FP8 | .040502 | 90.32% | .8821 |

The continuation reduced heldout classifier loss. Serving metrics are mixed:
final BF16 mean KL increased and prediction agreement decreased; final FP8 mean
and maximum KL decreased while prediction agreement decreased. These results
do not support replacing the selected baseline with the continuation as an
across-metric improvement. Both artifacts are retained; neither is labelled
universally quality-qualified.

## Continuation performance

Batch 1, eager, one warmup and two measured trials, 32 output tokens. Same
performance prompt and exact-budget CR2/window8192 policy as the baseline.

| Context | KV dtype | Live KV GiB | Peak allocated GiB | Decode tok/s |
| ---: | --- | ---: | ---: | ---: |
| 32K | BF16 | 4.063 | 44.30 | 30.47 |
| 32K | FP8 | 2.031 | 42.26 | 31.84 |
| 128K | BF16 | 13.813 | 72.94 | 16.80 |
| 128K | FP8 | 6.906 | 66.02 | 20.35 |

## Artifacts

- Baseline: `artifacts/nanbeige-real64/epoch10-export/final/`.
- Continuation: `artifacts/nanbeige-progressive/epoch-10/training/final/`.
- Per-epoch source and capture manifests, resume checkpoints and classifier
  histories: `artifacts/nanbeige-progressive/epoch-*/`.
- Development quality/performance: `artifacts/nanbeige-progressive/{validation,retrieval,perf}-*.json`.
- Final-set results and reference logits: `artifacts/nanbeige-final/`.
- Driver: `training/progressive_sidecar.py`; data builder:
  `training/prepare_real_data.py`; capture: `training/capture_real_sidecar.py`.

Training was at 32K and 64K. 128K serving was tested, but no 128K training run
was performed. Capture uses query stride16 and 1024 sampled eligible rows per
layer/stream, recorded in manifests. Source separation checks exact text hashes;
it is not a near-duplicate decontamination claim.

---

# Nanbeige long-context DMS memory and KV quality

Measured 2026-10-26 on RTX PRO 6000 Blackwell 96 GB GPUs in Mamba `fastdms`.
All runs use primitive-ai/Nanbeige4.3-33B-FP8 revision
`0140c157bc44507929a742c220865b0870052552`. DMS uses the two-update smoke sidecar
at `artifacts/nanbeige-smoke/trained512/dms_metadata.json` (window 64).

**Result: memory savings are real, but this DMS sidecar fails long-context
retrieval with either BF16 or FP8 KV. Do not use it as a quality-qualified policy.**

## Performance and memory

Batch 1, eager execution, 32 output tokens; one warmup and two measured trials.
Median speeds below use a repeated explanatory-text prompt. Live KV payload is
measured immediately after prefill, before freeing the request. Total peak is
PyTorch allocated memory during generation; it includes temporary activations
but excludes model-loading transients and CUDA allocator reserves.

| Input tokens | DMS KV dtype | Live KV GiB | Dense same-dtype KV GiB (calculated) | KV saved GiB | Total peak GiB | Prefill tok/s | Decode tok/s |
| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| 32,768 | BF16 | 2.031 | 6.500 | 4.469 | 42.27 | 4,140 | 31.67 |
| 32,768 | FP8 | 1.015 | 3.250 | 2.235 | 41.25 | 3,909 | 32.37 |
| 131,072 | BF16 | 8.686 | 26.000 | 17.314 | 67.81 | 1,990 | 19.24 |
| 131,072 | FP8 | 4.343 | 13.000 | 8.657 | 63.46 | 1,993 | 21.09 |

Retention is 31.24% at 32K and 33.41% at 128K, identical for the two KV dtypes
on this prompt. Actual compact arena allocations are slightly larger than live
payload: 2.040/1.020 GiB at 32K and 8.695/4.348 GiB at 128K (BF16/FP8).
DMS FP8 KV saves 21.657 GiB at 128K relative to dense BF16 KV payload.
These are prompt-dependent retention measurements, not a guaranteed compression
ratio. The configured CR2 training label target does not constrain inference to
exactly 2x retention compression.

The first dense BF16 quality reference exhausted memory at 128K while converting
FP8 linear inputs, despite its model and cache fitting. `_fp8_linear` now bounds
activation conversion to 4,096-row chunks. Per-token scaling is unchanged. The
updated path passes a long-row regression and the full 46-test suite. Performance
was rerun after this fix: use `chunked-dms-*.json`, not the earlier unprefixed
artifacts. The earlier DMS 128K peaks were 79.08/74.73 GiB; the final measurements
above supersede them.

## Quality comparison

All four variants use the same FP8 weights and exact same token inputs. Dense
BF16 KV is the reference. Tests consist of three synthetic, training-disjoint
512-token contexts (science, code, Japanese) with fixed continuations and four
retrieval contexts (32K/128K, answer at 10%/80% of the context). The retrieval
answer is the six-digit code `739281`.

The harness feeds the same target token after each prediction: these are
**teacher-forced** metrics, not free-generation task scores. Prefill is dense
in all variants, so the first prediction is excluded from aggregate KL and
agreement below. Only subsequent predictions exercise retained/quantized KV.
The short-context aggregate has 79 decode positions; retrieval has 24.

KL divergence measures changes in the whole next-token distribution (lower is
closer to reference); top-1 agreement measures identical most-likely tokens.
Neither alone proves general answer quality. Aggregates are weighted by scored
decode positions, not by case.

| KV variation | Short-context mean KL | Short-context top-1 agreement | Retrieval mean KL | Retrieval top-1 agreement | Retrieval cases with every target token correct |
| --- | ---: | ---: | ---: | ---: | ---: |
| Dense BF16 | 0 | 100% | 0 | 100% | 4/4 |
| Dense FP8 | 0.00827 | 92.4% | 0.000192 | 100% | 4/4 |
| DMS BF16 | 0.09164 | 82.3% | 1.4481 | 62.5% | 0/4 |
| DMS FP8 | 0.08718 | 87.3% | 1.5092 | 62.5% | 0/4 |

DMS retrieval target-token accuracy, including the initial dense prediction,
was 57.1% for 32K/10% and 71.4% for the other three cases, with either KV dtype.
Dense variants achieved 100% on all four. DMS retrieval target negative
log-likelihood rose to 1.14–1.40 nats/token versus 0.008–0.028 for the reference.
No predetermined statistical quality threshold was used; failing exact recovery
of the code in all four cases is sufficient to reject this smoke sidecar for
this retrieval use. Small differences between DMS dtypes on the short suite do
not establish that FP8 improves quality.

The near-identical retrieval failures in DMS BF16 and DMS FP8 implicate eviction,
not FP8 cache storage alone. This does not isolate poor training from transfer
of a BF16-trained sidecar onto FP8 weights. Dense FP8 looks promising on this
bounded suite but is not certified across tasks or longer generations.

## Reproduction and artifacts

Performance command (repeat with both KV dtypes and both lengths):

```bash
PYTHONNOUSERSITE=1 CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=4 FASTDMS_DIST_PORT=2360 \
mamba run -n fastdms python -m benchmarks.bench_nanbeige_precision \
  --model primitive-ai/Nanbeige4.3-33B-FP8 \
  --metadata artifacts/nanbeige-smoke/trained512/dms_metadata.json \
  --kv-dtype bfloat16 --lengths 131072 --decode 32 --repeats 2 \
  --output artifacts/nanbeige-long/chunked-dms-bfloat16-131072.json
```

Quality reference:

```bash
PYTHONNOUSERSITE=1 CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=4 FASTDMS_DIST_PORT=2360 \
mamba run -n fastdms python -m benchmarks.check_nanbeige_kv_quality \
  --model primitive-ai/Nanbeige4.3-33B-FP8 \
  --reference artifacts/nanbeige-quality/reference --write-reference \
  --output artifacts/nanbeige-quality/dense-bfloat16.json
```

For candidates omit `--write-reference`, select `--kv-dtype float8_e4m3fn` for
FP8 KV, and add the sidecar `--metadata` argument for DMS. Use a different output
file for each configuration. References store float32 logits and a SHA-256 of
the prompt/target token IDs; candidates reject input identity mismatches.

Raw performance: `artifacts/nanbeige-long/chunked-dms-*.json`.
Raw quality: `artifacts/nanbeige-quality/*.json`, seven cases per variant.
The quality suite completed 28 case/configuration combinations. GPUs 2/3 ran
unrelated workloads; clocks were not locked. Small speed differences should not
be treated as hardware-isolated measurements.

---

# Nanbeige BF16 / FP8 speed and memory comparison

Measured 2026-10-26 on NVIDIA RTX PRO 6000 Blackwell 96 GB GPUs, using the
`fastdms` Mamba environment (PyTorch 2.11.0+cu130). All eight configurations
completed. The regression suite passed 45 tests.

## Results at 4,096 input tokens

Batch size 1, 64 output tokens, eager execution, prefix cache cleared before each
trial. Speeds are medians of three trials after one warmup. Prefill includes the
first generated token; decode measures the remaining 63. Memory is the maximum
PyTorch allocated peak across those trials, excluding model-load transients.

| Model weights | Retention | KV storage | Prefill tok/s | Decode tok/s | Peak allocated GiB |
| --- | --- | --- | ---: | ---: | ---: |
| BF16 | Dense | BF16 | 4,421 | 23.29 | 63.64 |
| BF16 | Dense | FP8 | 4,245 | 21.18 | 63.21 |
| BF16 | DMS | BF16 | 4,197 | 22.88 | 63.06 |
| BF16 | DMS | FP8 | 4,094 | 22.97 | 62.93 |
| primitive-ai FP8 | Dense | BF16 | 6,949 | 39.58 | 35.50 |
| primitive-ai FP8 | Dense | FP8 | 6,836 | 29.42 | 35.07 |
| primitive-ai FP8 | DMS | BF16 | 6,686 | 37.08 | 34.92 |
| primitive-ai FP8 | DMS | FP8 | 6,571 | 35.94 | 34.80 |

With dense BF16 KV, FP8 weights improve prefill by 1.57x and decode by 1.70x;
peak allocated memory falls by 44.2%. Resident model tensors, including model
buffers, decrease from 61.94 to 33.34 GiB. Embeddings and the output head remain
BF16 in the primitive-ai checkpoint.

DMS is not faster at this length: with FP8 weights/BF16 KV it reduces peak memory
by about 0.58 GiB but lowers decode speed by 6.3%. The dense FP8 KV path converts
cache blocks to BF16 for FlashAttention on every decode step, explaining why
storage savings do not imply a speedup. Compact DMS attention consumes FP8 KV
without that dense conversion path.

## Results at 512 input tokens

| Model weights | Retention | KV storage | Prefill tok/s | Decode tok/s | Peak allocated GiB |
| --- | --- | --- | ---: | ---: | ---: |
| BF16 | Dense | BF16 | 4,523 | 23.63 | 62.96 |
| BF16 | Dense | FP8 | 4,817 | 21.24 | 62.53 |
| BF16 | DMS | BF16 | 3,814 | 23.14 | 62.17 |
| BF16 | DMS | FP8 | 3,647 | 23.13 | 62.14 |
| primitive-ai FP8 | Dense | BF16 | 7,186 | 40.74 | 34.50 |
| primitive-ai FP8 | Dense | FP8 | 7,149 | 29.85 | 34.07 |
| primitive-ai FP8 | DMS | BF16 | 5,338 | 36.47 | 33.71 |
| primitive-ai FP8 | DMS | FP8 | 5,303 | 36.05 | 33.68 |

Dense cache capacity is fixed at the larger benchmark limit (4,096 + 64 tokens,
rounded to 256-token pages), including for the 512-token row. DMS uses lazy
per-layer allocation. These are allocated-memory comparisons, not comparisons
of live KV bytes alone. Raw results also include reserved and device-used bytes;
those are not interchangeable with allocated memory.

## Reproduction

The implementation is `benchmarks/bench_nanbeige_precision.py`. Run each row in
a fresh process. Use distinct `FASTDMS_DIST_PORT` values for simultaneous runs.
The benchmark disables compact CUDA graphs and the fused greedy generation loop
so prefill/decode timing is split consistently.

```bash
mamba activate fastdms
export PYTHONNOUSERSITE=1 OMP_NUM_THREADS=4
CUDA_VISIBLE_DEVICES=0 FASTDMS_DIST_PORT=2340 \
python -m benchmarks.bench_nanbeige_precision \
    --model primitive-ai/Nanbeige4.3-33B-FP8 \
    --kv-dtype bfloat16 \
    --output artifacts/nanbeige-precision/fp8-dense-bfloat16.json
```

For BF16 weights use `--model Nanbeige/Nanbeige4.3-33B`. For FP8 KV use
`--kv-dtype float8_e4m3fn`. For DMS add
`--metadata artifacts/nanbeige-smoke/trained512/dms_metadata.json`.
Defaults are `--lengths 512 4096 --decode 64 --repeats 3`.

Exact local model revisions:

- BF16: `c1093c501cc6460427c0ac634db7223d0450c0ee`.
- primitive-ai FP8: `0140c157bc44507929a742c220865b0870052552`.

Raw results: `artifacts/nanbeige-precision/{bf16,fp8}-{dense,dms}-{bfloat16,float8_e4m3fn}.json`.
BF16 ran on GPU 1, FP8 on GPU 0, concurrently. Both are the same GPU model;
clocks/power were not locked. Other workloads occupied GPUs 2 and 3. Treat small
percentage differences as provisional, not hardware-isolated performance claims.

## Scope and limitations

The FP8 loader preserves the checkpoint's channel-wise E4M3 weights and scales
and uses dynamic per-token FP8 activations. It does not dequantize/requantize the
checkpoint. Tests cover packed QKV values, scales, linear outputs and avoiding
accidental re-quantization. A prefix-cache scheduling defect discovered during
benchmark preparation was fixed and regression-tested.

These runs measure inference, not FP8 training. The DMS sidecar received only two
smoke-training updates on repository text. The prompt is repeated explanatory
text, not a task-quality suite. No claim is made about retained answer quality,
long-context accuracy, quantized-model sidecar transfer quality, CUDA-graph
throughput, concurrent serving, or maximum supported context. The sidecar's
fingerprint binds the BF16 base, not the FP8 checkpoint. Do not promote this
sidecar on speed/memory evidence alone.

For the measured workloads, FP8 weights with dense BF16 KV provide the best
speed/memory tradeoff. DMS and FP8 KV should be reconsidered at longer contexts
only after the sidecar passes quality checks.