Hugging Face's logo

LordNeel
/
Hy3-GGUF

Text Generation
GGUF
llama.cpp
hy3
hunyuan
quantized
imatrix
mixture-of-experts
conversational
Model card Files
xet
 Community
LordNeel commited on
Commit
f4cdebf
·
verified ·
1 Parent(s): 3e1e088

Add Hy3 GGUF KLD quality metrics and updated model card

Browse files
Files changed (15) hide show
  1. README.md +76 -61
  2. assets/mean_kld.svg +18 -0
  3. assets/ppl_ratio.svg +18 -0
  4. assets/rms_delta_p.svg +18 -0
  5. assets/same_top_p.svg +18 -0
  6. assets/size_vs_kld.svg +41 -0
  7. benchmarks/eval_status.json +25 -17
  8. benchmarks/kld_eval_manifest.json +19 -0
  9. benchmarks/kld_logs/Hy3-IQ2_M-imatrix_wikitext2_kld.log +186 -0
  10. benchmarks/kld_logs/Hy3-Q3_K_L-imatrix_wikitext2_kld.log +186 -0
  11. benchmarks/kld_logs/Hy3-Q4_K_M-imatrix_wikitext2_kld.log +186 -0
  12. benchmarks/kld_logs/Hy3-Q6_K-imatrix_wikitext2_kld.log +186 -0
  13. benchmarks/kld_logs/base_bf16_wikitext2_c512_chunks128.log +16 -0
  14. benchmarks/kld_wikitext2_c512_chunks128.json +187 -0
  15. scripts/update_hy3_card_from_kld.py +516 -0
README.md CHANGED
@@ -1,5 +1,5 @@
1
  ---
2
- license: other
3
  base_model: tencent/Hy3
4
  library_name: llama.cpp
5
  tags:
@@ -9,13 +9,14 @@ tags:
9
  - quantized
10
  - imatrix
11
  - llama.cpp
 
12
  ---
13
 
14
  # Hy3 GGUF Quants
15
 
16
- High-quality GGUF quantizations for [`tencent/Hy3`](https://huggingface.co/tencent/Hy3), produced with the Hy3-enabled `llama.cpp` branch and uploaded as portable GGUF artifacts.
17
 
18
- This repo intentionally contains **2-bit, 3-bit, 4-bit, and 6-bit-class GGUFs**. Q8 was skipped by request after the higher-priority quant set was completed.
19
 
20
  ## Files
21
 
@@ -25,104 +26,112 @@ This repo intentionally contains **2-bit, 3-bit, 4-bit, and 6-bit-class GGUFs**.
25
  - `Hy3-Q6_K-imatrix.gguf`
26
  - `calibration/hy3_imatrix_calibration.txt`
27
  - `imatrix/Hy3-imatrix.gguf`
28
- - `benchmarks/*.json`
29
  - `benchmarks/summary.json`
30
- - `benchmarks/eval_status.json`
 
 
31
 
32
- ## Which File Should I Use?
33
 
34
- | Use case | Recommended file | Rationale |
35
  |---|---|---|
36
- | Default portable GGUF serving | `Hy3-Q4_K_M-imatrix.gguf` | Best balance in this set: strong size reduction, best measured prompt TPS, and solid generation TPS. |
37
- | Higher quality / less quantization loss | `Hy3-Q6_K-imatrix.gguf` | Largest uploaded GGUF, highest effective BPW. Use when memory/storage is available. |
38
- | Smallest distributable file | `Hy3-IQ2_M-imatrix.gguf` | Smallest artifact. Use only when size matters more than quality. |
39
- | Mid-size fallback | `Hy3-Q3_K_L-imatrix.gguf` | Smaller than Q4, but Q4 is generally the better tradeoff if it fits. |
40
 
41
- ## Benchmark Summary
42
 
43
- Benchmarks were run with `llama-bench`, CUDA backend, 8x NVIDIA RTX PRO 6000 Blackwell Server Edition, `-ngl 999`, `--split-mode layer`, `--tensor-split 1/1/1/1/1/1/1/1`, prompt length `512`, generation length `128`, repetitions `3`, batch `1024`, ubatch `512`, F16 KV cache.
44
 
45
- The BF16 no-MTP GGUF reference size used for compression ratios was `549.66 GiB` (`590197887712` bytes). The uploaded GGUFs exclude the MTP/NextN block; this is the intended normal-serving path and avoids missing imatrix coverage for speculative-only tensors.
46
 
47
  | Artifact | llama.cpp type | Size GiB | Effective BPW | Compression | Prompt tok/s | Generate tok/s |
48
- |---|---:|---:|---:|---:|---:|---:|
49
  | `Hy3-IQ2_M-imatrix.gguf` | hy_v3 ?B IQ2_M - 2.7 bpw | 90.32 | 2.63 | 6.09x | 1249.41 | 78.66 |
50
- | `Hy3-Q3_K_L-imatrix.gguf` | hy_v3 ?B Q3_K - Large | 143.06 | 4.17 | 3.84x | 1348.49 | 63.32 |
51
  | `Hy3-Q4_K_M-imatrix.gguf` | hy_v3 ?B Q4_K - Medium | 167.08 | 4.86 | 3.29x | 1494.96 | 67.34 |
52
  | `Hy3-Q6_K-imatrix.gguf` | hy_v3 ?B Q6_K | 226.24 | 6.59 | 2.43x | 1232.39 | 57.39 |
53
 
54
- Raw benchmark data is in [`benchmarks/summary.json`](benchmarks/summary.json) and the per-file `llama-bench` JSON files.
55
 
56
- ## Charts
57
 
58
- ![Generation throughput](assets/generation_tps.svg)
59
 
60
- ![Prompt throughput](assets/prompt_tps.svg)
61
 
62
- ![Model size](assets/model_size.svg)
 
 
 
63
 
64
- ![Compression](assets/compression.svg)
 
 
 
 
 
65
 
66
- ![Size vs generation throughput](assets/size_vs_generation_tps.svg)
67
 
68
- ## Quality Metrics
69
 
70
- ### Completed
 
 
71
 
72
- | Metric | Status | Notes |
73
- |---|---|---|
74
- | Imatrix calibration | Complete | `Hy3-imatrix.gguf`, 96 chunks, calibration text from project JSONL. |
75
- | Load/decode benchmark | Complete | Each uploaded GGUF loaded and completed `llama-bench` prompt/decode runs. |
76
- | Throughput | Complete | Prompt and generation TPS are reported above. |
77
- | Artifact size / effective BPW / compression | Complete | Derived from benchmark metadata and BF16 no-MTP source size. |
78
 
79
- ### Not Reported In This Release
80
 
81
- These are intentionally **not** filled with placeholder values:
82
 
83
- | Metric | Status | Why |
84
- |---|---|---|
85
- | Perplexity / PPL | Not measured in this release | `llama-perplexity` was built after the upload stream. A separate held-out eval pass should be run before publishing PPL. |
86
- | KLD vs BF16 | Not measured in this release | Requires a BF16 reference logits file from `llama-perplexity --kl-divergence-base`; the 590 GB BF16 intermediate was deleted after upload cleanup. |
87
- | Same top-p / top-token agreement | Not measured in this release | Emitted by `llama-perplexity --kl-divergence` only when comparing against BF16 reference logits. |
88
 
89
- Recommended full-logit eval pattern:
90
 
91
- ```bash
92
- # 1. Generate BF16 reference logits on a fixed held-out eval text.
93
- llama-perplexity -m Hy3-BF16.gguf -f eval.txt -c 512 -b 1024 -ub 512 -ngl 999 --split-mode layer --tensor-split 1,1,1,1,1,1,1,1 --kl-divergence-base Hy3-BF16.eval.kld
94
 
95
- # 2. Compare each quant against the BF16 reference logits.
96
- llama-perplexity -m Hy3-Q4_K_M-imatrix.gguf -f eval.txt -c 512 -b 1024 -ub 512 -ngl 999 --split-mode layer --tensor-split 1,1,1,1,1,1,1,1 --kl-divergence-base Hy3-BF16.eval.kld --kl-divergence
97
- ```
98
 
99
- Eval status JSON: [`benchmarks/eval_status.json`](benchmarks/eval_status.json).
100
 
 
 
 
101
 
102
  ## Production Serving Reference
103
 
104
- These GGUFs are portable quant artifacts. For production throughput on the tested 8-GPU RunPod machine, the faster serving profile was the separate FP8 SGLang path, not GGUF llama.cpp:
105
 
106
  - SGLang 0.5.5 Hy3 backport, tensor parallel `TP=8`
107
  - FP8 weights and FP8 E4M3 KV cache with calibrated K/V scales
108
  - CUDA graph batch sizes `[1,2,4,8,16,32]`
109
- - Best measured batched result: **688.76 output tok/s** at concurrency `32` over `128` successful requests
110
 
111
- Raw reference JSON: [`benchmarks/sglang_fp8_tp8_reference.json`](benchmarks/sglang_fp8_tp8_reference.json).
112
 
 
113
 
114
  ## llama.cpp Serving Example
115
 
116
- Use `layer` split for this Hy3 GGUF backend. `tensor` split crashed in CUDA decode during testing, even on small probes.
117
 
118
  ```bash
119
- export HF_TOKEN=... # required while this repo is private
120
-
121
- llama-server --hf-repo LordNeel/Hy3-GGUF --hf-file Hy3-Q4_K_M-imatrix.gguf -ngl 999 --split-mode layer --tensor-split 1,1,1,1,1,1,1,1 -c 8192 -b 1024 -ub 512
 
 
 
 
 
 
 
 
122
  ```
123
 
124
- For throughput-oriented production on the tested 8-GPU pod, prefer the FP8 SGLang serving profile rather than GGUF llama.cpp.
125
-
126
  ## Quantization Recipe
127
 
128
  - Base model: `tencent/Hy3`
@@ -130,23 +139,29 @@ For throughput-oriented production on the tested 8-GPU pod, prefer the FP8 SGLan
130
  - Source GGUF: BF16, `--no-mtp`, `--fuse-gate-up-exps`
131
  - Tensor count in no-MTP BF16 GGUF: 1199
132
  - Imatrix: `llama-imatrix`, 96 chunks, calibration text from `hy3_calibration_512.jsonl`
133
- - Quant command pattern:
134
 
135
  ```bash
136
- llama-quantize --imatrix Hy3-imatrix.gguf --leave-output-tensor --token-embedding-type q8_0 Hy3-BF16.gguf Hy3-Q4_K_M-imatrix.gguf Q4_K_M 64
 
 
 
 
 
 
 
137
  ```
138
 
139
- The reproduction runner used for this upload is available at [`scripts/run_hy3_gguf_stream_upload.sh`](scripts/run_hy3_gguf_stream_upload.sh).
140
 
141
  ## Known Limitations
142
 
143
- - KLD, PPL, and same top-p/top-token agreement are not reported yet. The model card records this explicitly in `benchmarks/eval_status.json`.
144
  - The uploaded GGUFs exclude the MTP/NextN block. That is intentional for normal serving.
145
  - `llama.cpp --split-mode tensor` was unstable for this Hy3 GGUF path on the tested CUDA stack; use `--split-mode layer`.
146
  - GPTQ4/SGLang was tested separately and classified invalid because generation collapsed into repeated punctuation. Those artifacts are not published here.
147
- - Q8 was skipped by request.
148
 
149
- ## Hardware Used For Benchmarks
150
 
151
  - 8x NVIDIA RTX PRO 6000 Blackwell Server Edition, about 97 GB VRAM each
152
  - CUDA 12.8 / driver 570.195.03
 
1
  ---
2
+ license: apache-2.0
3
  base_model: tencent/Hy3
4
  library_name: llama.cpp
5
  tags:
 
9
  - quantized
10
  - imatrix
11
  - llama.cpp
12
+ - mixture-of-experts
13
  ---
14
 
15
  # Hy3 GGUF Quants
16
 
17
+ High-quality GGUF quantizations for [`tencent/Hy3`](https://huggingface.co/tencent/Hy3), produced with the Hy3-enabled `llama.cpp` branch. This repo contains 2-bit, 3-bit, 4-bit, and 6-bit-class imatrix GGUFs. Q8 was intentionally skipped by request.
18
 
19
+ Hy3 is a 295B-parameter MoE model with 21B active parameters. These GGUFs are converted from the BF16 base model with the MTP/NextN block excluded for normal serving.
20
 
21
  ## Files
22
 
 
26
  - `Hy3-Q6_K-imatrix.gguf`
27
  - `calibration/hy3_imatrix_calibration.txt`
28
  - `imatrix/Hy3-imatrix.gguf`
 
29
  - `benchmarks/summary.json`
30
+ - `benchmarks/kld_wikitext2_c512_chunks128.json`
31
+ - `benchmarks/kld_logs/*.log`
32
+ - `assets/*.svg`
33
 
34
+ ## Recommended Files
35
 
36
+ | Use case | Recommended file | Why |
37
  |---|---|---|
38
+ | Highest quality GGUF | `Hy3-Q6_K-imatrix.gguf` | Lowest measured KLD in this set: 0.020736; same top-p 95.123%. |
39
+ | Default portable GGUF | `Hy3-Q4_K_M-imatrix.gguf` | Best balance of quality, size, and throughput if 167 GiB fits. |
40
+ | Smaller fallback | `Hy3-Q3_K_L-imatrix.gguf` | Use when Q4 does not fit; quality loss is materially higher than Q4. |
41
+ | Minimum size | `Hy3-IQ2_M-imatrix.gguf` | Smallest artifact. Use only when size is the primary constraint. |
42
 
43
+ ## Throughput
44
 
45
+ `llama-bench` settings: CUDA backend, 8x NVIDIA RTX PRO 6000 Blackwell Server Edition, `-ngl 999`, `--split-mode layer`, `--tensor-split 1/1/1/1/1/1/1/1`, prompt length `512`, generation length `128`, reps `3`, batch `1024`, ubatch `512`, F16 KV cache.
46
 
47
+ BF16 no-MTP GGUF reference size used for compression ratios: `549.66 GiB` (`590197887712` bytes).
48
 
49
  | Artifact | llama.cpp type | Size GiB | Effective BPW | Compression | Prompt tok/s | Generate tok/s |
50
+ | --- | --- | --- | --- | --- | --- | --- |
51
  | `Hy3-IQ2_M-imatrix.gguf` | hy_v3 ?B IQ2_M - 2.7 bpw | 90.32 | 2.63 | 6.09x | 1249.41 | 78.66 |
52
+ | `Hy3-Q3_K_L-imatrix.gguf` | hy_v3 ?B Q3_K - Large | 143.06 | 4.16 | 3.84x | 1348.49 | 63.32 |
53
  | `Hy3-Q4_K_M-imatrix.gguf` | hy_v3 ?B Q4_K - Medium | 167.08 | 4.86 | 3.29x | 1494.96 | 67.34 |
54
  | `Hy3-Q6_K-imatrix.gguf` | hy_v3 ?B Q6_K | 226.24 | 6.59 | 2.43x | 1232.39 | 57.39 |
55
 
56
+ Raw throughput data: [`benchmarks/summary.json`](benchmarks/summary.json).
57
 
58
+ ## Quality Metrics
59
 
60
+ Quality was measured with `llama-perplexity --kl-divergence` against a BF16 no-MTP GGUF reference generated from `tencent/Hy3`.
61
 
62
+ Eval setup:
63
 
64
+ - Corpus: WikiText-2 raw test from the llama.cpp `scripts/get-wikitext-2.sh` helper.
65
+ - Slice: 128 chunks at context 512, nominally 65,536 tokens.
66
+ - Runtime flags: `-c 512 --chunks 128 -b 1024 -ub 512 -ngl 999 --split-mode layer --tensor-split 1,1,1,1,1,1,1,1`.
67
+ - The BF16 reference logits file was kept local as a large intermediate and is not uploaded.
68
 
69
+ | Artifact | Size GiB | BPW | PPL(Q) | PPL ratio | Mean KLD | Median KLD | RMS delta p | Same top-p |
70
+ | --- | --- | --- | --- | --- | --- | --- | --- | --- |
71
+ | `Hy3-IQ2_M-imatrix.gguf` | 90.32 | 2.63 | 6.656017 +/- 0.099280 | 1.392263 +/- 0.011591 | 0.531406 +/- 0.005920 | 0.156656 | 24.788 +/- 0.186% | 74.749 +/- 0.240% |
72
+ | `Hy3-Q3_K_L-imatrix.gguf` | 143.06 | 4.16 | 5.357597 +/- 0.079670 | 1.120668 +/- 0.005183 | 0.162360 +/- 0.002297 | 0.031625 | 13.675 +/- 0.152% | 86.832 +/- 0.187% |
73
+ | `Hy3-Q4_K_M-imatrix.gguf` | 167.08 | 4.86 | 5.039535 +/- 0.072751 | 1.054138 +/- 0.003620 | 0.090429 +/- 0.001394 | 0.016753 | 10.559 +/- 0.134% | 89.997 +/- 0.166% |
74
+ | `Hy3-Q6_K-imatrix.gguf` | 226.24 | 6.59 | 4.770523 +/- 0.067294 | 0.997868 +/- 0.001625 | 0.020736 +/- 0.000393 | 0.003765 | 5.193 +/- 0.089% | 95.123 +/- 0.119% |
75
 
76
+ Lower KLD, lower PPL ratio, and lower RMS delta p are better. Higher same top-p is better. In llama.cpp's KLD output, "same top p" is the percentage of positions where the quant and BF16 reference agree on the highest-probability token.
77
 
78
+ Raw quality data:
79
 
80
+ - [`benchmarks/kld_wikitext2_c512_chunks128.json`](benchmarks/kld_wikitext2_c512_chunks128.json)
81
+ - [`benchmarks/kld_eval_manifest.json`](benchmarks/kld_eval_manifest.json)
82
+ - [`benchmarks/kld_logs/`](benchmarks/kld_logs/)
83
 
84
+ ## Charts
 
 
 
 
 
85
 
86
+ ![Mean KLD](assets/mean_kld.svg)
87
 
88
+ ![PPL ratio](assets/ppl_ratio.svg)
89
 
90
+ ![Same top-p](assets/same_top_p.svg)
 
 
 
 
91
 
92
+ ![RMS delta p](assets/rms_delta_p.svg)
93
 
94
+ ![Size vs KLD](assets/size_vs_kld.svg)
 
 
95
 
96
+ ![Generation throughput](assets/generation_tps.svg)
 
 
97
 
98
+ ![Prompt throughput](assets/prompt_tps.svg)
99
 
100
+ ![Model size](assets/model_size.svg)
101
+
102
+ ![Compression](assets/compression.svg)
103
 
104
  ## Production Serving Reference
105
 
106
+ These GGUFs are portable llama.cpp artifacts. For maximum throughput on the tested 8-GPU RunPod machine, the faster serving profile was a separate FP8 SGLang path rather than GGUF:
107
 
108
  - SGLang 0.5.5 Hy3 backport, tensor parallel `TP=8`
109
  - FP8 weights and FP8 E4M3 KV cache with calibrated K/V scales
110
  - CUDA graph batch sizes `[1,2,4,8,16,32]`
111
+ - Best measured batched result: `688.76` aggregate output tok/s at concurrency `32` over 128 successful requests
112
 
113
+ Raw serving reference: [`benchmarks/sglang_fp8_tp8_reference.json`](benchmarks/sglang_fp8_tp8_reference.json).
114
 
115
+ Tencent's base model card provides both vLLM and SGLang recipes for Hy3. On this tested pod, the practical production recommendation is SGLang FP8 TP=8 for throughput; use GGUF llama.cpp for portability or memory/packaging constraints.
116
 
117
  ## llama.cpp Serving Example
118
 
119
+ Use `layer` split for this Hy3 GGUF backend. `tensor` split crashed in CUDA decode during testing.
120
 
121
  ```bash
122
+ export HF_TOKEN=...
123
+
124
+ llama-server \
125
+ --hf-repo LordNeel/Hy3-GGUF \
126
+ --hf-file Hy3-Q4_K_M-imatrix.gguf \
127
+ -ngl 999 \
128
+ --split-mode layer \
129
+ --tensor-split 1,1,1,1,1,1,1,1 \
130
+ -c 8192 \
131
+ -b 1024 \
132
+ -ub 512
133
  ```
134
 
 
 
135
  ## Quantization Recipe
136
 
137
  - Base model: `tencent/Hy3`
 
139
  - Source GGUF: BF16, `--no-mtp`, `--fuse-gate-up-exps`
140
  - Tensor count in no-MTP BF16 GGUF: 1199
141
  - Imatrix: `llama-imatrix`, 96 chunks, calibration text from `hy3_calibration_512.jsonl`
142
+ - Q8: skipped by request
143
 
144
  ```bash
145
+ llama-quantize \
146
+ --imatrix Hy3-imatrix.gguf \
147
+ --leave-output-tensor \
148
+ --token-embedding-type q8_0 \
149
+ Hy3-BF16.gguf \
150
+ Hy3-Q4_K_M-imatrix.gguf \
151
+ Q4_K_M \
152
+ 64
153
  ```
154
 
155
+ The reproduction runner used for the upload is available at [`scripts/run_hy3_gguf_stream_upload.sh`](scripts/run_hy3_gguf_stream_upload.sh).
156
 
157
  ## Known Limitations
158
 
159
+ - KLD/PPL numbers are from a bounded WikiText-2 slice, not a full downstream task benchmark.
160
  - The uploaded GGUFs exclude the MTP/NextN block. That is intentional for normal serving.
161
  - `llama.cpp --split-mode tensor` was unstable for this Hy3 GGUF path on the tested CUDA stack; use `--split-mode layer`.
162
  - GPTQ4/SGLang was tested separately and classified invalid because generation collapsed into repeated punctuation. Those artifacts are not published here.
 
163
 
164
+ ## Hardware
165
 
166
  - 8x NVIDIA RTX PRO 6000 Blackwell Server Edition, about 97 GB VRAM each
167
  - CUDA 12.8 / driver 570.195.03
assets/mean_kld.svg ADDED
assets/ppl_ratio.svg ADDED
assets/rms_delta_p.svg ADDED
assets/same_top_p.svg ADDED
assets/size_vs_kld.svg ADDED
benchmarks/eval_status.json CHANGED
@@ -1,29 +1,37 @@
1
  {
2
- "created_at_utc": "2026-07-09",
3
  "benchmark_status": "complete",
4
  "measured_metrics": [
5
  "artifact_size_bytes",
6
  "effective_bits_per_weight",
7
  "compression_vs_bf16_no_mtp",
8
  "llama_bench_prompt_tokens_per_second",
9
- "llama_bench_generation_tokens_per_second"
 
 
 
 
 
 
10
  ],
 
 
 
 
 
 
 
 
 
 
11
  "not_reported_metrics": {
12
- "perplexity": {
13
- "status": "not_measured_in_release",
14
- "reason": "llama-perplexity was built after the GGUF upload pipeline. A separate eval pass should be run on a fixed held-out set before publishing PPL numbers."
15
- },
16
- "kld_vs_bf16": {
17
- "status": "not_measured_in_release",
18
- "reason": "KLD requires a BF16 reference logits file from llama-perplexity --kl-divergence-base. The 590 GB BF16 GGUF intermediate was deleted after upload cleanup to reduce disk pressure."
19
  },
20
- "same_top_p_top_token_agreement": {
21
- "status": "not_measured_in_release",
22
- "reason": "This is emitted by llama-perplexity only when KLD mode is run against BF16 reference logits."
23
  }
24
- },
25
- "recommended_full_logit_eval_commands": [
26
- "llama-perplexity -m Hy3-BF16.gguf -f eval.txt -c 512 -b 1024 -ub 512 -ngl 999 --split-mode layer --tensor-split 1,1,1,1,1,1,1,1 --kl-divergence-base Hy3-BF16.eval.kld",
27
- "llama-perplexity -m Hy3-Q4_K_M-imatrix.gguf -f eval.txt -c 512 -b 1024 -ub 512 -ngl 999 --split-mode layer --tensor-split 1,1,1,1,1,1,1,1 --kl-divergence-base Hy3-BF16.eval.kld --kl-divergence"
28
- ]
29
  }
 
1
  {
2
+ "created_at_utc": "2026-07-09T06:30:26Z",
3
  "benchmark_status": "complete",
4
  "measured_metrics": [
5
  "artifact_size_bytes",
6
  "effective_bits_per_weight",
7
  "compression_vs_bf16_no_mtp",
8
  "llama_bench_prompt_tokens_per_second",
9
+ "llama_bench_generation_tokens_per_second",
10
+ "wiki_text_2_perplexity",
11
+ "kld_vs_bf16_reference_logits",
12
+ "ppl_ratio_vs_bf16",
13
+ "mean_delta_token_probability",
14
+ "rms_delta_token_probability",
15
+ "same_top_p_agreement"
16
  ],
17
+ "quality_eval": {
18
+ "status": "complete",
19
+ "summary_file": "benchmarks/kld_wikitext2_c512_chunks128.json",
20
+ "manifest_file": "benchmarks/kld_eval_manifest.json",
21
+ "corpus": "WikiText-2 raw test",
22
+ "chunks": 128,
23
+ "context_size": 512,
24
+ "nominal_tokens": 65536,
25
+ "reference": "BF16 no-MTP GGUF generated from tencent/Hy3"
26
+ },
27
  "not_reported_metrics": {
28
+ "full_downstream_task_accuracy": {
29
+ "status": "not_measured_in_this_release",
30
+ "reason": "This release focuses on GGUF artifact quality and serving throughput. Downstream task harnesses such as MMLU, GPQA, SWE-bench, or coding evals were not run for these GGUF files."
 
 
 
 
31
  },
32
+ "full_wikitext2_file": {
33
+ "status": "bounded_slice",
34
+ "reason": "KLD uses a 128 chunk x 512 context slice to keep BF16 reference logits manageable for a 295B MoE model."
35
  }
36
+ }
 
 
 
 
37
  }
benchmarks/kld_eval_manifest.json ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "created_at_utc": "2026-07-09T06:30:26Z",
3
+ "purpose": "Full-logit quantization quality comparison against BF16 no-MTP GGUF reference.",
4
+ "reference_logits_file": {
5
+ "local_path": "kld_eval/Hy3-BF16-wikitext2-c512-chunks128.kld",
6
+ "size_note": "Not uploaded to Hugging Face because it is a large intermediate binary."
7
+ },
8
+ "commands": {
9
+ "bf16_reference": "llama-perplexity -m Hy3-BF16.gguf -f wiki.test.raw -c 512 --chunks 128 -b 1024 -ub 512 -ngl 999 --split-mode layer --tensor-split 1,1,1,1,1,1,1,1 --kl-divergence-base Hy3-BF16-wikitext2-c512-chunks128.kld",
10
+ "quant_compare": "llama-perplexity -m <quant.gguf> -f wiki.test.raw -c 512 --chunks 128 -b 1024 -ub 512 -ngl 999 --split-mode layer --tensor-split 1,1,1,1,1,1,1,1 --kl-divergence-base Hy3-BF16-wikitext2-c512-chunks128.kld --kl-divergence"
11
+ },
12
+ "logs": [
13
+ "benchmarks/kld_logs/base_bf16_wikitext2_c512_chunks128.log",
14
+ "benchmarks/kld_logs/Hy3-IQ2_M-imatrix_wikitext2_kld.log",
15
+ "benchmarks/kld_logs/Hy3-Q3_K_L-imatrix_wikitext2_kld.log",
16
+ "benchmarks/kld_logs/Hy3-Q4_K_M-imatrix_wikitext2_kld.log",
17
+ "benchmarks/kld_logs/Hy3-Q6_K-imatrix_wikitext2_kld.log"
18
+ ]
19
+ }
benchmarks/kld_logs/Hy3-IQ2_M-imatrix_wikitext2_kld.log ADDED
@@ -0,0 +1,186 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ started_at_utc=2026-07-09T06:20:53Z
2
+ phase=download model=Hy3-IQ2_M-imatrix.gguf
3
+ ✓ Downloaded
4
+ path: /hy3data/gguf/hy3/kld_eval/models/Hy3-IQ2_M-imatrix.gguf
5
+ -rw-r--r-- 1 root root 91G Jul 9 06:24 /hy3data/gguf/hy3/kld_eval/models/Hy3-IQ2_M-imatrix.gguf
6
+ phase=perplexity model=Hy3-IQ2_M-imatrix.gguf
7
+ 0.02.925.979 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
8
+ 0.13.983.931 I
9
+ 0.13.984.090 I system_info: n_threads = 112 (n_threads_batch = 112) / 224 | CUDA : ARCHS = 1200 | USE_GRAPHS = 1 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
10
+ 0.14.012.901 I kl_divergence: computing over 128 chunks, n_ctx=512, batch_size=1024, n_seq=2
11
+ 0.15.241.841 I kl_divergence: 1.23 seconds per pass - ETA 1.30 minutes
12
+
13
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
14
+ 1 3.5085 ± 0.4519 0.32550 ± 0.09784 0.43449 ± 0.07074 24.564 ± 2.355 % 84.706 ± 2.258 %
15
+ 2 5.2756 ± 0.6367 0.42321 ± 0.07883 0.54471 ± 0.06086 24.307 ± 1.613 % 80.196 ± 1.766 %
16
+ 3 4.2737 ± 0.4022 0.45138 ± 0.06241 0.53929 ± 0.04891 27.124 ± 1.346 % 80.915 ± 1.422 %
17
+ 4 4.1185 ± 0.3260 0.54150 ± 0.05703 0.62817 ± 0.04517 30.401 ± 1.160 % 78.627 ± 1.284 %
18
+ 5 4.2949 ± 0.2994 0.57906 ± 0.05201 0.67202 ± 0.04038 31.570 ± 1.011 % 76.471 ± 1.188 %
19
+ 6 4.6335 ± 0.2979 0.61078 ± 0.04830 0.73861 ± 0.03766 32.758 ± 0.912 % 74.052 ± 1.121 %
20
+ 7 4.8994 ± 0.2952 0.63345 ± 0.04492 0.75841 ± 0.03424 33.307 ± 0.829 % 72.381 ± 1.059 %
21
+ 8 5.0381 ± 0.2827 0.68166 ± 0.04200 0.78831 ± 0.03211 34.460 ± 0.773 % 71.225 ± 1.003 %
22
+ 9 5.4637 ± 0.2949 0.73428 ± 0.04035 0.83155 ± 0.03144 34.842 ± 0.725 % 70.458 ± 0.953 %
23
+ 10 5.7119 ± 0.2954 0.76881 ± 0.03929 0.87226 ± 0.03065 35.232 ± 0.684 % 69.451 ± 0.912 %
24
+ 11 5.8170 ± 0.2875 0.74393 ± 0.03739 0.86609 ± 0.02882 34.833 ± 0.656 % 69.483 ± 0.870 %
25
+ 12 6.0175 ± 0.2870 0.71298 ± 0.03493 0.84018 ± 0.02712 34.041 ± 0.629 % 69.771 ± 0.830 %
26
+ 13 6.3463 ± 0.2965 0.67940 ± 0.03286 0.80492 ± 0.02529 32.983 ± 0.606 % 70.136 ± 0.795 %
27
+ 14 6.3046 ± 0.2836 0.64497 ± 0.03111 0.77576 ± 0.02390 32.307 ± 0.586 % 70.756 ± 0.761 %
28
+ 15 6.4275 ± 0.2808 0.61749 ± 0.02986 0.76016 ± 0.02290 31.835 ± 0.566 % 71.111 ± 0.733 %
29
+ 16 6.3447 ± 0.2671 0.58917 ± 0.02837 0.73670 ± 0.02170 31.218 ± 0.548 % 71.275 ± 0.708 %
30
+ 17 6.3925 ± 0.2625 0.56159 ± 0.02705 0.71205 ± 0.02060 30.514 ± 0.531 % 71.926 ± 0.683 %
31
+ 18 6.6892 ± 0.2706 0.54714 ± 0.02594 0.69582 ± 0.01970 30.029 ± 0.516 % 72.070 ± 0.662 %
32
+ 19 6.4846 ± 0.2554 0.52972 ± 0.02518 0.68286 ± 0.01901 29.836 ± 0.505 % 72.714 ± 0.640 %
33
+ 20 6.4726 ± 0.2478 0.51254 ± 0.02446 0.67666 ± 0.01831 29.542 ± 0.490 % 72.706 ± 0.624 %
34
+ 21 6.5114 ± 0.2430 0.49616 ± 0.02362 0.66542 ± 0.01759 29.214 ± 0.478 % 72.736 ± 0.609 %
35
+ 22 6.3478 ± 0.2310 0.48019 ± 0.02283 0.65416 ± 0.01697 28.985 ± 0.466 % 73.048 ± 0.592 %
36
+ 23 6.1171 ± 0.2171 0.47264 ± 0.02216 0.64196 ± 0.01645 28.755 ± 0.456 % 73.606 ± 0.576 %
37
+ 24 5.8400 ± 0.2016 0.46723 ± 0.02164 0.62974 ± 0.01618 28.447 ± 0.447 % 74.265 ± 0.559 %
38
+ 25 5.6412 ± 0.1893 0.45609 ± 0.02100 0.62086 ± 0.01570 28.462 ± 0.438 % 74.698 ± 0.545 %
39
+ 26 5.4537 ± 0.1783 0.44868 ± 0.02054 0.61247 ± 0.01540 28.432 ± 0.429 % 75.098 ± 0.531 %
40
+ 27 5.3099 ± 0.1688 0.44215 ± 0.01998 0.60618 ± 0.01494 28.423 ± 0.420 % 75.178 ± 0.521 %
41
+ 28 5.3170 ± 0.1656 0.43350 ± 0.01947 0.59978 ± 0.01456 28.187 ± 0.412 % 75.350 ± 0.510 %
42
+ 29 5.2735 ± 0.1606 0.43193 ± 0.01908 0.59812 ± 0.01421 28.167 ± 0.403 % 75.281 ± 0.502 %
43
+ 30 5.2711 ± 0.1577 0.42265 ± 0.01863 0.59210 ± 0.01387 27.915 ± 0.396 % 75.320 ± 0.493 %
44
+ 31 5.3729 ± 0.1592 0.41321 ± 0.01824 0.58536 ± 0.01349 27.666 ± 0.389 % 75.357 ± 0.485 %
45
+ 32 5.3748 ± 0.1567 0.40474 ± 0.01794 0.58413 ± 0.01322 27.515 ± 0.382 % 75.343 ± 0.477 %
46
+ 33 5.3812 ± 0.1545 0.40024 ± 0.01757 0.58035 ± 0.01292 27.431 ± 0.375 % 75.354 ± 0.470 %
47
+ 34 5.4462 ± 0.1543 0.39310 ± 0.01731 0.58088 ± 0.01271 27.363 ± 0.369 % 75.225 ± 0.464 %
48
+ 35 5.5499 ± 0.1556 0.38870 ± 0.01706 0.58056 ± 0.01258 27.214 ± 0.363 % 75.025 ± 0.458 %
49
+ 36 5.6152 ± 0.1553 0.38367 ± 0.01674 0.57696 ± 0.01231 27.096 ± 0.358 % 75.000 ± 0.452 %
50
+ 37 5.6014 ± 0.1525 0.37723 ± 0.01643 0.56921 ± 0.01207 26.913 ± 0.353 % 75.167 ± 0.445 %
51
+ 38 5.6571 ± 0.1526 0.36930 ± 0.01612 0.56297 ± 0.01180 26.659 ± 0.347 % 75.150 ± 0.439 %
52
+ 39 5.7589 ± 0.1547 0.36858 ± 0.01587 0.55721 ± 0.01159 26.471 ± 0.343 % 75.194 ± 0.433 %
53
+ 40 5.7545 ± 0.1525 0.36364 ± 0.01555 0.55039 ± 0.01136 26.262 ± 0.339 % 75.255 ± 0.427 %
54
+ 41 5.8511 ± 0.1539 0.35495 ± 0.01527 0.54515 ± 0.01113 26.031 ± 0.334 % 75.371 ± 0.421 %
55
+ 42 5.9032 ± 0.1534 0.35197 ± 0.01500 0.54260 ± 0.01093 25.902 ± 0.330 % 75.275 ± 0.417 %
56
+ 43 5.8823 ± 0.1505 0.34749 ± 0.01473 0.53800 ± 0.01072 25.750 ± 0.326 % 75.276 ± 0.412 %
57
+ 44 5.9438 ± 0.1507 0.34754 ± 0.01458 0.53535 ± 0.01060 25.570 ± 0.322 % 75.152 ± 0.408 %
58
+ 45 6.0418 ± 0.1526 0.34467 ± 0.01444 0.53161 ± 0.01050 25.423 ± 0.318 % 75.172 ± 0.403 %
59
+ 46 6.1312 ± 0.1535 0.33859 ± 0.01421 0.52758 ± 0.01030 25.233 ± 0.314 % 75.064 ± 0.399 %
60
+ 47 6.1640 ± 0.1529 0.33490 ± 0.01396 0.52329 ± 0.01011 25.112 ± 0.311 % 75.127 ± 0.395 %
61
+ 48 6.1327 ± 0.1502 0.35023 ± 0.01398 0.53657 ± 0.01020 25.778 ± 0.311 % 74.918 ± 0.392 %
62
+ 49 6.1128 ± 0.1479 0.35643 ± 0.01396 0.54739 ± 0.01021 26.130 ± 0.308 % 74.694 ± 0.389 %
63
+ 50 6.1434 ± 0.1470 0.35862 ± 0.01382 0.54863 ± 0.01010 26.143 ± 0.305 % 74.612 ± 0.385 %
64
+ 51 6.1625 ± 0.1458 0.35516 ± 0.01365 0.54744 ± 0.00996 26.049 ± 0.301 % 74.587 ± 0.382 %
65
+ 52 6.1903 ± 0.1450 0.35179 ± 0.01347 0.54594 ± 0.00982 25.938 ± 0.298 % 74.600 ± 0.378 %
66
+ 53 6.1897 ± 0.1435 0.35150 ± 0.01334 0.54476 ± 0.00969 25.873 ± 0.295 % 74.584 ± 0.375 %
67
+ 54 6.2500 ± 0.1439 0.34859 ± 0.01317 0.54253 ± 0.00954 25.717 ± 0.292 % 74.575 ± 0.371 %
68
+ 55 6.2305 ± 0.1418 0.34518 ± 0.01303 0.54105 ± 0.00941 25.706 ± 0.289 % 74.624 ± 0.367 %
69
+ 56 6.2915 ± 0.1422 0.34574 ± 0.01292 0.54110 ± 0.00934 25.634 ± 0.286 % 74.608 ± 0.364 %
70
+ 57 6.3021 ± 0.1409 0.34242 ± 0.01276 0.54021 ± 0.00922 25.564 ± 0.283 % 74.641 ± 0.361 %
71
+ 58 6.3351 ± 0.1406 0.34576 ± 0.01271 0.54105 ± 0.00917 25.535 ± 0.280 % 74.638 ± 0.358 %
72
+ 59 6.3632 ± 0.1400 0.34602 ± 0.01259 0.54063 ± 0.00907 25.497 ± 0.278 % 74.629 ± 0.355 %
73
+ 60 6.3911 ± 0.1393 0.34384 ± 0.01246 0.53961 ± 0.00897 25.441 ± 0.275 % 74.693 ± 0.352 %
74
+ 61 6.4363 ± 0.1393 0.33999 ± 0.01233 0.53757 ± 0.00886 25.393 ± 0.273 % 74.645 ± 0.349 %
75
+ 62 6.4747 ± 0.1388 0.34020 ± 0.01221 0.53679 ± 0.00875 25.319 ± 0.270 % 74.586 ± 0.346 %
76
+ 63 6.5193 ± 0.1387 0.33797 ± 0.01208 0.53469 ± 0.00864 25.241 ± 0.268 % 74.553 ± 0.344 %
77
+ 64 6.5470 ± 0.1383 0.33666 ± 0.01198 0.53433 ± 0.00858 25.174 ± 0.266 % 74.547 ± 0.341 %
78
+ 65 6.6132 ± 0.1391 0.33252 ± 0.01186 0.53257 ± 0.00847 25.069 ± 0.263 % 74.618 ± 0.338 %
79
+ 66 6.6235 ± 0.1379 0.33062 ± 0.01177 0.53321 ± 0.00840 25.028 ± 0.261 % 74.557 ± 0.336 %
80
+ 67 6.6287 ± 0.1369 0.33116 ± 0.01168 0.53325 ± 0.00834 24.994 ± 0.259 % 74.522 ± 0.333 %
81
+ 68 6.6095 ± 0.1353 0.33265 ± 0.01161 0.53388 ± 0.00830 25.009 ± 0.258 % 74.504 ± 0.331 %
82
+ 69 6.6406 ± 0.1349 0.32993 ± 0.01150 0.53215 ± 0.00821 24.940 ± 0.256 % 74.521 ± 0.329 %
83
+ 70 6.6414 ± 0.1338 0.33416 ± 0.01143 0.53626 ± 0.00819 25.062 ± 0.254 % 74.398 ± 0.327 %
84
+ 71 6.6431 ± 0.1329 0.33569 ± 0.01133 0.53718 ± 0.00812 25.087 ± 0.252 % 74.366 ± 0.324 %
85
+ 72 6.6103 ± 0.1311 0.33487 ± 0.01124 0.53625 ± 0.00805 25.103 ± 0.250 % 74.417 ± 0.322 %
86
+ 73 6.6251 ± 0.1306 0.33341 ± 0.01115 0.53683 ± 0.00799 25.055 ± 0.249 % 74.424 ± 0.320 %
87
+ 74 6.5947 ± 0.1290 0.33306 ± 0.01105 0.53625 ± 0.00793 25.071 ± 0.247 % 74.436 ± 0.318 %
88
+ 75 6.6151 ± 0.1286 0.33064 ± 0.01094 0.53382 ± 0.00784 24.992 ± 0.245 % 74.484 ± 0.315 %
89
+ 76 6.5373 ± 0.1259 0.33079 ± 0.01086 0.53395 ± 0.00780 25.047 ± 0.244 % 74.536 ± 0.313 %
90
+ 77 6.4454 ± 0.1230 0.33479 ± 0.01079 0.53479 ± 0.00777 25.225 ± 0.243 % 74.632 ± 0.311 %
91
+ 78 6.3968 ± 0.1212 0.33903 ± 0.01076 0.53793 ± 0.00776 25.391 ± 0.242 % 74.651 ± 0.308 %
92
+ 79 6.4445 ± 0.1217 0.34093 ± 0.01075 0.54182 ± 0.00776 25.455 ± 0.240 % 74.569 ± 0.307 %
93
+ 80 6.4171 ± 0.1202 0.34287 ± 0.01070 0.54404 ± 0.00773 25.573 ± 0.239 % 74.525 ± 0.305 %
94
+ 81 6.4071 ± 0.1190 0.34827 ± 0.01065 0.54693 ± 0.00768 25.725 ± 0.237 % 74.442 ± 0.304 %
95
+ 82 6.3920 ± 0.1178 0.35069 ± 0.01060 0.54969 ± 0.00765 25.818 ± 0.236 % 74.376 ± 0.302 %
96
+ 83 6.4520 ± 0.1186 0.34926 ± 0.01055 0.54987 ± 0.00761 25.769 ± 0.234 % 74.392 ± 0.300 %
97
+ 84 6.4731 ± 0.1185 0.35438 ± 0.01051 0.55382 ± 0.00761 25.878 ± 0.233 % 74.337 ± 0.298 %
98
+ 85 6.4775 ± 0.1178 0.35484 ± 0.01045 0.55575 ± 0.00757 25.931 ± 0.231 % 74.265 ± 0.297 %
99
+ 86 6.4832 ± 0.1172 0.35462 ± 0.01037 0.55513 ± 0.00752 25.900 ± 0.230 % 74.254 ± 0.295 %
100
+ 87 6.4853 ± 0.1166 0.35267 ± 0.01031 0.55415 ± 0.00746 25.848 ± 0.228 % 74.298 ± 0.293 %
101
+ 88 6.5393 ± 0.1171 0.35375 ± 0.01025 0.55484 ± 0.00740 25.809 ± 0.227 % 74.220 ± 0.292 %
102
+ 89 6.5258 ± 0.1160 0.35423 ± 0.01018 0.55516 ± 0.00735 25.818 ± 0.225 % 74.179 ± 0.291 %
103
+ 90 6.5472 ± 0.1157 0.35413 ± 0.01011 0.55394 ± 0.00728 25.786 ± 0.224 % 74.148 ± 0.289 %
104
+ 91 6.5565 ± 0.1152 0.35376 ± 0.01004 0.55327 ± 0.00722 25.710 ± 0.222 % 74.178 ± 0.287 %
105
+ 92 6.5663 ± 0.1147 0.35440 ± 0.01000 0.55589 ± 0.00720 25.762 ± 0.221 % 74.088 ± 0.286 %
106
+ 93 6.5623 ± 0.1140 0.35369 ± 0.00994 0.55555 ± 0.00714 25.733 ± 0.219 % 74.084 ± 0.285 %
107
+ 94 6.5453 ± 0.1130 0.35375 ± 0.00987 0.55498 ± 0.00709 25.719 ± 0.218 % 74.122 ± 0.283 %
108
+ 95 6.5416 ± 0.1122 0.35325 ± 0.00983 0.55594 ± 0.00706 25.697 ± 0.217 % 74.114 ± 0.281 %
109
+ 96 6.5381 ± 0.1115 0.35285 ± 0.00977 0.55627 ± 0.00701 25.681 ± 0.216 % 74.109 ± 0.280 %
110
+ 97 6.5709 ± 0.1117 0.35085 ± 0.00973 0.55835 ± 0.00698 25.652 ± 0.214 % 74.004 ± 0.279 %
111
+ 98 6.5910 ± 0.1116 0.34902 ± 0.00968 0.55935 ± 0.00694 25.635 ± 0.213 % 73.966 ± 0.278 %
112
+ 99 6.6093 ± 0.1115 0.34795 ± 0.00964 0.55933 ± 0.00691 25.611 ± 0.212 % 73.987 ± 0.276 %
113
+ 100 6.6000 ± 0.1108 0.34893 ± 0.00960 0.55902 ± 0.00687 25.624 ± 0.211 % 74.027 ± 0.275 %
114
+ 101 6.5970 ± 0.1102 0.34929 ± 0.00955 0.55868 ± 0.00683 25.598 ± 0.210 % 74.067 ± 0.273 %
115
+ 102 6.5662 ± 0.1091 0.35231 ± 0.00952 0.56024 ± 0.00683 25.693 ± 0.209 % 74.064 ± 0.272 %
116
+ 103 6.5923 ± 0.1091 0.35027 ± 0.00947 0.55909 ± 0.00677 25.645 ± 0.208 % 74.072 ± 0.270 %
117
+ 104 6.6264 ± 0.1093 0.34867 ± 0.00941 0.55788 ± 0.00673 25.567 ± 0.207 % 74.099 ± 0.269 %
118
+ 105 6.6769 ± 0.1098 0.34768 ± 0.00935 0.55674 ± 0.00668 25.500 ± 0.206 % 74.099 ± 0.268 %
119
+ 106 6.6706 ± 0.1091 0.34656 ± 0.00931 0.55615 ± 0.00665 25.500 ± 0.205 % 74.110 ± 0.266 %
120
+ 107 6.7250 ± 0.1097 0.34413 ± 0.00924 0.55427 ± 0.00660 25.414 ± 0.204 % 74.154 ± 0.265 %
121
+ 108 6.7443 ± 0.1096 0.34218 ± 0.00918 0.55208 ± 0.00657 25.332 ± 0.203 % 74.234 ± 0.264 %
122
+ 109 6.7630 ± 0.1095 0.34128 ± 0.00911 0.55041 ± 0.00652 25.279 ± 0.202 % 74.236 ± 0.262 %
123
+ 110 6.8133 ± 0.1102 0.33952 ± 0.00905 0.54786 ± 0.00647 25.202 ± 0.201 % 74.285 ± 0.261 %
124
+ 111 6.8641 ± 0.1107 0.33772 ± 0.00900 0.54547 ± 0.00642 25.120 ± 0.200 % 74.315 ± 0.260 %
125
+ 112 6.8981 ± 0.1109 0.33752 ± 0.00895 0.54419 ± 0.00638 25.064 ± 0.199 % 74.310 ± 0.259 %
126
+ 113 6.8515 ± 0.1096 0.34161 ± 0.00892 0.54610 ± 0.00638 25.213 ± 0.199 % 74.350 ± 0.257 %
127
+ 114 6.8375 ± 0.1088 0.34153 ± 0.00888 0.54448 ± 0.00634 25.161 ± 0.198 % 74.420 ± 0.256 %
128
+ 115 6.8356 ± 0.1083 0.34147 ± 0.00883 0.54400 ± 0.00630 25.132 ± 0.197 % 74.404 ± 0.255 %
129
+ 116 6.8198 ± 0.1075 0.33925 ± 0.00880 0.54305 ± 0.00627 25.112 ± 0.196 % 74.459 ± 0.254 %
130
+ 117 6.8366 ± 0.1073 0.33883 ± 0.00875 0.54240 ± 0.00623 25.073 ± 0.195 % 74.449 ± 0.253 %
131
+ 118 6.8308 ± 0.1068 0.33668 ± 0.00871 0.54138 ± 0.00620 25.020 ± 0.194 % 74.463 ± 0.251 %
132
+ 119 6.8156 ± 0.1061 0.33628 ± 0.00868 0.54031 ± 0.00618 24.974 ± 0.194 % 74.497 ± 0.250 %
133
+ 120 6.7889 ± 0.1052 0.33471 ± 0.00863 0.53853 ± 0.00615 24.932 ± 0.193 % 74.559 ± 0.249 %
134
+ 121 6.7692 ± 0.1043 0.33258 ± 0.00859 0.53730 ± 0.00611 24.885 ± 0.192 % 74.604 ± 0.248 %
135
+ 122 6.7613 ± 0.1037 0.33194 ± 0.00854 0.53592 ± 0.00607 24.851 ± 0.191 % 74.629 ± 0.247 %
136
+ 123 6.7174 ± 0.1024 0.33187 ± 0.00849 0.53475 ± 0.00604 24.850 ± 0.190 % 74.698 ± 0.245 %
137
+ 124 6.7033 ± 0.1018 0.33115 ± 0.00847 0.53476 ± 0.00603 24.861 ± 0.189 % 74.693 ± 0.245 %
138
+ 125 6.6877 ± 0.1011 0.33133 ± 0.00843 0.53377 ± 0.00599 24.859 ± 0.189 % 74.692 ± 0.244 %
139
+ 126 6.6639 ± 0.1003 0.33074 ± 0.00840 0.53285 ± 0.00597 24.847 ± 0.188 % 74.728 ± 0.242 %
140
+ 127 6.6511 ± 0.0996 0.33165 ± 0.00837 0.53228 ± 0.00595 24.833 ± 0.187 % 74.748 ± 0.241 %
141
+ 128 6.6560 ± 0.0993 0.33093 ± 0.00833 0.53141 ± 0.00592 24.788 ± 0.186 % 74.749 ± 0.240 %
142
+
143
+ ====== Perplexity statistics ======
144
+ Mean PPL(Q) : 6.656017 ± 0.099280
145
+ Mean PPL(base) : 4.780718 ± 0.067547
146
+ Cor(ln(PPL(Q)), ln(PPL(base))): 83.70%
147
+ Mean ln(PPL(Q)/PPL(base)) : 0.330931 ± 0.008325
148
+ Mean PPL(Q)/PPL(base) : 1.392263 ± 0.011591
149
+ Mean PPL(Q)-PPL(base) : 1.875300 ± 0.056504
150
+
151
+ ====== KL divergence statistics ======
152
+ Mean KLD: 0.531406 ± 0.005920
153
+ Maximum KLD: 16.845301
154
+ 99.9% KLD: 10.952547
155
+ 99.0% KLD: 5.220232
156
+ 95.0% KLD: 2.358272
157
+ 90.0% KLD: 1.421285
158
+ Median KLD: 0.156656
159
+ 10.0% KLD: 0.000407
160
+ 5.0% KLD: 0.000077
161
+ 1.0% KLD: 0.000007
162
+ 0.1% KLD: 0.000000
163
+ Minimum KLD: -0.000003
164
+
165
+ ====== Token probability statistics ======
166
+ Mean Δp: -7.734 ± 0.130 %
167
+ Maximum Δp: 99.956%
168
+ 99.9% Δp: 81.723%
169
+ 99.0% Δp: 46.546%
170
+ 95.0% Δp: 16.613%
171
+ 90.0% Δp: 6.843%
172
+ 75.0% Δp: 0.056%
173
+ Median Δp: -0.369%
174
+ 25.0% Δp: -9.313%
175
+ 10.0% Δp: -36.043%
176
+ 5.0% Δp: -62.426%
177
+ 1.0% Δp: -94.788%
178
+ 0.1% Δp: -99.983%
179
+ Minimum Δp: -100.000%
180
+ RMS Δp : 24.788 ± 0.186 %
181
+ Same top p: 74.749 ± 0.240 %
182
+
183
+ finished_at_utc=2026-07-09T06:25:35Z
184
+ elapsed_seconds=282
185
+ Filesystem Size Used Avail Use% Mounted on
186
+ overlay 1.5T 1.2T 278G 82% /
benchmarks/kld_logs/Hy3-Q3_K_L-imatrix_wikitext2_kld.log ADDED
@@ -0,0 +1,186 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ started_at_utc=2026-07-09T06:18:01Z
2
+ phase=resume_download model=Hy3-Q3_K_L-imatrix.gguf
3
+ ✓ Downloaded
4
+ path: /hy3data/gguf/hy3/kld_eval/models/Hy3-Q3_K_L-imatrix.gguf
5
+ -rw-r--r-- 1 root root 144G Jul 9 06:19 /hy3data/gguf/hy3/kld_eval/models/Hy3-Q3_K_L-imatrix.gguf
6
+ phase=perplexity model=Hy3-Q3_K_L-imatrix.gguf
7
+ 0.02.937.934 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
8
+ 0.20.445.870 I
9
+ 0.20.446.030 I system_info: n_threads = 112 (n_threads_batch = 112) / 224 | CUDA : ARCHS = 1200 | USE_GRAPHS = 1 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
10
+ 0.20.476.574 I kl_divergence: computing over 128 chunks, n_ctx=512, batch_size=1024, n_seq=2
11
+ 0.21.676.047 I kl_divergence: 1.20 seconds per pass - ETA 1.27 minutes
12
+
13
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
14
+ 1 2.7516 ± 0.3245 0.08250 ± 0.04461 0.12010 ± 0.02432 15.111 ± 2.127 % 90.588 ± 1.832 %
15
+ 2 3.7441 ± 0.3875 0.08029 ± 0.03187 0.12895 ± 0.01523 13.088 ± 1.330 % 89.020 ± 1.386 %
16
+ 3 2.9796 ± 0.2431 0.09067 ± 0.02484 0.12440 ± 0.01413 13.500 ± 1.152 % 90.458 ± 1.063 %
17
+ 4 2.7758 ± 0.1897 0.14694 ± 0.02865 0.17255 ± 0.01743 15.752 ± 0.995 % 89.216 ± 0.972 %
18
+ 5 2.8596 ± 0.1735 0.17233 ± 0.02694 0.19808 ± 0.01601 16.617 ± 0.860 % 87.765 ± 0.918 %
19
+ 6 3.0892 ± 0.1747 0.20539 ± 0.02584 0.22180 ± 0.01517 17.946 ± 0.769 % 86.536 ± 0.873 %
20
+ 7 3.2374 ± 0.1752 0.21911 ± 0.02460 0.23471 ± 0.01421 18.406 ± 0.710 % 85.714 ± 0.828 %
21
+ 8 3.2263 ± 0.1625 0.23595 ± 0.02361 0.24884 ± 0.01340 18.886 ± 0.653 % 85.245 ± 0.785 %
22
+ 9 3.3603 ± 0.1615 0.24818 ± 0.02268 0.27260 ± 0.01358 19.532 ± 0.624 % 84.837 ± 0.749 %
23
+ 10 3.3967 ± 0.1533 0.24906 ± 0.02199 0.28592 ± 0.01329 19.998 ± 0.597 % 84.000 ± 0.726 %
24
+ 11 3.5201 ± 0.1534 0.24163 ± 0.02090 0.28755 ± 0.01262 19.942 ± 0.569 % 83.922 ± 0.694 %
25
+ 12 3.7231 ± 0.1596 0.23285 ± 0.01975 0.27964 ± 0.01183 19.539 ± 0.545 % 83.824 ± 0.666 %
26
+ 13 4.0302 ± 0.1715 0.22533 ± 0.01860 0.26672 ± 0.01099 18.956 ± 0.522 % 84.163 ± 0.634 %
27
+ 14 4.0999 ± 0.1692 0.21466 ± 0.01752 0.25425 ± 0.01030 18.490 ± 0.504 % 84.594 ± 0.604 %
28
+ 15 4.2445 ± 0.1710 0.20253 ± 0.01657 0.24293 ± 0.00966 18.007 ± 0.486 % 84.993 ± 0.578 %
29
+ 16 4.2795 ± 0.1671 0.19539 ± 0.01574 0.23386 ± 0.00911 17.614 ± 0.468 % 85.025 ± 0.559 %
30
+ 17 4.4000 ± 0.1691 0.18807 ± 0.01496 0.22448 ± 0.00861 17.179 ± 0.453 % 85.283 ± 0.538 %
31
+ 18 4.6422 ± 0.1770 0.18183 ± 0.01440 0.21751 ± 0.00817 16.860 ± 0.439 % 85.381 ± 0.522 %
32
+ 19 4.5796 ± 0.1710 0.18189 ± 0.01393 0.21232 ± 0.00787 16.665 ± 0.427 % 85.738 ± 0.502 %
33
+ 20 4.6068 ± 0.1675 0.17250 ± 0.01349 0.20967 ± 0.00753 16.489 ± 0.413 % 85.843 ± 0.488 %
34
+ 21 4.6904 ± 0.1675 0.16812 ± 0.01301 0.20487 ± 0.00722 16.208 ± 0.402 % 85.770 ± 0.477 %
35
+ 22 4.6197 ± 0.1615 0.16242 ± 0.01255 0.20145 ± 0.00696 16.066 ± 0.393 % 86.043 ± 0.463 %
36
+ 23 4.4822 ± 0.1530 0.16165 ± 0.01226 0.19804 ± 0.00678 15.993 ± 0.384 % 86.377 ± 0.448 %
37
+ 24 4.2906 ± 0.1416 0.15893 ± 0.01187 0.19413 ± 0.00658 15.891 ± 0.376 % 86.748 ± 0.433 %
38
+ 25 4.1688 ± 0.1340 0.15361 ± 0.01152 0.19064 ± 0.00636 15.809 ± 0.369 % 86.965 ± 0.422 %
39
+ 26 4.0458 ± 0.1264 0.15008 ± 0.01123 0.18648 ± 0.00616 15.681 ± 0.361 % 87.210 ± 0.410 %
40
+ 27 3.9622 ± 0.1206 0.14938 ± 0.01098 0.18345 ± 0.00596 15.558 ± 0.352 % 87.393 ± 0.400 %
41
+ 28 3.9904 ± 0.1192 0.14648 ± 0.01069 0.18086 ± 0.00577 15.401 ± 0.344 % 87.339 ± 0.394 %
42
+ 29 3.9619 ± 0.1157 0.14596 ± 0.01060 0.18310 ± 0.00571 15.625 ± 0.341 % 87.289 ± 0.387 %
43
+ 30 3.9889 ± 0.1148 0.14393 ± 0.01036 0.18236 ± 0.00559 15.527 ± 0.334 % 87.294 ± 0.381 %
44
+ 31 4.0947 ± 0.1173 0.14154 ± 0.01017 0.18124 ± 0.00548 15.436 ± 0.329 % 87.362 ± 0.374 %
45
+ 32 4.1208 ± 0.1163 0.13906 ± 0.00995 0.18005 ± 0.00533 15.315 ± 0.322 % 87.377 ± 0.368 %
46
+ 33 4.1303 ± 0.1148 0.13568 ± 0.00973 0.17834 ± 0.00519 15.269 ± 0.315 % 87.392 ± 0.362 %
47
+ 34 4.2002 ± 0.1156 0.13332 ± 0.00956 0.17684 ± 0.00505 15.153 ± 0.309 % 87.324 ± 0.357 %
48
+ 35 4.2859 ± 0.1167 0.13025 ± 0.00937 0.17569 ± 0.00493 15.052 ± 0.304 % 87.160 ± 0.354 %
49
+ 36 4.3434 ± 0.1171 0.12686 ± 0.00917 0.17455 ± 0.00486 14.938 ± 0.298 % 87.135 ± 0.349 %
50
+ 37 4.3465 ± 0.1155 0.12357 ± 0.00898 0.17172 ± 0.00474 14.830 ± 0.293 % 87.175 ± 0.344 %
51
+ 38 4.4221 ± 0.1167 0.12299 ± 0.00880 0.16971 ± 0.00463 14.691 ± 0.289 % 87.090 ± 0.341 %
52
+ 39 4.4920 ± 0.1180 0.12013 ± 0.00865 0.16736 ± 0.00453 14.562 ± 0.285 % 87.179 ± 0.335 %
53
+ 40 4.5049 ± 0.1166 0.11882 ± 0.00846 0.16438 ± 0.00443 14.417 ± 0.281 % 87.275 ± 0.330 %
54
+ 41 4.6060 ± 0.1186 0.11566 ± 0.00830 0.16214 ± 0.00433 14.287 ± 0.277 % 87.384 ± 0.325 %
55
+ 42 4.6527 ± 0.1182 0.11391 ± 0.00816 0.16085 ± 0.00424 14.198 ± 0.273 % 87.358 ± 0.321 %
56
+ 43 4.6481 ± 0.1164 0.11200 ± 0.00802 0.15925 ± 0.00415 14.097 ± 0.269 % 87.405 ± 0.317 %
57
+ 44 4.6893 ± 0.1161 0.11048 ± 0.00787 0.15792 ± 0.00406 14.012 ± 0.266 % 87.389 ± 0.313 %
58
+ 45 4.7755 ± 0.1177 0.10946 ± 0.00773 0.15610 ± 0.00400 13.887 ± 0.263 % 87.399 ± 0.310 %
59
+ 46 4.8753 ± 0.1197 0.10940 ± 0.00761 0.15454 ± 0.00392 13.787 ± 0.259 % 87.383 ± 0.307 %
60
+ 47 4.9167 ± 0.1198 0.10881 ± 0.00749 0.15310 ± 0.00385 13.699 ± 0.256 % 87.401 ± 0.303 %
61
+ 48 4.8203 ± 0.1157 0.10944 ± 0.00745 0.15449 ± 0.00384 13.891 ± 0.256 % 87.459 ± 0.299 %
62
+ 49 4.8096 ± 0.1141 0.11666 ± 0.00752 0.15971 ± 0.00391 14.233 ± 0.257 % 87.363 ± 0.297 %
63
+ 50 4.8231 ± 0.1133 0.11666 ± 0.00745 0.15996 ± 0.00385 14.223 ± 0.254 % 87.247 ± 0.295 %
64
+ 51 4.8475 ± 0.1126 0.11514 ± 0.00735 0.15980 ± 0.00379 14.212 ± 0.251 % 87.228 ± 0.293 %
65
+ 52 4.8833 ± 0.1125 0.11464 ± 0.00727 0.15993 ± 0.00374 14.183 ± 0.248 % 87.149 ± 0.291 %
66
+ 53 4.8855 ± 0.1114 0.11487 ± 0.00720 0.15997 ± 0.00370 14.170 ± 0.245 % 87.125 ± 0.288 %
67
+ 54 4.9475 ± 0.1122 0.11489 ± 0.00712 0.15904 ± 0.00364 14.081 ± 0.242 % 87.095 ± 0.286 %
68
+ 55 4.9399 ± 0.1109 0.11307 ± 0.00704 0.15853 ± 0.00359 14.041 ± 0.239 % 87.144 ± 0.283 %
69
+ 56 4.9859 ± 0.1111 0.11316 ± 0.00697 0.15812 ± 0.00353 13.978 ± 0.236 % 87.094 ± 0.281 %
70
+ 57 5.0065 ± 0.1106 0.11226 ± 0.00690 0.15806 ± 0.00349 13.969 ± 0.234 % 87.045 ± 0.279 %
71
+ 58 5.0154 ± 0.1098 0.11215 ± 0.00684 0.15788 ± 0.00347 13.960 ± 0.231 % 87.093 ± 0.276 %
72
+ 59 5.0334 ± 0.1093 0.11157 ± 0.00677 0.15738 ± 0.00342 13.921 ± 0.229 % 87.139 ± 0.273 %
73
+ 60 5.0551 ± 0.1088 0.10934 ± 0.00670 0.15666 ± 0.00337 13.860 ± 0.227 % 87.170 ± 0.270 %
74
+ 61 5.1075 ± 0.1094 0.10875 ± 0.00663 0.15598 ± 0.00332 13.828 ± 0.225 % 87.181 ± 0.268 %
75
+ 62 5.1359 ± 0.1090 0.10855 ± 0.00658 0.15562 ± 0.00328 13.791 ± 0.222 % 87.141 ± 0.266 %
76
+ 63 5.1810 ± 0.1092 0.10819 ± 0.00651 0.15554 ± 0.00325 13.765 ± 0.220 % 87.127 ± 0.264 %
77
+ 64 5.2062 ± 0.1089 0.10749 ± 0.00644 0.15497 ± 0.00321 13.710 ± 0.218 % 87.114 ± 0.262 %
78
+ 65 5.2736 ± 0.1100 0.10616 ± 0.00639 0.15510 ± 0.00317 13.678 ± 0.216 % 87.083 ± 0.261 %
79
+ 66 5.2849 ± 0.1092 0.10485 ± 0.00632 0.15453 ± 0.00313 13.624 ± 0.214 % 87.077 ± 0.259 %
80
+ 67 5.2845 ± 0.1082 0.10452 ± 0.00626 0.15447 ± 0.00310 13.599 ± 0.212 % 87.035 ± 0.257 %
81
+ 68 5.2610 ± 0.1068 0.10445 ± 0.00624 0.15449 ± 0.00307 13.607 ± 0.211 % 87.036 ± 0.255 %
82
+ 69 5.3063 ± 0.1071 0.10563 ± 0.00617 0.15363 ± 0.00303 13.548 ± 0.209 % 87.030 ± 0.253 %
83
+ 70 5.2913 ± 0.1058 0.10690 ± 0.00616 0.15604 ± 0.00304 13.745 ± 0.209 % 86.958 ± 0.252 %
84
+ 71 5.2704 ± 0.1045 0.10422 ± 0.00618 0.15947 ± 0.00313 13.840 ± 0.209 % 86.882 ± 0.251 %
85
+ 72 5.2489 ± 0.1032 0.10425 ± 0.00614 0.15954 ± 0.00312 13.863 ± 0.207 % 86.928 ± 0.249 %
86
+ 73 5.2726 ± 0.1031 0.10506 ± 0.00610 0.15993 ± 0.00309 13.834 ± 0.205 % 86.898 ± 0.247 %
87
+ 74 5.2444 ± 0.1017 0.10395 ± 0.00604 0.15913 ± 0.00306 13.778 ± 0.204 % 86.974 ± 0.245 %
88
+ 75 5.2690 ± 0.1017 0.10313 ± 0.00598 0.15849 ± 0.00302 13.732 ± 0.202 % 86.959 ± 0.244 %
89
+ 76 5.2055 ± 0.0995 0.10299 ± 0.00592 0.15789 ± 0.00300 13.694 ± 0.201 % 87.007 ± 0.242 %
90
+ 77 5.1136 ± 0.0968 0.10333 ± 0.00588 0.15732 ± 0.00298 13.711 ± 0.199 % 87.120 ± 0.239 %
91
+ 78 5.0658 ± 0.0951 0.10574 ± 0.00590 0.15875 ± 0.00300 13.867 ± 0.199 % 87.144 ± 0.237 %
92
+ 79 5.0984 ± 0.0954 0.10664 ± 0.00590 0.16025 ± 0.00299 13.954 ± 0.197 % 87.074 ± 0.236 %
93
+ 80 5.0697 ± 0.0940 0.10717 ± 0.00586 0.16082 ± 0.00297 14.013 ± 0.197 % 87.049 ± 0.235 %
94
+ 81 5.0458 ± 0.0928 0.10941 ± 0.00584 0.16250 ± 0.00296 14.145 ± 0.195 % 86.957 ± 0.234 %
95
+ 82 5.0368 ± 0.0919 0.11242 ± 0.00584 0.16399 ± 0.00297 14.241 ± 0.195 % 86.930 ± 0.233 %
96
+ 83 5.0980 ± 0.0929 0.11372 ± 0.00581 0.16457 ± 0.00296 14.223 ± 0.193 % 86.898 ± 0.232 %
97
+ 84 5.0926 ± 0.0923 0.11451 ± 0.00579 0.16588 ± 0.00295 14.318 ± 0.192 % 86.849 ± 0.231 %
98
+ 85 5.0953 ± 0.0918 0.11483 ± 0.00577 0.16663 ± 0.00294 14.349 ± 0.191 % 86.787 ± 0.230 %
99
+ 86 5.1077 ± 0.0915 0.11616 ± 0.00576 0.16727 ± 0.00293 14.323 ± 0.190 % 86.762 ± 0.229 %
100
+ 87 5.1168 ± 0.0912 0.11567 ± 0.00573 0.16729 ± 0.00291 14.303 ± 0.188 % 86.770 ± 0.227 %
101
+ 88 5.1586 ± 0.0915 0.11658 ± 0.00571 0.16814 ± 0.00290 14.312 ± 0.187 % 86.698 ± 0.227 %
102
+ 89 5.1487 ± 0.0907 0.11721 ± 0.00568 0.16898 ± 0.00289 14.360 ± 0.186 % 86.675 ± 0.226 %
103
+ 90 5.1691 ± 0.0906 0.11779 ± 0.00564 0.16888 ± 0.00286 14.345 ± 0.185 % 86.636 ± 0.225 %
104
+ 91 5.1884 ± 0.0904 0.11972 ± 0.00563 0.16964 ± 0.00285 14.355 ± 0.184 % 86.602 ± 0.224 %
105
+ 92 5.1936 ± 0.0900 0.11989 ± 0.00561 0.17076 ± 0.00285 14.370 ± 0.183 % 86.539 ± 0.223 %
106
+ 93 5.1951 ± 0.0896 0.12007 ± 0.00559 0.17108 ± 0.00284 14.357 ± 0.182 % 86.519 ± 0.222 %
107
+ 94 5.1835 ± 0.0888 0.12049 ± 0.00556 0.17117 ± 0.00282 14.371 ± 0.181 % 86.512 ± 0.221 %
108
+ 95 5.1814 ± 0.0882 0.12013 ± 0.00554 0.17175 ± 0.00280 14.381 ± 0.179 % 86.485 ± 0.220 %
109
+ 96 5.1778 ± 0.0876 0.11958 ± 0.00550 0.17160 ± 0.00278 14.358 ± 0.178 % 86.471 ± 0.219 %
110
+ 97 5.2087 ± 0.0879 0.11852 ± 0.00550 0.17290 ± 0.00278 14.355 ± 0.177 % 86.396 ± 0.218 %
111
+ 98 5.2333 ± 0.0880 0.11835 ± 0.00547 0.17352 ± 0.00276 14.339 ± 0.176 % 86.383 ± 0.217 %
112
+ 99 5.2540 ± 0.0881 0.11847 ± 0.00545 0.17369 ± 0.00275 14.314 ± 0.175 % 86.389 ± 0.216 %
113
+ 100 5.2402 ± 0.0874 0.11822 ± 0.00542 0.17367 ± 0.00273 14.348 ± 0.174 % 86.408 ± 0.215 %
114
+ 101 5.2377 ± 0.0870 0.11855 ± 0.00539 0.17379 ± 0.00272 14.347 ± 0.173 % 86.395 ± 0.214 %
115
+ 102 5.2002 ± 0.0858 0.11906 ± 0.00536 0.17385 ± 0.00270 14.382 ± 0.172 % 86.398 ± 0.213 %
116
+ 103 5.2315 ± 0.0861 0.11907 ± 0.00534 0.17376 ± 0.00268 14.351 ± 0.171 % 86.381 ± 0.212 %
117
+ 104 5.2673 ± 0.0864 0.11913 ± 0.00530 0.17356 ± 0.00266 14.323 ± 0.170 % 86.384 ± 0.211 %
118
+ 105 5.3135 ± 0.0869 0.11927 ± 0.00527 0.17328 ± 0.00264 14.288 ± 0.169 % 86.360 ± 0.210 %
119
+ 106 5.3126 ± 0.0864 0.11892 ± 0.00523 0.17271 ± 0.00263 14.277 ± 0.169 % 86.389 ± 0.209 %
120
+ 107 5.3672 ± 0.0872 0.11861 ± 0.00520 0.17217 ± 0.00261 14.228 ± 0.168 % 86.384 ± 0.208 %
121
+ 108 5.3892 ± 0.0872 0.11787 ± 0.00515 0.17110 ± 0.00258 14.182 ± 0.167 % 86.456 ± 0.206 %
122
+ 109 5.4044 ± 0.0871 0.11703 ± 0.00512 0.17038 ± 0.00256 14.148 ± 0.166 % 86.498 ± 0.205 %
123
+ 110 5.4512 ± 0.0879 0.11648 ± 0.00508 0.16952 ± 0.00254 14.103 ± 0.165 % 86.506 ± 0.204 %
124
+ 111 5.4979 ± 0.0883 0.11577 ± 0.00504 0.16861 ± 0.00252 14.056 ± 0.164 % 86.539 ± 0.203 %
125
+ 112 5.5299 ± 0.0887 0.11643 ± 0.00504 0.16883 ± 0.00252 14.031 ± 0.164 % 86.527 ± 0.202 %
126
+ 113 5.4771 ± 0.0873 0.11771 ± 0.00503 0.16936 ± 0.00253 14.109 ± 0.164 % 86.559 ± 0.201 %
127
+ 114 5.4629 ± 0.0866 0.11709 ± 0.00499 0.16862 ± 0.00251 14.072 ± 0.163 % 86.591 ± 0.200 %
128
+ 115 5.4587 ± 0.0861 0.11653 ± 0.00496 0.16832 ± 0.00249 14.035 ± 0.162 % 86.578 ± 0.199 %
129
+ 116 5.4574 ± 0.0858 0.11640 ± 0.00494 0.16797 ± 0.00247 14.020 ± 0.161 % 86.602 ± 0.198 %
130
+ 117 5.4766 ± 0.0858 0.11701 ± 0.00491 0.16777 ± 0.00246 13.996 ± 0.160 % 86.600 ± 0.197 %
131
+ 118 5.4825 ± 0.0856 0.11679 ± 0.00488 0.16715 ± 0.00244 13.962 ± 0.159 % 86.617 ± 0.196 %
132
+ 119 5.4743 ± 0.0851 0.11712 ± 0.00486 0.16670 ± 0.00242 13.931 ± 0.159 % 86.640 ± 0.195 %
133
+ 120 5.4606 ± 0.0845 0.11699 ± 0.00482 0.16589 ± 0.00241 13.889 ± 0.158 % 86.693 ± 0.194 %
134
+ 121 5.4543 ± 0.0840 0.11662 ± 0.00480 0.16566 ± 0.00239 13.865 ± 0.157 % 86.693 ± 0.193 %
135
+ 122 5.4495 ± 0.0834 0.11624 ± 0.00477 0.16515 ± 0.00237 13.831 ± 0.156 % 86.689 ± 0.193 %
136
+ 123 5.4121 ± 0.0824 0.11579 ± 0.00473 0.16427 ± 0.00235 13.793 ± 0.156 % 86.762 ± 0.191 %
137
+ 124 5.4031 ± 0.0819 0.11552 ± 0.00471 0.16429 ± 0.00236 13.781 ± 0.155 % 86.774 ± 0.191 %
138
+ 125 5.3865 ± 0.0813 0.11497 ± 0.00470 0.16368 ± 0.00234 13.750 ± 0.154 % 86.792 ± 0.190 %
139
+ 126 5.3662 ± 0.0806 0.11415 ± 0.00468 0.16338 ± 0.00233 13.730 ± 0.153 % 86.794 ± 0.189 %
140
+ 127 5.3505 ± 0.0799 0.11407 ± 0.00465 0.16272 ± 0.00231 13.700 ± 0.153 % 86.821 ± 0.188 %
141
+ 128 5.3576 ± 0.0797 0.11392 ± 0.00462 0.16236 ± 0.00230 13.675 ± 0.152 % 86.832 ± 0.187 %
142
+
143
+ ====== Perplexity statistics ======
144
+ Mean PPL(Q) : 5.357597 ± 0.079670
145
+ Mean PPL(base) : 4.780718 ± 0.067547
146
+ Cor(ln(PPL(Q)), ln(PPL(base))): 95.04%
147
+ Mean ln(PPL(Q)/PPL(base)) : 0.113925 ± 0.004625
148
+ Mean PPL(Q)/PPL(base) : 1.120668 ± 0.005183
149
+ Mean PPL(Q)-PPL(base) : 0.576880 ± 0.026089
150
+
151
+ ====== KL divergence statistics ======
152
+ Mean KLD: 0.162360 ± 0.002297
153
+ Maximum KLD: 9.277530
154
+ 99.9% KLD: 4.673191
155
+ 99.0% KLD: 2.077389
156
+ 95.0% KLD: 0.756051
157
+ 90.0% KLD: 0.402834
158
+ Median KLD: 0.031625
159
+ 10.0% KLD: 0.000036
160
+ 5.0% KLD: 0.000007
161
+ 1.0% KLD: 0.000000
162
+ 0.1% KLD: -0.000003
163
+ Minimum KLD: -0.000010
164
+
165
+ ====== Token probability statistics ======
166
+ Mean Δp: -1.512 ± 0.075 %
167
+ Maximum Δp: 96.193%
168
+ 99.9% Δp: 73.711%
169
+ 99.0% Δp: 37.750%
170
+ 95.0% Δp: 13.300%
171
+ 90.0% Δp: 6.217%
172
+ 75.0% Δp: 0.506%
173
+ Median Δp: -0.002%
174
+ 25.0% Δp: -1.522%
175
+ 10.0% Δp: -10.820%
176
+ 5.0% Δp: -22.136%
177
+ 1.0% Δp: -60.383%
178
+ 0.1% Δp: -90.961%
179
+ Minimum Δp: -99.320%
180
+ RMS Δp : 13.675 ± 0.152 %
181
+ Same top p: 86.832 ± 0.187 %
182
+
183
+ finished_at_utc=2026-07-09T06:20:32Z
184
+ elapsed_seconds=151
185
+ Filesystem Size Used Avail Use% Mounted on
186
+ overlay 1.5T 1.3T 225G 86% /
benchmarks/kld_logs/Hy3-Q4_K_M-imatrix_wikitext2_kld.log ADDED
@@ -0,0 +1,186 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ started_at_utc=2026-07-09T05:52:43Z
2
+ phase=download model=Hy3-Q4_K_M-imatrix.gguf
3
+ ✓ Downloaded
4
+ path: /hy3data/gguf/hy3/kld_eval/models/Hy3-Q4_K_M-imatrix.gguf
5
+ -rw-r--r-- 1 root root 168G Jul 9 05:58 /hy3data/gguf/hy3/kld_eval/models/Hy3-Q4_K_M-imatrix.gguf
6
+ phase=perplexity model=Hy3-Q4_K_M-imatrix.gguf
7
+ 0.02.940.386 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
8
+ 0.23.282.008 I
9
+ 0.23.282.188 I system_info: n_threads = 112 (n_threads_batch = 112) / 224 | CUDA : ARCHS = 1200 | USE_GRAPHS = 1 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
10
+ 0.23.313.328 I kl_divergence: computing over 128 chunks, n_ctx=512, batch_size=1024, n_seq=2
11
+ 0.24.397.432 I kl_divergence: 1.08 seconds per pass - ETA 1.15 minutes
12
+
13
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
14
+ 1 2.5849 ± 0.2937 0.01999 ± 0.02627 0.04040 ± 0.00684 7.264 ± 1.251 % 93.725 ± 1.522 %
15
+ 2 3.5584 ± 0.3645 0.02943 ± 0.02609 0.05840 ± 0.00578 8.246 ± 0.751 % 91.176 ± 1.257 %
16
+ 3 2.8391 ± 0.2255 0.04239 ± 0.01963 0.06254 ± 0.00675 10.217 ± 0.893 % 92.418 ± 0.958 %
17
+ 4 2.4914 ± 0.1580 0.03882 ± 0.01881 0.08492 ± 0.00789 12.367 ± 0.818 % 92.059 ± 0.847 %
18
+ 5 2.5787 ± 0.1459 0.06891 ± 0.01969 0.10258 ± 0.00909 12.769 ± 0.766 % 91.843 ± 0.767 %
19
+ 6 2.7786 ± 0.1481 0.09941 ± 0.01913 0.11774 ± 0.00906 13.615 ± 0.708 % 91.176 ± 0.725 %
20
+ 7 2.9118 ± 0.1489 0.11311 ± 0.01921 0.13129 ± 0.00898 14.394 ± 0.656 % 90.140 ± 0.706 %
21
+ 8 2.8614 ± 0.1344 0.11592 ± 0.01837 0.14433 ± 0.00874 15.181 ± 0.619 % 89.559 ± 0.677 %
22
+ 9 2.9560 ± 0.1322 0.11999 ± 0.01777 0.15826 ± 0.00855 15.764 ± 0.583 % 89.107 ± 0.650 %
23
+ 10 3.0245 ± 0.1287 0.13301 ± 0.01727 0.16866 ± 0.00853 16.159 ± 0.560 % 88.078 ± 0.642 %
24
+ 11 3.1419 ± 0.1300 0.12796 ± 0.01631 0.16597 ± 0.00793 15.886 ± 0.528 % 88.128 ± 0.611 %
25
+ 12 3.3330 ± 0.1367 0.12216 ± 0.01522 0.15930 ± 0.00733 15.526 ± 0.502 % 88.268 ± 0.582 %
26
+ 13 3.6126 ± 0.1474 0.11596 ± 0.01423 0.15195 ± 0.00682 15.094 ± 0.483 % 88.356 ± 0.557 %
27
+ 14 3.6812 ± 0.1455 0.10692 ± 0.01348 0.14532 ± 0.00638 14.688 ± 0.464 % 88.543 ± 0.533 %
28
+ 15 3.8547 ± 0.1504 0.10620 ± 0.01275 0.13985 ± 0.00600 14.359 ± 0.447 % 88.680 ± 0.512 %
29
+ 16 3.8993 ± 0.1474 0.10235 ± 0.01207 0.13461 ± 0.00567 14.042 ± 0.432 % 88.897 ± 0.492 %
30
+ 17 4.0184 ± 0.1495 0.09735 ± 0.01145 0.12887 ± 0.00535 13.687 ± 0.418 % 89.158 ± 0.472 %
31
+ 18 4.2642 ± 0.1581 0.09690 ± 0.01100 0.12496 ± 0.00508 13.382 ± 0.405 % 89.216 ± 0.458 %
32
+ 19 4.1887 ± 0.1518 0.09268 ± 0.01065 0.12082 ± 0.00484 13.171 ± 0.392 % 89.598 ± 0.439 %
33
+ 20 4.2246 ± 0.1491 0.08590 ± 0.01030 0.12048 ± 0.00468 13.084 ± 0.381 % 89.647 ± 0.427 %
34
+ 21 4.3040 ± 0.1489 0.08216 ± 0.00995 0.11842 ± 0.00450 12.932 ± 0.370 % 89.711 ± 0.415 %
35
+ 22 4.2592 ± 0.1446 0.08115 ± 0.00960 0.11551 ± 0.00431 12.763 ± 0.360 % 89.893 ± 0.402 %
36
+ 23 4.1232 ± 0.1363 0.07817 ± 0.00926 0.11259 ± 0.00416 12.595 ± 0.352 % 90.111 ± 0.390 %
37
+ 24 3.9526 ± 0.1263 0.07687 ± 0.00896 0.11027 ± 0.00402 12.484 ± 0.343 % 90.376 ± 0.377 %
38
+ 25 3.8461 ± 0.1197 0.07305 ± 0.00867 0.10797 ± 0.00389 12.366 ± 0.335 % 90.573 ± 0.366 %
39
+ 26 3.7379 ± 0.1132 0.07090 ± 0.00840 0.10519 ± 0.00375 12.181 ± 0.328 % 90.814 ± 0.355 %
40
+ 27 3.6667 ± 0.1081 0.07186 ± 0.00818 0.10356 ± 0.00363 12.117 ± 0.320 % 90.864 ± 0.347 %
41
+ 28 3.6978 ± 0.1070 0.07032 ± 0.00795 0.10149 ± 0.00351 11.976 ± 0.313 % 90.924 ± 0.340 %
42
+ 29 3.6715 ± 0.1039 0.06984 ± 0.00785 0.10151 ± 0.00343 12.019 ± 0.305 % 90.926 ± 0.334 %
43
+ 30 3.7090 ± 0.1035 0.07118 ± 0.00771 0.10184 ± 0.00340 11.957 ± 0.301 % 90.928 ± 0.328 %
44
+ 31 3.8135 ± 0.1061 0.07040 ± 0.00759 0.10120 ± 0.00332 11.896 ± 0.295 % 90.879 ± 0.324 %
45
+ 32 3.8395 ± 0.1054 0.06838 ± 0.00748 0.10142 ± 0.00324 11.844 ± 0.289 % 90.748 ± 0.321 %
46
+ 33 3.8540 ± 0.1041 0.06645 ± 0.00732 0.10069 ± 0.00316 11.808 ± 0.284 % 90.707 ± 0.317 %
47
+ 34 3.9196 ± 0.1048 0.06418 ± 0.00717 0.09976 ± 0.00308 11.748 ± 0.278 % 90.588 ± 0.314 %
48
+ 35 3.9996 ± 0.1057 0.06112 ± 0.00710 0.09908 ± 0.00300 11.634 ± 0.273 % 90.443 ± 0.311 %
49
+ 36 4.0639 ± 0.1065 0.06036 ± 0.00695 0.09807 ± 0.00292 11.535 ± 0.268 % 90.403 ± 0.307 %
50
+ 37 4.0809 ± 0.1056 0.06052 ± 0.00679 0.09621 ± 0.00285 11.422 ± 0.263 % 90.493 ± 0.302 %
51
+ 38 4.1527 ± 0.1068 0.06013 ± 0.00664 0.09482 ± 0.00278 11.308 ± 0.259 % 90.475 ± 0.298 %
52
+ 39 4.2289 ± 0.1083 0.05977 ± 0.00651 0.09329 ± 0.00271 11.182 ± 0.256 % 90.508 ± 0.294 %
53
+ 40 4.2412 ± 0.1071 0.05851 ± 0.00637 0.09164 ± 0.00265 11.076 ± 0.252 % 90.618 ± 0.289 %
54
+ 41 4.3453 ± 0.1093 0.05741 ± 0.00625 0.09032 ± 0.00259 10.964 ± 0.248 % 90.655 ± 0.285 %
55
+ 42 4.3984 ± 0.1094 0.05771 ± 0.00615 0.09003 ± 0.00255 10.978 ± 0.246 % 90.616 ± 0.282 %
56
+ 43 4.3943 ± 0.1077 0.05585 ± 0.00604 0.08886 ± 0.00250 10.887 ± 0.243 % 90.606 ± 0.279 %
57
+ 44 4.4383 ± 0.1075 0.05547 ± 0.00592 0.08781 ± 0.00244 10.793 ± 0.239 % 90.544 ± 0.276 %
58
+ 45 4.5199 ± 0.1091 0.05447 ± 0.00582 0.08652 ± 0.00239 10.695 ± 0.236 % 90.553 ± 0.273 %
59
+ 46 4.6132 ± 0.1109 0.05413 ± 0.00574 0.08615 ± 0.00235 10.667 ± 0.234 % 90.426 ± 0.272 %
60
+ 47 4.6573 ± 0.1112 0.05460 ± 0.00565 0.08545 ± 0.00231 10.599 ± 0.230 % 90.396 ± 0.269 %
61
+ 48 4.5753 ± 0.1076 0.05727 ± 0.00569 0.08734 ± 0.00239 10.805 ± 0.230 % 90.384 ± 0.266 %
62
+ 49 4.5369 ± 0.1053 0.05830 ± 0.00571 0.08968 ± 0.00241 11.078 ± 0.232 % 90.292 ± 0.265 %
63
+ 50 4.5506 ± 0.1044 0.05851 ± 0.00564 0.09011 ± 0.00238 11.084 ± 0.229 % 90.290 ± 0.262 %
64
+ 51 4.5807 ± 0.1039 0.05853 ± 0.00557 0.09044 ± 0.00235 11.079 ± 0.226 % 90.250 ± 0.260 %
65
+ 52 4.6131 ± 0.1038 0.05770 ± 0.00553 0.09087 ± 0.00234 11.067 ± 0.223 % 90.189 ± 0.258 %
66
+ 53 4.6111 ± 0.1027 0.05707 ± 0.00544 0.09038 ± 0.00230 11.033 ± 0.220 % 90.174 ± 0.256 %
67
+ 54 4.6697 ± 0.1032 0.05710 ± 0.00537 0.08991 ± 0.00226 10.978 ± 0.218 % 90.138 ± 0.254 %
68
+ 55 4.6647 ± 0.1021 0.05574 ± 0.00532 0.08991 ± 0.00226 10.952 ± 0.215 % 90.196 ± 0.251 %
69
+ 56 4.7077 ± 0.1022 0.05574 ± 0.00526 0.08982 ± 0.00223 10.925 ± 0.213 % 90.126 ± 0.250 %
70
+ 57 4.7260 ± 0.1016 0.05462 ± 0.00522 0.08964 ± 0.00220 10.890 ± 0.210 % 90.086 ± 0.248 %
71
+ 58 4.7308 ± 0.1008 0.05375 ± 0.00515 0.08943 ± 0.00217 10.851 ± 0.208 % 90.081 ± 0.246 %
72
+ 59 4.7532 ± 0.1004 0.05431 ± 0.00510 0.08928 ± 0.00214 10.854 ± 0.206 % 90.096 ± 0.244 %
73
+ 60 4.7827 ± 0.1002 0.05395 ± 0.00505 0.08900 ± 0.00211 10.821 ± 0.204 % 90.150 ± 0.241 %
74
+ 61 4.8321 ± 0.1008 0.05332 ± 0.00501 0.08874 ± 0.00209 10.811 ± 0.203 % 90.164 ± 0.239 %
75
+ 62 4.8559 ± 0.1003 0.05248 ± 0.00494 0.08836 ± 0.00206 10.753 ± 0.201 % 90.108 ± 0.237 %
76
+ 63 4.8970 ± 0.1004 0.05183 ± 0.00488 0.08809 ± 0.00203 10.709 ± 0.199 % 90.109 ± 0.236 %
77
+ 64 4.9251 ± 0.1003 0.05199 ± 0.00483 0.08774 ± 0.00200 10.669 ± 0.197 % 90.092 ± 0.234 %
78
+ 65 4.9913 ± 0.1013 0.05114 ± 0.00479 0.08781 ± 0.00198 10.643 ± 0.195 % 90.081 ± 0.232 %
79
+ 66 5.0074 ± 0.1008 0.05091 ± 0.00475 0.08757 ± 0.00196 10.602 ± 0.193 % 90.089 ± 0.230 %
80
+ 67 5.0098 ± 0.0999 0.05115 ± 0.00471 0.08746 ± 0.00193 10.579 ± 0.191 % 90.097 ± 0.229 %
81
+ 68 4.9880 ± 0.0986 0.05117 ± 0.00467 0.08718 ± 0.00191 10.564 ± 0.189 % 90.121 ± 0.227 %
82
+ 69 5.0248 ± 0.0986 0.05112 ± 0.00462 0.08676 ± 0.00188 10.519 ± 0.188 % 90.156 ± 0.225 %
83
+ 70 5.0103 ± 0.0974 0.05233 ± 0.00463 0.08816 ± 0.00190 10.691 ± 0.188 % 90.101 ± 0.224 %
84
+ 71 4.9973 ± 0.0964 0.05100 ± 0.00464 0.08996 ± 0.00197 10.712 ± 0.186 % 90.086 ± 0.222 %
85
+ 72 4.9730 ± 0.0951 0.05027 ± 0.00460 0.08961 ± 0.00195 10.685 ± 0.184 % 90.158 ± 0.220 %
86
+ 73 4.9896 ± 0.0948 0.04990 ± 0.00456 0.08980 ± 0.00193 10.674 ± 0.183 % 90.105 ± 0.219 %
87
+ 74 4.9669 ± 0.0936 0.04960 ± 0.00451 0.08921 ± 0.00190 10.629 ± 0.181 % 90.159 ± 0.217 %
88
+ 75 4.9917 ± 0.0936 0.04906 ± 0.00447 0.08884 ± 0.00188 10.604 ± 0.180 % 90.180 ± 0.215 %
89
+ 76 4.9314 ± 0.0916 0.04889 ± 0.00443 0.08838 ± 0.00186 10.584 ± 0.178 % 90.217 ± 0.213 %
90
+ 77 4.8417 ± 0.0890 0.04869 ± 0.00439 0.08807 ± 0.00186 10.610 ± 0.178 % 90.283 ± 0.211 %
91
+ 78 4.7874 ± 0.0872 0.04922 ± 0.00438 0.08828 ± 0.00185 10.652 ± 0.178 % 90.342 ± 0.209 %
92
+ 79 4.8128 ± 0.0874 0.04898 ± 0.00437 0.08924 ± 0.00184 10.718 ± 0.176 % 90.320 ± 0.208 %
93
+ 80 4.7849 ± 0.0862 0.04936 ± 0.00435 0.08961 ± 0.00183 10.767 ± 0.175 % 90.279 ± 0.207 %
94
+ 81 4.7574 ± 0.0849 0.05056 ± 0.00434 0.09042 ± 0.00183 10.851 ± 0.174 % 90.230 ± 0.207 %
95
+ 82 4.7362 ± 0.0839 0.05087 ± 0.00431 0.09124 ± 0.00183 10.892 ± 0.173 % 90.220 ± 0.205 %
96
+ 83 4.7863 ± 0.0846 0.05063 ± 0.00429 0.09132 ± 0.00181 10.891 ± 0.171 % 90.187 ± 0.204 %
97
+ 84 4.7807 ± 0.0841 0.05133 ± 0.00428 0.09226 ± 0.00181 10.958 ± 0.170 % 90.159 ± 0.204 %
98
+ 85 4.7820 ± 0.0836 0.05137 ± 0.00427 0.09281 ± 0.00181 11.010 ± 0.170 % 90.150 ± 0.202 %
99
+ 86 4.7851 ± 0.0831 0.05091 ± 0.00423 0.09277 ± 0.00179 10.979 ± 0.169 % 90.182 ± 0.201 %
100
+ 87 4.8012 ± 0.0830 0.05200 ± 0.00422 0.09290 ± 0.00178 10.983 ± 0.168 % 90.160 ± 0.200 %
101
+ 88 4.8350 ± 0.0831 0.05181 ± 0.00420 0.09312 ± 0.00176 10.971 ± 0.166 % 90.125 ± 0.199 %
102
+ 89 4.8256 ± 0.0823 0.05238 ± 0.00417 0.09316 ± 0.00175 10.966 ± 0.165 % 90.108 ± 0.198 %
103
+ 90 4.8424 ± 0.0822 0.05252 ± 0.00415 0.09313 ± 0.00174 10.943 ± 0.164 % 90.074 ± 0.197 %
104
+ 91 4.8546 ± 0.0819 0.05323 ± 0.00413 0.09330 ± 0.00173 10.939 ± 0.162 % 90.041 ± 0.197 %
105
+ 92 4.8623 ± 0.0816 0.05397 ± 0.00413 0.09414 ± 0.00173 10.988 ± 0.161 % 89.983 ± 0.196 %
106
+ 93 4.8664 ± 0.0813 0.05472 ± 0.00412 0.09440 ± 0.00172 11.000 ± 0.161 % 89.952 ± 0.195 %
107
+ 94 4.8525 ± 0.0805 0.05450 ± 0.00411 0.09456 ± 0.00172 11.021 ± 0.160 % 89.962 ± 0.194 %
108
+ 95 4.8508 ± 0.0800 0.05421 ± 0.00410 0.09515 ± 0.00171 11.054 ± 0.159 % 89.903 ± 0.194 %
109
+ 96 4.8496 ± 0.0795 0.05410 ± 0.00408 0.09543 ± 0.00170 11.059 ± 0.158 % 89.886 ± 0.193 %
110
+ 97 4.8813 ± 0.0799 0.05361 ± 0.00408 0.09588 ± 0.00169 11.066 ± 0.157 % 89.812 ± 0.192 %
111
+ 98 4.9109 ± 0.0802 0.05477 ± 0.00408 0.09638 ± 0.00169 11.044 ± 0.156 % 89.776 ± 0.192 %
112
+ 99 4.9299 ± 0.0803 0.05480 ± 0.00406 0.09628 ± 0.00167 11.034 ± 0.155 % 89.772 ± 0.191 %
113
+ 100 4.9192 ± 0.0797 0.05501 ± 0.00403 0.09606 ± 0.00166 11.008 ± 0.154 % 89.812 ± 0.189 %
114
+ 101 4.9150 ± 0.0793 0.05496 ± 0.00402 0.09616 ± 0.00165 11.010 ± 0.153 % 89.816 ± 0.188 %
115
+ 102 4.8793 ± 0.0782 0.05536 ± 0.00399 0.09622 ± 0.00164 11.036 ± 0.152 % 89.819 ± 0.188 %
116
+ 103 4.9076 ± 0.0784 0.05516 ± 0.00397 0.09611 ± 0.00163 11.023 ± 0.151 % 89.812 ± 0.187 %
117
+ 104 4.9390 ± 0.0787 0.05477 ± 0.00394 0.09607 ± 0.00162 11.006 ± 0.151 % 89.796 ± 0.186 %
118
+ 105 4.9827 ± 0.0792 0.05499 ± 0.00391 0.09584 ± 0.00161 10.976 ± 0.150 % 89.785 ± 0.185 %
119
+ 106 4.9840 ± 0.0788 0.05507 ± 0.00389 0.09549 ± 0.00159 10.948 ± 0.149 % 89.793 ± 0.184 %
120
+ 107 5.0345 ± 0.0794 0.05461 ± 0.00386 0.09512 ± 0.00158 10.911 ± 0.148 % 89.797 ± 0.183 %
121
+ 108 5.0576 ± 0.0795 0.05438 ± 0.00383 0.09463 ± 0.00157 10.868 ± 0.148 % 89.818 ± 0.182 %
122
+ 109 5.0738 ± 0.0795 0.05391 ± 0.00381 0.09421 ± 0.00156 10.835 ± 0.147 % 89.811 ± 0.181 %
123
+ 110 5.1205 ± 0.0803 0.05390 ± 0.00378 0.09384 ± 0.00154 10.804 ± 0.146 % 89.786 ± 0.181 %
124
+ 111 5.1669 ± 0.0808 0.05367 ± 0.00376 0.09326 ± 0.00153 10.759 ± 0.145 % 89.800 ± 0.180 %
125
+ 112 5.1952 ± 0.0810 0.05400 ± 0.00373 0.09310 ± 0.00152 10.737 ± 0.144 % 89.793 ± 0.179 %
126
+ 113 5.1435 ± 0.0797 0.05488 ± 0.00373 0.09340 ± 0.00152 10.769 ± 0.144 % 89.832 ± 0.178 %
127
+ 114 5.1318 ± 0.0791 0.05456 ± 0.00371 0.09321 ± 0.00151 10.760 ± 0.143 % 89.828 ± 0.177 %
128
+ 115 5.1283 ± 0.0786 0.05410 ± 0.00368 0.09314 ± 0.00151 10.750 ± 0.142 % 89.804 ± 0.177 %
129
+ 116 5.1258 ± 0.0783 0.05371 ± 0.00366 0.09297 ± 0.00150 10.734 ± 0.142 % 89.838 ± 0.176 %
130
+ 117 5.1383 ± 0.0782 0.05326 ± 0.00364 0.09279 ± 0.00149 10.719 ± 0.141 % 89.864 ± 0.175 %
131
+ 118 5.1427 ± 0.0780 0.05283 ± 0.00362 0.09253 ± 0.00148 10.694 ± 0.140 % 89.847 ± 0.174 %
132
+ 119 5.1339 ± 0.0774 0.05293 ± 0.00360 0.09225 ± 0.00147 10.681 ± 0.140 % 89.880 ± 0.173 %
133
+ 120 5.1213 ± 0.0769 0.05284 ± 0.00357 0.09182 ± 0.00145 10.655 ± 0.139 % 89.918 ± 0.172 %
134
+ 121 5.1172 ± 0.0765 0.05282 ± 0.00356 0.09161 ± 0.00144 10.639 ± 0.138 % 89.934 ± 0.171 %
135
+ 122 5.1146 ± 0.0760 0.05281 ± 0.00353 0.09137 ± 0.00143 10.618 ± 0.138 % 89.936 ± 0.171 %
136
+ 123 5.0815 ± 0.0751 0.05277 ± 0.00351 0.09109 ± 0.00143 10.614 ± 0.137 % 89.973 ± 0.170 %
137
+ 124 5.0760 ± 0.0747 0.05307 ± 0.00350 0.09112 ± 0.00143 10.610 ± 0.137 % 89.984 ± 0.169 %
138
+ 125 5.0626 ± 0.0742 0.05294 ± 0.00349 0.09097 ± 0.00142 10.599 ± 0.136 % 89.995 ± 0.168 %
139
+ 126 5.0465 ± 0.0736 0.05273 ± 0.00347 0.09083 ± 0.00141 10.586 ± 0.136 % 90.000 ± 0.167 %
140
+ 127 5.0327 ± 0.0730 0.05283 ± 0.00345 0.09061 ± 0.00140 10.565 ± 0.135 % 90.032 ± 0.166 %
141
+ 128 5.0395 ± 0.0728 0.05272 ± 0.00343 0.09043 ± 0.00139 10.559 ± 0.134 % 89.997 ± 0.166 %
142
+
143
+ ====== Perplexity statistics ======
144
+ Mean PPL(Q) : 5.039535 ± 0.072751
145
+ Mean PPL(base) : 4.780718 ± 0.067547
146
+ Cor(ln(PPL(Q)), ln(PPL(base))): 97.13%
147
+ Mean ln(PPL(Q)/PPL(base)) : 0.052723 ± 0.003434
148
+ Mean PPL(Q)/PPL(base) : 1.054138 ± 0.003620
149
+ Mean PPL(Q)-PPL(base) : 0.258817 ± 0.017576
150
+
151
+ ====== KL divergence statistics ======
152
+ Mean KLD: 0.090429 ± 0.001394
153
+ Maximum KLD: 7.732064
154
+ 99.9% KLD: 2.956301
155
+ 99.0% KLD: 1.174762
156
+ 95.0% KLD: 0.406217
157
+ 90.0% KLD: 0.215485
158
+ Median KLD: 0.016753
159
+ 10.0% KLD: 0.000021
160
+ 5.0% KLD: 0.000005
161
+ 1.0% KLD: 0.000000
162
+ 0.1% KLD: -0.000003
163
+ Minimum KLD: -0.000004
164
+
165
+ ====== Token probability statistics ======
166
+ Mean Δp: -0.821 ± 0.058 %
167
+ Maximum Δp: 95.498%
168
+ 99.9% Δp: 64.379%
169
+ 99.0% Δp: 30.095%
170
+ 95.0% Δp: 10.592%
171
+ 90.0% Δp: 4.906%
172
+ 75.0% Δp: 0.405%
173
+ Median Δp: -0.001%
174
+ 25.0% Δp: -1.016%
175
+ 10.0% Δp: -7.164%
176
+ 5.0% Δp: -15.151%
177
+ 1.0% Δp: -43.622%
178
+ 0.1% Δp: -78.308%
179
+ Minimum Δp: -96.507%
180
+ RMS Δp : 10.559 ± 0.134 %
181
+ Same top p: 89.997 ± 0.166 %
182
+
183
+ finished_at_utc=2026-07-09T05:59:20Z
184
+ elapsed_seconds=397
185
+ Filesystem Size Used Avail Use% Mounted on
186
+ overlay 1.5T 1.3T 201G 87% /
benchmarks/kld_logs/Hy3-Q6_K-imatrix_wikitext2_kld.log ADDED
@@ -0,0 +1,186 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ started_at_utc=2026-07-09T05:40:28Z
2
+ phase=download model=Hy3-Q6_K-imatrix.gguf
3
+ ✓ Downloaded
4
+ path: /hy3data/gguf/hy3/kld_eval/models/Hy3-Q6_K-imatrix.gguf
5
+ -rw-r--r-- 1 root root 227G Jul 9 05:50 /hy3data/gguf/hy3/kld_eval/models/Hy3-Q6_K-imatrix.gguf
6
+ phase=perplexity model=Hy3-Q6_K-imatrix.gguf
7
+ 0.03.033.761 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
8
+ 0.30.779.555 I
9
+ 0.30.779.738 I system_info: n_threads = 112 (n_threads_batch = 112) / 224 | CUDA : ARCHS = 1200 | USE_GRAPHS = 1 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
10
+ 0.30.811.921 I kl_divergence: computing over 128 chunks, n_ctx=512, batch_size=1024, n_seq=2
11
+ 0.32.038.682 I kl_divergence: 1.23 seconds per pass - ETA 1.30 minutes
12
+
13
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
14
+ 1 2.5110 ± 0.2769 -0.00899 ± 0.02006 0.01250 ± 0.00288 4.314 ± 0.977 % 96.863 ± 1.094 %
15
+ 2 3.4338 ± 0.3427 -0.00623 ± 0.01416 0.01657 ± 0.00204 4.450 ± 0.553 % 96.667 ± 0.796 %
16
+ 3 2.7237 ± 0.2124 0.00090 ± 0.01026 0.01324 ± 0.00145 4.068 ± 0.430 % 97.516 ± 0.563 %
17
+ 4 2.3908 ± 0.1474 -0.00239 ± 0.00933 0.02028 ± 0.00262 6.121 ± 0.618 % 96.471 ± 0.578 %
18
+ 5 2.4095 ± 0.1312 0.00105 ± 0.00856 0.02453 ± 0.00262 6.560 ± 0.538 % 95.843 ± 0.559 %
19
+ 6 2.5616 ± 0.1314 0.01809 ± 0.00924 0.03207 ± 0.00340 7.604 ± 0.587 % 95.490 ± 0.531 %
20
+ 7 2.6384 ± 0.1285 0.01450 ± 0.00905 0.03246 ± 0.00302 7.465 ± 0.524 % 95.294 ± 0.501 %
21
+ 8 2.5837 ± 0.1149 0.01383 ± 0.00861 0.03419 ± 0.00283 7.462 ± 0.470 % 95.343 ± 0.467 %
22
+ 9 2.6768 ± 0.1135 0.02077 ± 0.00858 0.03850 ± 0.00282 7.873 ± 0.450 % 94.902 ± 0.459 %
23
+ 10 2.6860 ± 0.1074 0.01431 ± 0.00822 0.03989 ± 0.00263 8.041 ± 0.412 % 94.392 ± 0.456 %
24
+ 11 2.8039 ± 0.1094 0.01415 ± 0.00779 0.03995 ± 0.00246 8.014 ± 0.392 % 94.367 ± 0.435 %
25
+ 12 2.9882 ± 0.1157 0.01299 ± 0.00730 0.03851 ± 0.00227 7.783 ± 0.371 % 94.379 ± 0.416 %
26
+ 13 3.2494 ± 0.1260 0.01001 ± 0.00681 0.03668 ± 0.00211 7.547 ± 0.354 % 94.449 ± 0.398 %
27
+ 14 3.3404 ± 0.1265 0.00979 ± 0.00640 0.03491 ± 0.00196 7.350 ± 0.339 % 94.566 ± 0.379 %
28
+ 15 3.4950 ± 0.1305 0.00823 ± 0.00604 0.03335 ± 0.00184 7.154 ± 0.326 % 94.667 ± 0.363 %
29
+ 16 3.5502 ± 0.1288 0.00856 ± 0.00572 0.03227 ± 0.00174 6.989 ± 0.314 % 94.779 ± 0.348 %
30
+ 17 3.6803 ± 0.1319 0.00945 ± 0.00543 0.03115 ± 0.00165 6.842 ± 0.303 % 94.925 ± 0.333 %
31
+ 18 3.9082 ± 0.1396 0.00972 ± 0.00519 0.03016 ± 0.00156 6.680 ± 0.293 % 94.837 ± 0.327 %
32
+ 19 3.8514 ± 0.1343 0.00872 ± 0.00499 0.02919 ± 0.00148 6.557 ± 0.283 % 95.026 ± 0.312 %
33
+ 20 3.9039 ± 0.1331 0.00694 ± 0.00487 0.02936 ± 0.00143 6.589 ± 0.273 % 94.980 ± 0.306 %
34
+ 21 3.9885 ± 0.1335 0.00601 ± 0.00471 0.02914 ± 0.00139 6.536 ± 0.265 % 94.995 ± 0.298 %
35
+ 22 3.9518 ± 0.1298 0.00625 ± 0.00456 0.02855 ± 0.00133 6.447 ± 0.256 % 95.080 ± 0.289 %
36
+ 23 3.8387 ± 0.1228 0.00668 ± 0.00439 0.02761 ± 0.00128 6.342 ± 0.250 % 95.209 ± 0.279 %
37
+ 24 3.6833 ± 0.1138 0.00631 ± 0.00425 0.02711 ± 0.00124 6.283 ± 0.243 % 95.343 ± 0.269 %
38
+ 25 3.5989 ± 0.1084 0.00661 ± 0.00412 0.02679 ± 0.00121 6.273 ± 0.242 % 95.357 ± 0.264 %
39
+ 26 3.5032 ± 0.1026 0.00605 ± 0.00402 0.02612 ± 0.00117 6.206 ± 0.236 % 95.430 ± 0.256 %
40
+ 27 3.4312 ± 0.0978 0.00547 ± 0.00393 0.02565 ± 0.00113 6.149 ± 0.230 % 95.483 ± 0.250 %
41
+ 28 3.4626 ± 0.0969 0.00461 ± 0.00385 0.02522 ± 0.00109 6.078 ± 0.225 % 95.490 ± 0.246 %
42
+ 29 3.4437 ± 0.0942 0.00578 ± 0.00379 0.02547 ± 0.00107 6.166 ± 0.219 % 95.470 ± 0.242 %
43
+ 30 3.4751 ± 0.0937 0.00604 ± 0.00371 0.02508 ± 0.00103 6.095 ± 0.214 % 95.477 ± 0.238 %
44
+ 31 3.5753 ± 0.0963 0.00590 ± 0.00366 0.02520 ± 0.00102 6.125 ± 0.211 % 95.421 ± 0.235 %
45
+ 32 3.6069 ± 0.0959 0.00588 ± 0.00361 0.02529 ± 0.00100 6.094 ± 0.206 % 95.355 ± 0.233 %
46
+ 33 3.6287 ± 0.0951 0.00620 ± 0.00352 0.02497 ± 0.00097 6.069 ± 0.202 % 95.365 ± 0.229 %
47
+ 34 3.6959 ± 0.0959 0.00539 ± 0.00347 0.02469 ± 0.00094 6.026 ± 0.198 % 95.340 ± 0.226 %
48
+ 35 3.7771 ± 0.0971 0.00386 ± 0.00342 0.02444 ± 0.00092 5.974 ± 0.194 % 95.339 ± 0.223 %
49
+ 36 3.8391 ± 0.0979 0.00345 ± 0.00336 0.02420 ± 0.00090 5.940 ± 0.191 % 95.381 ± 0.219 %
50
+ 37 3.8570 ± 0.0971 0.00410 ± 0.00329 0.02383 ± 0.00087 5.882 ± 0.187 % 95.432 ± 0.215 %
51
+ 38 3.9250 ± 0.0981 0.00375 ± 0.00322 0.02353 ± 0.00085 5.834 ± 0.184 % 95.408 ± 0.213 %
52
+ 39 3.9936 ± 0.0994 0.00252 ± 0.00316 0.02312 ± 0.00083 5.772 ± 0.182 % 95.425 ± 0.210 %
53
+ 40 4.0097 ± 0.0984 0.00238 ± 0.00309 0.02273 ± 0.00081 5.717 ± 0.179 % 95.471 ± 0.206 %
54
+ 41 4.1122 ± 0.1007 0.00227 ± 0.00303 0.02234 ± 0.00079 5.660 ± 0.176 % 95.485 ± 0.203 %
55
+ 42 4.1643 ± 0.1007 0.00301 ± 0.00297 0.02207 ± 0.00077 5.608 ± 0.174 % 95.490 ± 0.201 %
56
+ 43 4.1663 ± 0.0993 0.00255 ± 0.00292 0.02183 ± 0.00076 5.565 ± 0.171 % 95.486 ± 0.198 %
57
+ 44 4.2078 ± 0.0992 0.00213 ± 0.00287 0.02160 ± 0.00074 5.516 ± 0.169 % 95.437 ± 0.197 %
58
+ 45 4.2895 ± 0.1009 0.00213 ± 0.00281 0.02127 ± 0.00073 5.460 ± 0.167 % 95.460 ± 0.194 %
59
+ 46 4.3761 ± 0.1024 0.00137 ± 0.00278 0.02118 ± 0.00071 5.456 ± 0.166 % 95.388 ± 0.194 %
60
+ 47 4.4175 ± 0.1027 0.00174 ± 0.00275 0.02114 ± 0.00070 5.458 ± 0.164 % 95.369 ± 0.192 %
61
+ 48 4.3267 ± 0.0990 0.00141 ± 0.00271 0.02107 ± 0.00069 5.476 ± 0.162 % 95.392 ± 0.190 %
62
+ 49 4.2819 ± 0.0966 0.00045 ± 0.00273 0.02137 ± 0.00069 5.549 ± 0.159 % 95.366 ± 0.188 %
63
+ 50 4.2942 ± 0.0958 0.00051 ± 0.00269 0.02136 ± 0.00067 5.528 ± 0.157 % 95.388 ± 0.186 %
64
+ 51 4.3214 ± 0.0954 0.00024 ± 0.00266 0.02127 ± 0.00066 5.502 ± 0.155 % 95.394 ± 0.184 %
65
+ 52 4.3553 ± 0.0953 0.00020 ± 0.00264 0.02141 ± 0.00066 5.517 ± 0.153 % 95.370 ± 0.182 %
66
+ 53 4.3574 ± 0.0944 0.00049 ± 0.00261 0.02138 ± 0.00065 5.515 ± 0.151 % 95.346 ± 0.181 %
67
+ 54 4.4129 ± 0.0949 0.00055 ± 0.00259 0.02134 ± 0.00064 5.497 ± 0.149 % 95.316 ± 0.180 %
68
+ 55 4.4125 ± 0.0941 0.00017 ± 0.00256 0.02124 ± 0.00063 5.483 ± 0.147 % 95.330 ± 0.178 %
69
+ 56 4.4519 ± 0.0942 -0.00011 ± 0.00253 0.02119 ± 0.00062 5.451 ± 0.146 % 95.315 ± 0.177 %
70
+ 57 4.4736 ± 0.0938 -0.00026 ± 0.00251 0.02116 ± 0.00061 5.430 ± 0.144 % 95.294 ± 0.176 %
71
+ 58 4.4811 ± 0.0931 -0.00048 ± 0.00250 0.02126 ± 0.00061 5.420 ± 0.142 % 95.287 ± 0.174 %
72
+ 59 4.4999 ± 0.0927 -0.00045 ± 0.00247 0.02125 ± 0.00060 5.415 ± 0.140 % 95.248 ± 0.173 %
73
+ 60 4.5287 ± 0.0925 -0.00063 ± 0.00244 0.02113 ± 0.00059 5.388 ± 0.139 % 95.242 ± 0.172 %
74
+ 61 4.5771 ± 0.0931 -0.00090 ± 0.00243 0.02120 ± 0.00059 5.408 ± 0.139 % 95.223 ± 0.171 %
75
+ 62 4.6030 ± 0.0927 -0.00099 ± 0.00240 0.02115 ± 0.00058 5.388 ± 0.137 % 95.231 ± 0.169 %
76
+ 63 4.6454 ± 0.0930 -0.00093 ± 0.00237 0.02106 ± 0.00057 5.371 ± 0.136 % 95.219 ± 0.168 %
77
+ 64 4.6683 ± 0.0928 -0.00155 ± 0.00235 0.02102 ± 0.00056 5.354 ± 0.134 % 95.202 ± 0.167 %
78
+ 65 4.7330 ± 0.0938 -0.00199 ± 0.00234 0.02103 ± 0.00056 5.343 ± 0.133 % 95.216 ± 0.166 %
79
+ 66 4.7483 ± 0.0933 -0.00223 ± 0.00232 0.02101 ± 0.00055 5.322 ± 0.131 % 95.223 ± 0.164 %
80
+ 67 4.7507 ± 0.0925 -0.00195 ± 0.00229 0.02098 ± 0.00054 5.306 ± 0.130 % 95.212 ± 0.163 %
81
+ 68 4.7299 ± 0.0913 -0.00196 ± 0.00227 0.02088 ± 0.00053 5.289 ± 0.128 % 95.213 ± 0.162 %
82
+ 69 4.7661 ± 0.0914 -0.00174 ± 0.00225 0.02083 ± 0.00053 5.286 ± 0.128 % 95.186 ± 0.161 %
83
+ 70 4.7491 ± 0.0902 -0.00120 ± 0.00224 0.02098 ± 0.00052 5.326 ± 0.126 % 95.182 ± 0.160 %
84
+ 71 4.7431 ± 0.0894 -0.00119 ± 0.00222 0.02100 ± 0.00052 5.317 ± 0.125 % 95.189 ± 0.159 %
85
+ 72 4.7222 ± 0.0882 -0.00148 ± 0.00221 0.02095 ± 0.00051 5.306 ± 0.124 % 95.218 ± 0.157 %
86
+ 73 4.7394 ± 0.0881 -0.00155 ± 0.00219 0.02105 ± 0.00051 5.293 ± 0.123 % 95.197 ± 0.157 %
87
+ 74 4.7189 ± 0.0870 -0.00164 ± 0.00217 0.02093 ± 0.00050 5.274 ± 0.122 % 95.231 ± 0.155 %
88
+ 75 4.7456 ± 0.0871 -0.00150 ± 0.00215 0.02075 ± 0.00050 5.246 ± 0.121 % 95.258 ± 0.154 %
89
+ 76 4.6884 ± 0.0852 -0.00165 ± 0.00213 0.02086 ± 0.00054 5.236 ± 0.119 % 95.232 ± 0.153 %
90
+ 77 4.6051 ± 0.0828 -0.00141 ± 0.00211 0.02072 ± 0.00054 5.231 ± 0.119 % 95.264 ± 0.152 %
91
+ 78 4.5493 ± 0.0811 -0.00179 ± 0.00210 0.02092 ± 0.00054 5.307 ± 0.122 % 95.264 ± 0.151 %
92
+ 79 4.5726 ± 0.0813 -0.00222 ± 0.00210 0.02100 ± 0.00054 5.323 ± 0.121 % 95.269 ± 0.150 %
93
+ 80 4.5440 ± 0.0801 -0.00230 ± 0.00208 0.02098 ± 0.00053 5.333 ± 0.120 % 95.275 ± 0.149 %
94
+ 81 4.5153 ± 0.0789 -0.00166 ± 0.00207 0.02118 ± 0.00053 5.377 ± 0.119 % 95.197 ± 0.149 %
95
+ 82 4.4945 ± 0.0779 -0.00150 ± 0.00207 0.02130 ± 0.00053 5.409 ± 0.118 % 95.179 ± 0.148 %
96
+ 83 4.5433 ± 0.0786 -0.00148 ± 0.00205 0.02127 ± 0.00052 5.393 ± 0.117 % 95.195 ± 0.147 %
97
+ 84 4.5345 ± 0.0780 -0.00155 ± 0.00205 0.02147 ± 0.00052 5.418 ± 0.116 % 95.177 ± 0.146 %
98
+ 85 4.5347 ± 0.0776 -0.00173 ± 0.00205 0.02157 ± 0.00052 5.443 ± 0.115 % 95.165 ± 0.146 %
99
+ 86 4.5409 ± 0.0772 -0.00147 ± 0.00203 0.02158 ± 0.00051 5.446 ± 0.114 % 95.166 ± 0.145 %
100
+ 87 4.5522 ± 0.0770 -0.00126 ± 0.00202 0.02157 ± 0.00051 5.436 ± 0.113 % 95.163 ± 0.144 %
101
+ 88 4.5841 ± 0.0772 -0.00148 ± 0.00201 0.02163 ± 0.00050 5.435 ± 0.112 % 95.134 ± 0.144 %
102
+ 89 4.5721 ± 0.0764 -0.00157 ± 0.00199 0.02168 ± 0.00050 5.454 ± 0.111 % 95.122 ± 0.143 %
103
+ 90 4.5871 ± 0.0762 -0.00165 ± 0.00198 0.02167 ± 0.00049 5.448 ± 0.110 % 95.102 ± 0.142 %
104
+ 91 4.5953 ± 0.0759 -0.00168 ± 0.00197 0.02173 ± 0.00049 5.445 ± 0.109 % 95.096 ± 0.142 %
105
+ 92 4.6005 ± 0.0755 -0.00138 ± 0.00197 0.02180 ± 0.00048 5.454 ± 0.108 % 95.081 ± 0.141 %
106
+ 93 4.6023 ± 0.0752 -0.00108 ± 0.00196 0.02185 ± 0.00048 5.467 ± 0.107 % 95.050 ± 0.141 %
107
+ 94 4.5895 ± 0.0745 -0.00122 ± 0.00194 0.02180 ± 0.00048 5.463 ± 0.106 % 95.048 ± 0.140 %
108
+ 95 4.5885 ± 0.0740 -0.00138 ± 0.00194 0.02187 ± 0.00047 5.471 ± 0.105 % 95.034 ± 0.140 %
109
+ 96 4.5874 ± 0.0735 -0.00149 ± 0.00193 0.02190 ± 0.00047 5.470 ± 0.105 % 95.008 ± 0.139 %
110
+ 97 4.6208 ± 0.0739 -0.00124 ± 0.00193 0.02204 ± 0.00047 5.474 ± 0.104 % 94.975 ± 0.139 %
111
+ 98 4.6404 ± 0.0740 -0.00188 ± 0.00193 0.02219 ± 0.00047 5.485 ± 0.103 % 94.942 ± 0.139 %
112
+ 99 4.6570 ± 0.0741 -0.00215 ± 0.00192 0.02226 ± 0.00047 5.489 ± 0.103 % 94.902 ± 0.138 %
113
+ 100 4.6455 ± 0.0735 -0.00225 ± 0.00191 0.02223 ± 0.00046 5.489 ± 0.102 % 94.918 ± 0.138 %
114
+ 101 4.6406 ± 0.0731 -0.00248 ± 0.00190 0.02224 ± 0.00046 5.479 ± 0.101 % 94.917 ± 0.137 %
115
+ 102 4.6066 ± 0.0720 -0.00215 ± 0.00189 0.02221 ± 0.00045 5.474 ± 0.100 % 94.937 ± 0.136 %
116
+ 103 4.6343 ± 0.0723 -0.00213 ± 0.00187 0.02215 ± 0.00045 5.457 ± 0.100 % 94.936 ± 0.135 %
117
+ 104 4.6653 ± 0.0725 -0.00223 ± 0.00186 0.02210 ± 0.00045 5.447 ± 0.099 % 94.951 ± 0.134 %
118
+ 105 4.7055 ± 0.0730 -0.00224 ± 0.00185 0.02212 ± 0.00044 5.436 ± 0.098 % 94.947 ± 0.134 %
119
+ 106 4.7069 ± 0.0726 -0.00213 ± 0.00184 0.02203 ± 0.00044 5.424 ± 0.098 % 94.965 ± 0.133 %
120
+ 107 4.7555 ± 0.0733 -0.00240 ± 0.00182 0.02195 ± 0.00044 5.409 ± 0.097 % 94.964 ± 0.132 %
121
+ 108 4.7790 ± 0.0734 -0.00230 ± 0.00181 0.02184 ± 0.00043 5.390 ± 0.097 % 94.978 ± 0.132 %
122
+ 109 4.7965 ± 0.0734 -0.00229 ± 0.00180 0.02176 ± 0.00043 5.375 ± 0.096 % 94.992 ± 0.131 %
123
+ 110 4.8420 ± 0.0741 -0.00203 ± 0.00180 0.02174 ± 0.00043 5.363 ± 0.096 % 94.966 ± 0.131 %
124
+ 111 4.8867 ± 0.0746 -0.00209 ± 0.00178 0.02162 ± 0.00042 5.342 ± 0.095 % 94.976 ± 0.130 %
125
+ 112 4.9122 ± 0.0748 -0.00201 ± 0.00178 0.02164 ± 0.00042 5.327 ± 0.095 % 94.986 ± 0.129 %
126
+ 113 4.8603 ± 0.0736 -0.00176 ± 0.00177 0.02158 ± 0.00042 5.327 ± 0.094 % 95.010 ± 0.128 %
127
+ 114 4.8500 ± 0.0730 -0.00190 ± 0.00176 0.02148 ± 0.00042 5.312 ± 0.094 % 95.015 ± 0.128 %
128
+ 115 4.8482 ± 0.0726 -0.00207 ± 0.00175 0.02142 ± 0.00041 5.298 ± 0.093 % 95.032 ± 0.127 %
129
+ 116 4.8479 ± 0.0723 -0.00202 ± 0.00174 0.02140 ± 0.00041 5.288 ± 0.092 % 95.024 ± 0.126 %
130
+ 117 4.8625 ± 0.0723 -0.00192 ± 0.00173 0.02135 ± 0.00041 5.282 ± 0.092 % 95.036 ± 0.126 %
131
+ 118 4.8685 ± 0.0722 -0.00197 ± 0.00172 0.02125 ± 0.00040 5.268 ± 0.092 % 95.048 ± 0.125 %
132
+ 119 4.8599 ± 0.0717 -0.00192 ± 0.00171 0.02120 ± 0.00040 5.257 ± 0.091 % 95.050 ± 0.125 %
133
+ 120 4.8483 ± 0.0712 -0.00195 ± 0.00170 0.02110 ± 0.00040 5.246 ± 0.091 % 95.059 ± 0.124 %
134
+ 121 4.8434 ± 0.0707 -0.00217 ± 0.00169 0.02107 ± 0.00039 5.234 ± 0.090 % 95.067 ± 0.123 %
135
+ 122 4.8399 ± 0.0703 -0.00238 ± 0.00168 0.02101 ± 0.00039 5.225 ± 0.090 % 95.066 ± 0.123 %
136
+ 123 4.8090 ± 0.0695 -0.00234 ± 0.00167 0.02092 ± 0.00039 5.216 ± 0.089 % 95.084 ± 0.122 %
137
+ 124 4.8030 ± 0.0691 -0.00221 ± 0.00167 0.02101 ± 0.00040 5.232 ± 0.091 % 95.092 ± 0.121 %
138
+ 125 4.7910 ± 0.0686 -0.00220 ± 0.00166 0.02096 ± 0.00040 5.232 ± 0.090 % 95.096 ± 0.121 %
139
+ 126 4.7764 ± 0.0680 -0.00228 ± 0.00165 0.02086 ± 0.00040 5.218 ± 0.090 % 95.114 ± 0.120 %
140
+ 127 4.7631 ± 0.0675 -0.00222 ± 0.00164 0.02078 ± 0.00040 5.205 ± 0.089 % 95.134 ± 0.120 %
141
+ 128 4.7705 ± 0.0673 -0.00213 ± 0.00163 0.02074 ± 0.00039 5.193 ± 0.089 % 95.123 ± 0.119 %
142
+
143
+ ====== Perplexity statistics ======
144
+ Mean PPL(Q) : 4.770523 ± 0.067294
145
+ Mean PPL(base) : 4.780718 ± 0.067547
146
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.33%
147
+ Mean ln(PPL(Q)/PPL(base)) : -0.002135 ± 0.001629
148
+ Mean PPL(Q)/PPL(base) : 0.997868 ± 0.001625
149
+ Mean PPL(Q)-PPL(base) : -0.010195 ± 0.007782
150
+
151
+ ====== KL divergence statistics ======
152
+ Mean KLD: 0.020736 ± 0.000393
153
+ Maximum KLD: 4.501527
154
+ 99.9% KLD: 0.789244
155
+ 99.0% KLD: 0.259334
156
+ 95.0% KLD: 0.089398
157
+ 90.0% KLD: 0.047906
158
+ Median KLD: 0.003765
159
+ 10.0% KLD: 0.000006
160
+ 5.0% KLD: 0.000001
161
+ 1.0% KLD: -0.000001
162
+ 0.1% KLD: -0.000004
163
+ Minimum KLD: -0.000020
164
+
165
+ ====== Token probability statistics ======
166
+ Mean Δp: -0.131 ± 0.029 %
167
+ Maximum Δp: 84.180%
168
+ 99.9% Δp: 37.205%
169
+ 99.0% Δp: 15.961%
170
+ 95.0% Δp: 5.459%
171
+ 90.0% Δp: 2.583%
172
+ 75.0% Δp: 0.249%
173
+ Median Δp: -0.000%
174
+ 25.0% Δp: -0.382%
175
+ 10.0% Δp: -2.904%
176
+ 5.0% Δp: -6.108%
177
+ 1.0% Δp: -18.171%
178
+ 0.1% Δp: -41.951%
179
+ Minimum Δp: -85.202%
180
+ RMS Δp : 5.193 ± 0.089 %
181
+ Same top p: 95.123 ± 0.119 %
182
+
183
+ finished_at_utc=2026-07-09T05:52:01Z
184
+ elapsed_seconds=693
185
+ Filesystem Size Used Avail Use% Mounted on
186
+ overlay 1.5T 1.4T 142G 91% /
benchmarks/kld_logs/base_bf16_wikitext2_c512_chunks128.log ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ started_at_utc=2026-07-09T05:36:23Z
2
+ command=llama-perplexity -m /hy3data/gguf/hy3/Hy3-BF16.gguf -f /hy3data/gguf/hy3/kld_eval/corpus/wikitext-2-raw/wiki.test.raw -c 512 --chunks 128 -b 1024 -ub 512 -ngl 999 --split-mode layer --tensor-split 1,1,1,1,1,1,1,1 --kl-divergence-base /hy3data/gguf/hy3/kld_eval/Hy3-BF16-wikitext2-c512-chunks128.kld
3
+ 0.03.373.558 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
4
+ 1.15.005.955 I
5
+ 1.15.006.124 I system_info: n_threads = 112 (n_threads_batch = 112) / 224 | CUDA : ARCHS = 1200 | USE_GRAPHS = 1 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
6
+ 1.15.006.262 I perplexity: saving all logits to /hy3data/gguf/hy3/kld_eval/Hy3-BF16-wikitext2-c512-chunks128.kld
7
+ 1.15.006.271 I perplexity: tokenizing the input ..
8
+ 1.15.615.459 I perplexity: tokenization took 609.168 ms
9
+ 1.15.616.268 I perplexity: calculating perplexity over 128 chunks, n_ctx=512, batch_size=1024, n_seq=2
10
+ 1.17.771.949 I perplexity: 2.10 seconds per pass - ETA 2.23 minutes
11
+ [1]2.5337,[2]3.4815,[3]2.7435,[4]2.4111,[5]2.4187,[6]2.5278,[7]2.6178,[8]2.5631,[9]2.6354,[10]2.6602,[11]2.7763,[12]2.9612,[13]3.2287,[14]3.3189,[15]3.4772,[16]3.5303,[17]3.6557,[18]3.8805,[19]3.8276,[20]3.8862,[21]3.9736,[22]3.9357,[23]3.8224,[24]3.6686,[25]3.5831,[26]3.4895,[27]3.4194,[28]3.4535,[29]3.4304,[30]3.4606,[31]3.5612,[32]3.5925,[33]3.6128,[34]3.6825,[35]3.7690,[36]3.8324,[37]3.8488,[38]3.9179,[39]3.9915,[40]4.0080,[41]4.1107,[42]4.1595,[43]4.1632,[44]4.2063,[45]4.2882,[46]4.3783,[47]4.4178,[48]4.3284,[49]4.2875,[50]4.2994,[51]4.3276,[52]4.3616,[53]4.3623,[54]4.4175,[55]4.4187,[56]4.4593,[57]4.4816,[58]4.4899,[59]4.5085,[60]4.5380,[61]4.5876,[62]4.6140,[63]4.6560,[64]4.6818,[65]4.7487,[66]4.7651,[67]4.7661,[68]4.7452,[69]4.7803,[70]4.7607,[71]4.7550,[72]4.7353,[73]4.7528,[74]4.7325,[75]4.7589,[76]4.7021,[77]4.6174,[78]4.5631,[79]4.5884,[80]4.5600,[81]4.5283,[82]4.5066,[83]4.5554,[84]4.5468,[85]4.5478,[86]4.5527,[87]4.5630,[88]4.5960,[89]4.5843,[90]4.5997,[91]4.6079,[92]4.6117,[93]4.6121,[94]4.5999,[95]4.5996,[96]4.5989,[97]4.6317,[98]4.6543,[99]4.6722,[100]4.6610,[101]4.6571,[102]4.6214,[103]4.6491,[104]4.6806,[105]4.7210,[106]4.7218,[107]4.7718,[108]4.7948,[109]4.8123,[110]4.8576,[111]4.9026,[112]4.9283,[113]4.8756,[114]4.8660,[115]4.8650,[116]4.8644,[117]4.8784,[118]4.8847,[119]4.8757,[120]4.8642,[121]4.8603,[122]4.8578,[123]4.8265,[124]4.8200,[125]4.8079,[126]4.7936,[127]4.7799,[128]4.7869,
12
+ 3.07.250.900 I Final estimate: PPL = 4.7869 +/- 0.06785
13
+
14
+ finished_at_utc=2026-07-09T05:39:32Z
15
+ elapsed_seconds=189
16
+ -rw-r--r-- 1 root root 7.4G Jul 9 05:39 /hy3data/gguf/hy3/kld_eval/Hy3-BF16-wikitext2-c512-chunks128.kld
benchmarks/kld_wikitext2_c512_chunks128.json ADDED
@@ -0,0 +1,187 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "created_at_utc": "2026-07-09T06:30:26Z",
3
+ "base_model": "tencent/Hy3",
4
+ "reference_gguf": {
5
+ "file": "Hy3-BF16.gguf",
6
+ "local_size_bytes": 590197887712,
7
+ "no_mtp": true,
8
+ "tensor_count": 1199,
9
+ "standalone_ppl": 4.7869,
10
+ "standalone_ppl_stderr": 0.06785,
11
+ "log_file": "kld_eval/logs/base_bf16_wikitext2_c512_chunks128.log"
12
+ },
13
+ "eval": {
14
+ "corpus": "WikiText-2 raw test",
15
+ "corpus_source": "llama.cpp scripts/get-wikitext-2.sh, ggml-org/ci/wikitext-2-raw-v1.zip",
16
+ "corpus_file": "kld_eval/corpus/wikitext-2-raw/wiki.test.raw",
17
+ "context_size": 512,
18
+ "chunks": 128,
19
+ "nominal_tokens": 65536,
20
+ "batch_size": 1024,
21
+ "ubatch_size": 512,
22
+ "gpu_layers": 999,
23
+ "split_mode": "layer",
24
+ "tensor_split": "1,1,1,1,1,1,1,1",
25
+ "tool": "llama-perplexity",
26
+ "llama_cpp_build_commit": "56142c5f8",
27
+ "llama_cpp_build_number": 9909
28
+ },
29
+ "metrics": [
30
+ {
31
+ "artifact": "Hy3-IQ2_M-imatrix.gguf",
32
+ "short_label": "IQ2_M",
33
+ "model_type": "hy_v3 ?B IQ2_M - 2.7 bpw",
34
+ "model_size_bytes": 96976940288,
35
+ "model_size_gib": 90.32,
36
+ "effective_bits_per_weight": 2.629,
37
+ "prompt_tokens_per_second_avg": 1249.408577,
38
+ "generation_tokens_per_second_avg": 78.66486,
39
+ "ppl_q": 6.656017,
40
+ "ppl_q_stderr": 0.09928,
41
+ "ppl_base": 4.780718,
42
+ "ppl_base_stderr": 0.067547,
43
+ "ln_ppl_ratio": 0.330931,
44
+ "ln_ppl_ratio_stderr": 0.008325,
45
+ "ppl_ratio": 1.392263,
46
+ "ppl_ratio_stderr": 0.011591,
47
+ "delta_ppl": 1.8753,
48
+ "delta_ppl_stderr": 0.056504,
49
+ "mean_kld": 0.531406,
50
+ "mean_kld_stderr": 0.00592,
51
+ "rms_delta_p_percent": 24.788,
52
+ "rms_delta_p_percent_stderr": 0.186,
53
+ "same_top_p_percent": 74.749,
54
+ "same_top_p_percent_stderr": 0.24,
55
+ "mean_delta_p_percent": -7.734,
56
+ "mean_delta_p_percent_stderr": 0.13,
57
+ "ppl_correlation_percent": 83.7,
58
+ "kld_max": 16.845301,
59
+ "kld_p999": 10.952547,
60
+ "kld_p99": 5.220232,
61
+ "kld_p95": 2.358272,
62
+ "kld_p90": 1.421285,
63
+ "kld_median": 0.156656,
64
+ "log_file": "kld_eval/logs/Hy3-IQ2_M-imatrix_wikitext2_kld.log",
65
+ "started_at_utc": "2026-07-09T06:20:53Z",
66
+ "finished_at_utc": "2026-07-09T06:25:35Z",
67
+ "elapsed_seconds_including_download": 282
68
+ },
69
+ {
70
+ "artifact": "Hy3-Q3_K_L-imatrix.gguf",
71
+ "short_label": "Q3_K_L",
72
+ "model_type": "hy_v3 ?B Q3_K - Large",
73
+ "model_size_bytes": 153610415360,
74
+ "model_size_gib": 143.06,
75
+ "effective_bits_per_weight": 4.164,
76
+ "prompt_tokens_per_second_avg": 1348.49405,
77
+ "generation_tokens_per_second_avg": 63.323125,
78
+ "ppl_q": 5.357597,
79
+ "ppl_q_stderr": 0.07967,
80
+ "ppl_base": 4.780718,
81
+ "ppl_base_stderr": 0.067547,
82
+ "ln_ppl_ratio": 0.113925,
83
+ "ln_ppl_ratio_stderr": 0.004625,
84
+ "ppl_ratio": 1.120668,
85
+ "ppl_ratio_stderr": 0.005183,
86
+ "delta_ppl": 0.57688,
87
+ "delta_ppl_stderr": 0.026089,
88
+ "mean_kld": 0.16236,
89
+ "mean_kld_stderr": 0.002297,
90
+ "rms_delta_p_percent": 13.675,
91
+ "rms_delta_p_percent_stderr": 0.152,
92
+ "same_top_p_percent": 86.832,
93
+ "same_top_p_percent_stderr": 0.187,
94
+ "mean_delta_p_percent": -1.512,
95
+ "mean_delta_p_percent_stderr": 0.075,
96
+ "ppl_correlation_percent": 95.04,
97
+ "kld_max": 9.27753,
98
+ "kld_p999": 4.673191,
99
+ "kld_p99": 2.077389,
100
+ "kld_p95": 0.756051,
101
+ "kld_p90": 0.402834,
102
+ "kld_median": 0.031625,
103
+ "log_file": "kld_eval/logs/Hy3-Q3_K_L-imatrix_wikitext2_kld.log",
104
+ "started_at_utc": "2026-07-09T06:18:01Z",
105
+ "finished_at_utc": "2026-07-09T06:20:32Z",
106
+ "elapsed_seconds_including_download": 151
107
+ },
108
+ {
109
+ "artifact": "Hy3-Q4_K_M-imatrix.gguf",
110
+ "short_label": "Q4_K_M",
111
+ "model_type": "hy_v3 ?B Q4_K - Medium",
112
+ "model_size_bytes": 179397471488,
113
+ "model_size_gib": 167.08,
114
+ "effective_bits_per_weight": 4.863,
115
+ "prompt_tokens_per_second_avg": 1494.959574,
116
+ "generation_tokens_per_second_avg": 67.342185,
117
+ "ppl_q": 5.039535,
118
+ "ppl_q_stderr": 0.072751,
119
+ "ppl_base": 4.780718,
120
+ "ppl_base_stderr": 0.067547,
121
+ "ln_ppl_ratio": 0.052723,
122
+ "ln_ppl_ratio_stderr": 0.003434,
123
+ "ppl_ratio": 1.054138,
124
+ "ppl_ratio_stderr": 0.00362,
125
+ "delta_ppl": 0.258817,
126
+ "delta_ppl_stderr": 0.017576,
127
+ "mean_kld": 0.090429,
128
+ "mean_kld_stderr": 0.001394,
129
+ "rms_delta_p_percent": 10.559,
130
+ "rms_delta_p_percent_stderr": 0.134,
131
+ "same_top_p_percent": 89.997,
132
+ "same_top_p_percent_stderr": 0.166,
133
+ "mean_delta_p_percent": -0.821,
134
+ "mean_delta_p_percent_stderr": 0.058,
135
+ "ppl_correlation_percent": 97.13,
136
+ "kld_max": 7.732064,
137
+ "kld_p999": 2.956301,
138
+ "kld_p99": 1.174762,
139
+ "kld_p95": 0.406217,
140
+ "kld_p90": 0.215485,
141
+ "kld_median": 0.016753,
142
+ "log_file": "kld_eval/logs/Hy3-Q4_K_M-imatrix_wikitext2_kld.log",
143
+ "started_at_utc": "2026-07-09T05:52:43Z",
144
+ "finished_at_utc": "2026-07-09T05:59:20Z",
145
+ "elapsed_seconds_including_download": 397
146
+ },
147
+ {
148
+ "artifact": "Hy3-Q6_K-imatrix.gguf",
149
+ "short_label": "Q6_K",
150
+ "model_type": "hy_v3 ?B Q6_K",
151
+ "model_size_bytes": 242923187456,
152
+ "model_size_gib": 226.24,
153
+ "effective_bits_per_weight": 6.586,
154
+ "prompt_tokens_per_second_avg": 1232.390842,
155
+ "generation_tokens_per_second_avg": 57.385911,
156
+ "ppl_q": 4.770523,
157
+ "ppl_q_stderr": 0.067294,
158
+ "ppl_base": 4.780718,
159
+ "ppl_base_stderr": 0.067547,
160
+ "ln_ppl_ratio": -0.002135,
161
+ "ln_ppl_ratio_stderr": 0.001629,
162
+ "ppl_ratio": 0.997868,
163
+ "ppl_ratio_stderr": 0.001625,
164
+ "delta_ppl": -0.010195,
165
+ "delta_ppl_stderr": 0.007782,
166
+ "mean_kld": 0.020736,
167
+ "mean_kld_stderr": 0.000393,
168
+ "rms_delta_p_percent": 5.193,
169
+ "rms_delta_p_percent_stderr": 0.089,
170
+ "same_top_p_percent": 95.123,
171
+ "same_top_p_percent_stderr": 0.119,
172
+ "mean_delta_p_percent": -0.131,
173
+ "mean_delta_p_percent_stderr": 0.029,
174
+ "ppl_correlation_percent": 99.33,
175
+ "kld_max": 4.501527,
176
+ "kld_p999": 0.789244,
177
+ "kld_p99": 0.259334,
178
+ "kld_p95": 0.089398,
179
+ "kld_p90": 0.047906,
180
+ "kld_median": 0.003765,
181
+ "log_file": "kld_eval/logs/Hy3-Q6_K-imatrix_wikitext2_kld.log",
182
+ "started_at_utc": "2026-07-09T05:40:28Z",
183
+ "finished_at_utc": "2026-07-09T05:52:01Z",
184
+ "elapsed_seconds_including_download": 693
185
+ }
186
+ ]
187
+ }
scripts/update_hy3_card_from_kld.py ADDED
@@ -0,0 +1,516 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ from __future__ import annotations
3
+
4
+ import json
5
+ import math
6
+ import re
7
+ import shutil
8
+ from datetime import datetime, timezone
9
+ from pathlib import Path
10
+
11
+
12
+ ROOT = Path("/hy3data/gguf/hy3")
13
+ BENCH = ROOT / "benchmarks"
14
+ ASSETS = ROOT / "assets"
15
+ KLD_EVAL = ROOT / "kld_eval"
16
+ KLD_LOGS = KLD_EVAL / "logs"
17
+ OUT_LOGS = BENCH / "kld_logs"
18
+ KLD_JSON = BENCH / "kld_wikitext2_c512_chunks128.json"
19
+ MANIFEST_JSON = BENCH / "kld_eval_manifest.json"
20
+ EVAL_STATUS = BENCH / "eval_status.json"
21
+ README = ROOT / "README.md"
22
+
23
+ BASE_LOG = KLD_LOGS / "base_bf16_wikitext2_c512_chunks128.log"
24
+ QUANT_LOGS = {
25
+ "Hy3-IQ2_M-imatrix.gguf": KLD_LOGS / "Hy3-IQ2_M-imatrix_wikitext2_kld.log",
26
+ "Hy3-Q3_K_L-imatrix.gguf": KLD_LOGS / "Hy3-Q3_K_L-imatrix_wikitext2_kld.log",
27
+ "Hy3-Q4_K_M-imatrix.gguf": KLD_LOGS / "Hy3-Q4_K_M-imatrix_wikitext2_kld.log",
28
+ "Hy3-Q6_K-imatrix.gguf": KLD_LOGS / "Hy3-Q6_K-imatrix_wikitext2_kld.log",
29
+ }
30
+
31
+
32
+ def read_json(path: Path):
33
+ return json.loads(path.read_text())
34
+
35
+
36
+ def number_pair(text: str, pattern: str) -> tuple[float, float]:
37
+ match = re.search(pattern, text)
38
+ if not match:
39
+ raise ValueError(f"missing pattern: {pattern}")
40
+ return float(match.group(1)), float(match.group(2))
41
+
42
+
43
+ def number_value(text: str, pattern: str) -> float:
44
+ match = re.search(pattern, text)
45
+ if not match:
46
+ raise ValueError(f"missing pattern: {pattern}")
47
+ return float(match.group(1))
48
+
49
+
50
+ def parse_quant_log(path: Path) -> dict:
51
+ text = path.read_text(errors="replace")
52
+ pm = r"(?:\u00b1|\+/-)"
53
+ metrics = {
54
+ "ppl_q": number_pair(text, rf"Mean PPL\(Q\)\s*:\s*([\-0-9.]+)\s*{pm}\s*([\-0-9.]+)"),
55
+ "ppl_base": number_pair(text, rf"Mean PPL\(base\)\s*:\s*([\-0-9.]+)\s*{pm}\s*([\-0-9.]+)"),
56
+ "ln_ppl_ratio": number_pair(text, rf"Mean ln\(PPL\(Q\)/PPL\(base\)\)\s*:\s*([\-0-9.]+)\s*{pm}\s*([\-0-9.]+)"),
57
+ "ppl_ratio": number_pair(text, rf"Mean PPL\(Q\)/PPL\(base\)\s*:\s*([\-0-9.]+)\s*{pm}\s*([\-0-9.]+)"),
58
+ "delta_ppl": number_pair(text, rf"Mean PPL\(Q\)-PPL\(base\)\s*:\s*([\-0-9.]+)\s*{pm}\s*([\-0-9.]+)"),
59
+ "mean_kld": number_pair(text, rf"Mean\s+KLD:\s*([\-0-9.]+)\s*{pm}\s*([\-0-9.]+)"),
60
+ "rms_delta_p_percent": number_pair(text, rf"RMS\s+\S*p\s*:\s*([\-0-9.]+)\s*{pm}\s*([\-0-9.]+)\s*%"),
61
+ "same_top_p_percent": number_pair(text, rf"Same top p:\s*([\-0-9.]+)\s*{pm}\s*([\-0-9.]+)\s*%"),
62
+ "mean_delta_p_percent": number_pair(text, rf"Mean\s+\S*p:\s*([\-0-9.]+)\s*{pm}\s*([\-0-9.]+)\s*%"),
63
+ "ppl_correlation_percent": number_value(text, r"Cor\(ln\(PPL\(Q\)\), ln\(PPL\(base\)\)\):\s*([\-0-9.]+)%"),
64
+ "kld_max": number_value(text, r"Maximum KLD:\s*([\-0-9.]+)"),
65
+ "kld_p999": number_value(text, r"99\.9%\s+KLD:\s*([\-0-9.]+)"),
66
+ "kld_p99": number_value(text, r"99\.0%\s+KLD:\s*([\-0-9.]+)"),
67
+ "kld_p95": number_value(text, r"95\.0%\s+KLD:\s*([\-0-9.]+)"),
68
+ "kld_p90": number_value(text, r"90\.0%\s+KLD:\s*([\-0-9.]+)"),
69
+ "kld_median": number_value(text, r"Median\s+KLD:\s*([\-0-9.]+)"),
70
+ }
71
+ started = re.search(r"started_at_utc=([0-9T:Z-]+)", text)
72
+ finished = re.search(r"finished_at_utc=([0-9T:Z-]+)", text)
73
+ elapsed = re.search(r"elapsed_seconds=([0-9]+)", text)
74
+ return {
75
+ "log_file": str(path.relative_to(ROOT)),
76
+ "started_at_utc": started.group(1) if started else None,
77
+ "finished_at_utc": finished.group(1) if finished else None,
78
+ "elapsed_seconds_including_download": int(elapsed.group(1)) if elapsed else None,
79
+ "metrics": flatten_pairs(metrics),
80
+ }
81
+
82
+
83
+ def flatten_pairs(metrics: dict) -> dict:
84
+ out = {}
85
+ for key, value in metrics.items():
86
+ if isinstance(value, tuple):
87
+ out[key] = value[0]
88
+ out[f"{key}_stderr"] = value[1]
89
+ else:
90
+ out[key] = value
91
+ return out
92
+
93
+
94
+ def parse_base_log(path: Path) -> dict:
95
+ text = path.read_text(errors="replace")
96
+ match = re.search(r"Final estimate: PPL =\s*([0-9.]+)\s*(?:\u00b1|\+/-)\s*([0-9.]+)", text)
97
+ if not match:
98
+ raise ValueError("missing BF16 base PPL")
99
+ elapsed = re.search(r"elapsed_seconds=([0-9]+)", text)
100
+ return {
101
+ "log_file": str(path.relative_to(ROOT)),
102
+ "standalone_ppl": float(match.group(1)),
103
+ "standalone_ppl_stderr": float(match.group(2)),
104
+ "elapsed_seconds": int(elapsed.group(1)) if elapsed else None,
105
+ }
106
+
107
+
108
+ def fmt(value: float, digits: int = 3) -> str:
109
+ return f"{value:.{digits}f}"
110
+
111
+
112
+ def fmt6(value: float) -> str:
113
+ return f"{value:.6f}"
114
+
115
+
116
+ def bar_svg(title: str, filename: str, rows: list[dict], key: str, suffix: str = "", lower_better: bool = False) -> None:
117
+ ASSETS.mkdir(parents=True, exist_ok=True)
118
+ width, height = 820, 360
119
+ margin_l, margin_r, margin_t, margin_b = 180, 52, 58, 48
120
+ plot_w = width - margin_l - margin_r
121
+ bar_h = 34
122
+ gap = 22
123
+ max_value = max(row[key] for row in rows) or 1.0
124
+ palette = ["#3b6ea8", "#637c35", "#9a6735", "#7b5ea7"]
125
+ lines = [
126
+ f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" viewBox="0 0 {width} {height}">',
127
+ '<rect width="100%" height="100%" fill="#ffffff"/>',
128
+ f'<text x="{margin_l}" y="30" font-family="Arial, sans-serif" font-size="20" font-weight="700" fill="#202124">{title}</text>',
129
+ f'<line x1="{margin_l}" y1="{height - margin_b}" x2="{width - margin_r}" y2="{height - margin_b}" stroke="#d7dce2" stroke-width="1"/>',
130
+ ]
131
+ for i, row in enumerate(rows):
132
+ y = margin_t + i * (bar_h + gap)
133
+ value = row[key]
134
+ bar_w = max(2, plot_w * value / max_value)
135
+ label = row["short_label"]
136
+ color = palette[i % len(palette)]
137
+ lines.append(f'<text x="{margin_l - 14}" y="{y + 23}" text-anchor="end" font-family="Arial, sans-serif" font-size="14" fill="#30343b">{label}</text>')
138
+ lines.append(f'<rect x="{margin_l}" y="{y}" width="{bar_w:.1f}" height="{bar_h}" rx="4" fill="{color}"/>')
139
+ value_text = f"{fmt(value, 3)}{suffix}" if value < 100 else f"{fmt(value, 2)}{suffix}"
140
+ lines.append(f'<text x="{margin_l + bar_w + 10:.1f}" y="{y + 22}" font-family="Arial, sans-serif" font-size="14" fill="#202124">{value_text}</text>')
141
+ hint = "Lower is better" if lower_better else "Higher is better"
142
+ lines.append(f'<text x="{margin_l}" y="{height - 15}" font-family="Arial, sans-serif" font-size="12" fill="#5f6368">{hint}. WikiText-2, 128 chunks, ctx 512.</text>')
143
+ lines.append("</svg>")
144
+ (ASSETS / filename).write_text("\n".join(lines) + "\n")
145
+
146
+
147
+ def scatter_svg(rows: list[dict]) -> None:
148
+ width, height = 820, 430
149
+ ml, mr, mt, mb = 90, 44, 54, 74
150
+ pw, ph = width - ml - mr, height - mt - mb
151
+ max_x = max(row["model_size_gib"] for row in rows) * 1.08
152
+ max_y = max(row["mean_kld"] for row in rows) * 1.15
153
+ palette = ["#3b6ea8", "#637c35", "#9a6735", "#7b5ea7"]
154
+ lines = [
155
+ f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" viewBox="0 0 {width} {height}">',
156
+ '<rect width="100%" height="100%" fill="#ffffff"/>',
157
+ f'<text x="{ml}" y="30" font-family="Arial, sans-serif" font-size="20" font-weight="700" fill="#202124">Size vs mean KLD</text>',
158
+ f'<line x1="{ml}" y1="{height - mb}" x2="{width - mr}" y2="{height - mb}" stroke="#d7dce2"/>',
159
+ f'<line x1="{ml}" y1="{mt}" x2="{ml}" y2="{height - mb}" stroke="#d7dce2"/>',
160
+ f'<text x="{width / 2}" y="{height - 24}" text-anchor="middle" font-family="Arial, sans-serif" font-size="13" fill="#30343b">Artifact size (GiB)</text>',
161
+ f'<text x="22" y="{height / 2}" text-anchor="middle" transform="rotate(-90 22 {height / 2})" font-family="Arial, sans-serif" font-size="13" fill="#30343b">Mean KLD vs BF16</text>',
162
+ ]
163
+ for idx in range(6):
164
+ x_val = max_x * idx / 5
165
+ x = ml + pw * x_val / max_x
166
+ lines.append(f'<line x1="{x:.1f}" y1="{height - mb}" x2="{x:.1f}" y2="{height - mb + 5}" stroke="#aab0b8"/>')
167
+ lines.append(f'<text x="{x:.1f}" y="{height - mb + 22}" text-anchor="middle" font-family="Arial, sans-serif" font-size="11" fill="#5f6368">{x_val:.0f}</text>')
168
+ for idx in range(6):
169
+ y_val = max_y * idx / 5
170
+ y = height - mb - ph * y_val / max_y
171
+ lines.append(f'<line x1="{ml - 5}" y1="{y:.1f}" x2="{ml}" y2="{y:.1f}" stroke="#aab0b8"/>')
172
+ lines.append(f'<text x="{ml - 10}" y="{y + 4:.1f}" text-anchor="end" font-family="Arial, sans-serif" font-size="11" fill="#5f6368">{y_val:.2f}</text>')
173
+ for i, row in enumerate(rows):
174
+ x = ml + pw * row["model_size_gib"] / max_x
175
+ y = height - mb - ph * row["mean_kld"] / max_y
176
+ color = palette[i % len(palette)]
177
+ lines.append(f'<circle cx="{x:.1f}" cy="{y:.1f}" r="7" fill="{color}"/>')
178
+ lines.append(f'<text x="{x + 11:.1f}" y="{y - 10:.1f}" font-family="Arial, sans-serif" font-size="12" fill="#202124">{row["short_label"]}</text>')
179
+ lines.append(f'<text x="{ml}" y="{height - 8}" font-family="Arial, sans-serif" font-size="12" fill="#5f6368">Lower KLD is closer to BF16. WikiText-2, 128 chunks, ctx 512.</text>')
180
+ lines.append("</svg>")
181
+ (ASSETS / "size_vs_kld.svg").write_text("\n".join(lines) + "\n")
182
+
183
+
184
+ def md_table(headers: list[str], rows: list[list[str]]) -> str:
185
+ out = ["| " + " | ".join(headers) + " |", "| " + " | ".join(["---"] * len(headers)) + " |"]
186
+ out.extend("| " + " | ".join(row) + " |" for row in rows)
187
+ return "\n".join(out)
188
+
189
+
190
+ def copy_logs() -> None:
191
+ OUT_LOGS.mkdir(parents=True, exist_ok=True)
192
+ for path in [BASE_LOG, *QUANT_LOGS.values()]:
193
+ shutil.copy2(path, OUT_LOGS / path.name)
194
+
195
+
196
+ def main() -> None:
197
+ BENCH.mkdir(parents=True, exist_ok=True)
198
+ ASSETS.mkdir(parents=True, exist_ok=True)
199
+ summary = read_json(BENCH / "summary.json")
200
+ throughput_by_artifact = {row["artifact"]: row for row in summary}
201
+ short_labels = {
202
+ "Hy3-IQ2_M-imatrix.gguf": "IQ2_M",
203
+ "Hy3-Q3_K_L-imatrix.gguf": "Q3_K_L",
204
+ "Hy3-Q4_K_M-imatrix.gguf": "Q4_K_M",
205
+ "Hy3-Q6_K-imatrix.gguf": "Q6_K",
206
+ }
207
+ base = parse_base_log(BASE_LOG)
208
+ rows = []
209
+ for artifact, log_path in QUANT_LOGS.items():
210
+ parsed = parse_quant_log(log_path)
211
+ bench = throughput_by_artifact[artifact]
212
+ row = {
213
+ "artifact": artifact,
214
+ "short_label": short_labels[artifact],
215
+ "model_type": bench["model_type"],
216
+ "model_size_bytes": bench["model_size_bytes"],
217
+ "model_size_gib": bench["model_size_gib"],
218
+ "effective_bits_per_weight": round(bench["model_size_bytes"] / 590197887712 * 16, 3),
219
+ "prompt_tokens_per_second_avg": bench["prompt_tokens_per_second_avg"],
220
+ "generation_tokens_per_second_avg": bench["generation_tokens_per_second_avg"],
221
+ **parsed["metrics"],
222
+ "log_file": parsed["log_file"],
223
+ "started_at_utc": parsed["started_at_utc"],
224
+ "finished_at_utc": parsed["finished_at_utc"],
225
+ "elapsed_seconds_including_download": parsed["elapsed_seconds_including_download"],
226
+ }
227
+ rows.append(row)
228
+ rows.sort(key=lambda r: r["model_size_gib"])
229
+
230
+ kld_payload = {
231
+ "created_at_utc": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
232
+ "base_model": "tencent/Hy3",
233
+ "reference_gguf": {
234
+ "file": "Hy3-BF16.gguf",
235
+ "local_size_bytes": 590197887712,
236
+ "no_mtp": True,
237
+ "tensor_count": 1199,
238
+ "standalone_ppl": base["standalone_ppl"],
239
+ "standalone_ppl_stderr": base["standalone_ppl_stderr"],
240
+ "log_file": base["log_file"],
241
+ },
242
+ "eval": {
243
+ "corpus": "WikiText-2 raw test",
244
+ "corpus_source": "llama.cpp scripts/get-wikitext-2.sh, ggml-org/ci/wikitext-2-raw-v1.zip",
245
+ "corpus_file": "kld_eval/corpus/wikitext-2-raw/wiki.test.raw",
246
+ "context_size": 512,
247
+ "chunks": 128,
248
+ "nominal_tokens": 65536,
249
+ "batch_size": 1024,
250
+ "ubatch_size": 512,
251
+ "gpu_layers": 999,
252
+ "split_mode": "layer",
253
+ "tensor_split": "1,1,1,1,1,1,1,1",
254
+ "tool": "llama-perplexity",
255
+ "llama_cpp_build_commit": "56142c5f8",
256
+ "llama_cpp_build_number": 9909,
257
+ },
258
+ "metrics": rows,
259
+ }
260
+ KLD_JSON.write_text(json.dumps(kld_payload, indent=2) + "\n")
261
+
262
+ manifest = {
263
+ "created_at_utc": kld_payload["created_at_utc"],
264
+ "purpose": "Full-logit quantization quality comparison against BF16 no-MTP GGUF reference.",
265
+ "reference_logits_file": {
266
+ "local_path": "kld_eval/Hy3-BF16-wikitext2-c512-chunks128.kld",
267
+ "size_note": "Not uploaded to Hugging Face because it is a large intermediate binary.",
268
+ },
269
+ "commands": {
270
+ "bf16_reference": "llama-perplexity -m Hy3-BF16.gguf -f wiki.test.raw -c 512 --chunks 128 -b 1024 -ub 512 -ngl 999 --split-mode layer --tensor-split 1,1,1,1,1,1,1,1 --kl-divergence-base Hy3-BF16-wikitext2-c512-chunks128.kld",
271
+ "quant_compare": "llama-perplexity -m <quant.gguf> -f wiki.test.raw -c 512 --chunks 128 -b 1024 -ub 512 -ngl 999 --split-mode layer --tensor-split 1,1,1,1,1,1,1,1 --kl-divergence-base Hy3-BF16-wikitext2-c512-chunks128.kld --kl-divergence",
272
+ },
273
+ "logs": [f"benchmarks/kld_logs/{p.name}" for p in [BASE_LOG, *QUANT_LOGS.values()]],
274
+ }
275
+ MANIFEST_JSON.write_text(json.dumps(manifest, indent=2) + "\n")
276
+ copy_logs()
277
+
278
+ chart_rows = sorted(rows, key=lambda r: r["model_size_gib"])
279
+ bar_svg("Mean KLD vs BF16", "mean_kld.svg", chart_rows, "mean_kld", lower_better=True)
280
+ bar_svg("PPL ratio vs BF16", "ppl_ratio.svg", chart_rows, "ppl_ratio", lower_better=True)
281
+ bar_svg("Same top-p agreement", "same_top_p.svg", chart_rows, "same_top_p_percent", "%", lower_better=False)
282
+ bar_svg("RMS token-probability delta", "rms_delta_p.svg", chart_rows, "rms_delta_p_percent", "%", lower_better=True)
283
+ scatter_svg(chart_rows)
284
+
285
+ measured = [
286
+ "artifact_size_bytes",
287
+ "effective_bits_per_weight",
288
+ "compression_vs_bf16_no_mtp",
289
+ "llama_bench_prompt_tokens_per_second",
290
+ "llama_bench_generation_tokens_per_second",
291
+ "wiki_text_2_perplexity",
292
+ "kld_vs_bf16_reference_logits",
293
+ "ppl_ratio_vs_bf16",
294
+ "mean_delta_token_probability",
295
+ "rms_delta_token_probability",
296
+ "same_top_p_agreement",
297
+ ]
298
+ eval_status = {
299
+ "created_at_utc": kld_payload["created_at_utc"],
300
+ "benchmark_status": "complete",
301
+ "measured_metrics": measured,
302
+ "quality_eval": {
303
+ "status": "complete",
304
+ "summary_file": "benchmarks/kld_wikitext2_c512_chunks128.json",
305
+ "manifest_file": "benchmarks/kld_eval_manifest.json",
306
+ "corpus": "WikiText-2 raw test",
307
+ "chunks": 128,
308
+ "context_size": 512,
309
+ "nominal_tokens": 65536,
310
+ "reference": "BF16 no-MTP GGUF generated from tencent/Hy3",
311
+ },
312
+ "not_reported_metrics": {
313
+ "full_downstream_task_accuracy": {
314
+ "status": "not_measured_in_this_release",
315
+ "reason": "This release focuses on GGUF artifact quality and serving throughput. Downstream task harnesses such as MMLU, GPQA, SWE-bench, or coding evals were not run for these GGUF files.",
316
+ },
317
+ "full_wikitext2_file": {
318
+ "status": "bounded_slice",
319
+ "reason": "KLD uses a 128 chunk x 512 context slice to keep BF16 reference logits manageable for a 295B MoE model.",
320
+ },
321
+ },
322
+ }
323
+ EVAL_STATUS.write_text(json.dumps(eval_status, indent=2) + "\n")
324
+
325
+ quality_rows = []
326
+ for row in sorted(rows, key=lambda r: r["model_size_gib"]):
327
+ quality_rows.append([
328
+ f"`{row['artifact']}`",
329
+ fmt(row["model_size_gib"], 2),
330
+ fmt(row["effective_bits_per_weight"], 2),
331
+ f"{fmt6(row['ppl_q'])} +/- {fmt6(row['ppl_q_stderr'])}",
332
+ f"{fmt6(row['ppl_ratio'])} +/- {fmt6(row['ppl_ratio_stderr'])}",
333
+ f"{fmt6(row['mean_kld'])} +/- {fmt6(row['mean_kld_stderr'])}",
334
+ fmt6(row["kld_median"]),
335
+ f"{fmt(row['rms_delta_p_percent'], 3)} +/- {fmt(row['rms_delta_p_percent_stderr'], 3)}%",
336
+ f"{fmt(row['same_top_p_percent'], 3)} +/- {fmt(row['same_top_p_percent_stderr'], 3)}%",
337
+ ])
338
+
339
+ throughput_rows = []
340
+ for row in sorted(rows, key=lambda r: r["model_size_gib"]):
341
+ compression = 549.66 / row["model_size_gib"]
342
+ throughput_rows.append([
343
+ f"`{row['artifact']}`",
344
+ row["model_type"],
345
+ fmt(row["model_size_gib"], 2),
346
+ fmt(row["effective_bits_per_weight"], 2),
347
+ f"{compression:.2f}x",
348
+ fmt(row["prompt_tokens_per_second_avg"], 2),
349
+ fmt(row["generation_tokens_per_second_avg"], 2),
350
+ ])
351
+
352
+ readme = f"""---
353
+ license: apache-2.0
354
+ base_model: tencent/Hy3
355
+ library_name: llama.cpp
356
+ tags:
357
+ - gguf
358
+ - hy3
359
+ - hunyuan
360
+ - quantized
361
+ - imatrix
362
+ - llama.cpp
363
+ - mixture-of-experts
364
+ ---
365
+
366
+ # Hy3 GGUF Quants
367
+
368
+ High-quality GGUF quantizations for [`tencent/Hy3`](https://huggingface.co/tencent/Hy3), produced with the Hy3-enabled `llama.cpp` branch. This repo contains 2-bit, 3-bit, 4-bit, and 6-bit-class imatrix GGUFs. Q8 was intentionally skipped by request.
369
+
370
+ Hy3 is a 295B-parameter MoE model with 21B active parameters. These GGUFs are converted from the BF16 base model with the MTP/NextN block excluded for normal serving.
371
+
372
+ ## Files
373
+
374
+ - `Hy3-IQ2_M-imatrix.gguf`
375
+ - `Hy3-Q3_K_L-imatrix.gguf`
376
+ - `Hy3-Q4_K_M-imatrix.gguf`
377
+ - `Hy3-Q6_K-imatrix.gguf`
378
+ - `calibration/hy3_imatrix_calibration.txt`
379
+ - `imatrix/Hy3-imatrix.gguf`
380
+ - `benchmarks/summary.json`
381
+ - `benchmarks/kld_wikitext2_c512_chunks128.json`
382
+ - `benchmarks/kld_logs/*.log`
383
+ - `assets/*.svg`
384
+
385
+ ## Recommended Files
386
+
387
+ | Use case | Recommended file | Why |
388
+ |---|---|---|
389
+ | Highest quality GGUF | `Hy3-Q6_K-imatrix.gguf` | Lowest measured KLD in this set: {fmt6(next(r for r in rows if r['artifact'] == 'Hy3-Q6_K-imatrix.gguf')['mean_kld'])}; same top-p {fmt(next(r for r in rows if r['artifact'] == 'Hy3-Q6_K-imatrix.gguf')['same_top_p_percent'], 3)}%. |
390
+ | Default portable GGUF | `Hy3-Q4_K_M-imatrix.gguf` | Best balance of quality, size, and throughput if 167 GiB fits. |
391
+ | Smaller fallback | `Hy3-Q3_K_L-imatrix.gguf` | Use when Q4 does not fit; quality loss is materially higher than Q4. |
392
+ | Minimum size | `Hy3-IQ2_M-imatrix.gguf` | Smallest artifact. Use only when size is the primary constraint. |
393
+
394
+ ## Throughput
395
+
396
+ `llama-bench` settings: CUDA backend, 8x NVIDIA RTX PRO 6000 Blackwell Server Edition, `-ngl 999`, `--split-mode layer`, `--tensor-split 1/1/1/1/1/1/1/1`, prompt length `512`, generation length `128`, reps `3`, batch `1024`, ubatch `512`, F16 KV cache.
397
+
398
+ BF16 no-MTP GGUF reference size used for compression ratios: `549.66 GiB` (`590197887712` bytes).
399
+
400
+ {md_table(["Artifact", "llama.cpp type", "Size GiB", "Effective BPW", "Compression", "Prompt tok/s", "Generate tok/s"], throughput_rows)}
401
+
402
+ Raw throughput data: [`benchmarks/summary.json`](benchmarks/summary.json).
403
+
404
+ ## Quality Metrics
405
+
406
+ Quality was measured with `llama-perplexity --kl-divergence` against a BF16 no-MTP GGUF reference generated from `tencent/Hy3`.
407
+
408
+ Eval setup:
409
+
410
+ - Corpus: WikiText-2 raw test from the llama.cpp `scripts/get-wikitext-2.sh` helper.
411
+ - Slice: 128 chunks at context 512, nominally 65,536 tokens.
412
+ - Runtime flags: `-c 512 --chunks 128 -b 1024 -ub 512 -ngl 999 --split-mode layer --tensor-split 1,1,1,1,1,1,1,1`.
413
+ - The BF16 reference logits file was kept local as a large intermediate and is not uploaded.
414
+
415
+ {md_table(["Artifact", "Size GiB", "BPW", "PPL(Q)", "PPL ratio", "Mean KLD", "Median KLD", "RMS delta p", "Same top-p"], quality_rows)}
416
+
417
+ Lower KLD, lower PPL ratio, and lower RMS delta p are better. Higher same top-p is better. In llama.cpp's KLD output, "same top p" is the percentage of positions where the quant and BF16 reference agree on the highest-probability token.
418
+
419
+ Raw quality data:
420
+
421
+ - [`benchmarks/kld_wikitext2_c512_chunks128.json`](benchmarks/kld_wikitext2_c512_chunks128.json)
422
+ - [`benchmarks/kld_eval_manifest.json`](benchmarks/kld_eval_manifest.json)
423
+ - [`benchmarks/kld_logs/`](benchmarks/kld_logs/)
424
+
425
+ ## Charts
426
+
427
+ ![Mean KLD](assets/mean_kld.svg)
428
+
429
+ ![PPL ratio](assets/ppl_ratio.svg)
430
+
431
+ ![Same top-p](assets/same_top_p.svg)
432
+
433
+ ![RMS delta p](assets/rms_delta_p.svg)
434
+
435
+ ![Size vs KLD](assets/size_vs_kld.svg)
436
+
437
+ ![Generation throughput](assets/generation_tps.svg)
438
+
439
+ ![Prompt throughput](assets/prompt_tps.svg)
440
+
441
+ ![Model size](assets/model_size.svg)
442
+
443
+ ![Compression](assets/compression.svg)
444
+
445
+ ## Production Serving Reference
446
+
447
+ These GGUFs are portable llama.cpp artifacts. For maximum throughput on the tested 8-GPU RunPod machine, the faster serving profile was a separate FP8 SGLang path rather than GGUF:
448
+
449
+ - SGLang 0.5.5 Hy3 backport, tensor parallel `TP=8`
450
+ - FP8 weights and FP8 E4M3 KV cache with calibrated K/V scales
451
+ - CUDA graph batch sizes `[1,2,4,8,16,32]`
452
+ - Best measured batched result: `688.76` aggregate output tok/s at concurrency `32` over 128 successful requests
453
+
454
+ Raw serving reference: [`benchmarks/sglang_fp8_tp8_reference.json`](benchmarks/sglang_fp8_tp8_reference.json).
455
+
456
+ Tencent's base model card provides both vLLM and SGLang recipes for Hy3. On this tested pod, the practical production recommendation is SGLang FP8 TP=8 for throughput; use GGUF llama.cpp for portability or memory/packaging constraints.
457
+
458
+ ## llama.cpp Serving Example
459
+
460
+ Use `layer` split for this Hy3 GGUF backend. `tensor` split crashed in CUDA decode during testing.
461
+
462
+ ```bash
463
+ export HF_TOKEN=...
464
+
465
+ llama-server \\
466
+ --hf-repo LordNeel/Hy3-GGUF \\
467
+ --hf-file Hy3-Q4_K_M-imatrix.gguf \\
468
+ -ngl 999 \\
469
+ --split-mode layer \\
470
+ --tensor-split 1,1,1,1,1,1,1,1 \\
471
+ -c 8192 \\
472
+ -b 1024 \\
473
+ -ub 512
474
+ ```
475
+
476
+ ## Quantization Recipe
477
+
478
+ - Base model: `tencent/Hy3`
479
+ - Conversion: Hy3-enabled `llama.cpp`, build `9909`, commit `56142c5f8`
480
+ - Source GGUF: BF16, `--no-mtp`, `--fuse-gate-up-exps`
481
+ - Tensor count in no-MTP BF16 GGUF: 1199
482
+ - Imatrix: `llama-imatrix`, 96 chunks, calibration text from `hy3_calibration_512.jsonl`
483
+ - Q8: skipped by request
484
+
485
+ ```bash
486
+ llama-quantize \\
487
+ --imatrix Hy3-imatrix.gguf \\
488
+ --leave-output-tensor \\
489
+ --token-embedding-type q8_0 \\
490
+ Hy3-BF16.gguf \\
491
+ Hy3-Q4_K_M-imatrix.gguf \\
492
+ Q4_K_M \\
493
+ 64
494
+ ```
495
+
496
+ The reproduction runner used for the upload is available at [`scripts/run_hy3_gguf_stream_upload.sh`](scripts/run_hy3_gguf_stream_upload.sh).
497
+
498
+ ## Known Limitations
499
+
500
+ - KLD/PPL numbers are from a bounded WikiText-2 slice, not a full downstream task benchmark.
501
+ - The uploaded GGUFs exclude the MTP/NextN block. That is intentional for normal serving.
502
+ - `llama.cpp --split-mode tensor` was unstable for this Hy3 GGUF path on the tested CUDA stack; use `--split-mode layer`.
503
+ - GPTQ4/SGLang was tested separately and classified invalid because generation collapsed into repeated punctuation. Those artifacts are not published here.
504
+
505
+ ## Hardware
506
+
507
+ - 8x NVIDIA RTX PRO 6000 Blackwell Server Edition, about 97 GB VRAM each
508
+ - CUDA 12.8 / driver 570.195.03
509
+ - AMD EPYC 9B14 host, 224 logical CPUs
510
+ - About 1.5 TiB system RAM
511
+ """
512
+ README.write_text(readme)
513
+
514
+
515
+ if __name__ == "__main__":
516
+ main()