Clarify INT8 quantization in card and metadata
Browse files
README.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
-
base_model:
|
| 4 |
-
|
| 5 |
pipeline_tag: text-generation
|
| 6 |
library_name: coreai
|
| 7 |
tags:
|
|
@@ -9,17 +9,35 @@ tags:
|
|
| 9 |
- apple-silicon
|
| 10 |
- recurrent-transformer
|
| 11 |
- text-generation
|
|
|
|
|
|
|
| 12 |
---
|
| 13 |
|
| 14 |
-
# Nanbeige4.2-3B — Core AI
|
| 15 |
|
| 16 |
-
Core AI
|
| 17 |
|
| 18 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
## Architecture
|
| 21 |
|
| 22 |
-
Nanbeige4.2 is a looped Llama: **22 physical transformer blocks execute twice**, with RMS normalization after each pass and independent KV history. The bundle therefore has **44 executed/cache layers while storing one copy of the 22 physical blocks**.
|
| 23 |
|
| 24 |
## Bundle
|
| 25 |
|
|
@@ -33,7 +51,7 @@ gpu-pipelined/
|
|
| 33 |
tokenizer/
|
| 34 |
```
|
| 35 |
|
| 36 |
-
|
| 37 |
|
| 38 |
## Verification
|
| 39 |
|
|
@@ -41,7 +59,6 @@ Configuration: static-S=1, int8 body per-block-32, symmetric absmax int8 untied
|
|
| 41 |
- Quantized Core AI engine: token-exact vs fp32 on 24-token Paris, 16-token freezing-point, and 64-token decimal-reasoning gates.
|
| 42 |
- M4 Max Release benchmark, prompt 128 / generation 256 / 3 runs: **47.37 prefill / 46.35 decode tok/s**.
|
| 43 |
- 4,096-token boundary: **29.83 prefill / 32.80 decode tok/s**, 9.17 GiB peak RSS, zero swaps.
|
| 44 |
-
- int4 and mixed int4/int8 candidates failed the reasoning quality gate and are intentionally not published.
|
| 45 |
- iPhone `h18p` device acceptance is pending; no iPhone performance claim is made.
|
| 46 |
|
| 47 |
## License
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
base_model: Nanbeige/Nanbeige4.2-3B
|
| 4 |
+
base_model_relation: quantized
|
| 5 |
pipeline_tag: text-generation
|
| 6 |
library_name: coreai
|
| 7 |
tags:
|
|
|
|
| 9 |
- apple-silicon
|
| 10 |
- recurrent-transformer
|
| 11 |
- text-generation
|
| 12 |
+
- quantized
|
| 13 |
+
- int8
|
| 14 |
---
|
| 15 |
|
| 16 |
+
# Nanbeige4.2-3B — Core AI INT8
|
| 17 |
|
| 18 |
+
**INT8 weight-quantized Core AI LanguageBundle** converted from [`Nanbeige/Nanbeige4.2-3B`](https://huggingface.co/Nanbeige/Nanbeige4.2-3B), pinned to source revision [`5ff54fb7ed86ce8e216d78bff5417ab9981de3d4`](https://huggingface.co/Nanbeige/Nanbeige4.2-3B/tree/5ff54fb7ed86ce8e216d78bff5417ab9981de3d4).
|
| 19 |
|
| 20 |
+
This is a Core AI `.aimodel` bundle for Apple silicon. It is not a Transformers, GGUF, or MLX checkpoint.
|
| 21 |
+
|
| 22 |
+
Ported by [Vadim Smirnov (@ukint-vs)](https://github.com/ukint-vs). Source, reproducible conversion recipe, parity gates, and the optimization report are tracked in [john-rocky/coreai-model-zoo#5](https://github.com/john-rocky/coreai-model-zoo/issues/5).
|
| 23 |
+
|
| 24 |
+
## Format
|
| 25 |
+
|
| 26 |
+
| Property | Value |
|
| 27 |
+
|---|---|
|
| 28 |
+
| Published variant | **INT8 only** |
|
| 29 |
+
| Body weights | int8, per-block 32 (`int8hu`) |
|
| 30 |
+
| Untied LM head | symmetric absmax int8, per-block 32 (`--head-sym`) |
|
| 31 |
+
| Activations / KV cache | fp16 |
|
| 32 |
+
| Decode input | static sequence length 1 (`--static-ids`) |
|
| 33 |
+
| Bundle size | approximately 4.59 GiB |
|
| 34 |
+
| Verified context | 4,096 tokens |
|
| 35 |
+
|
| 36 |
+
The int4 and mixed int4/int8 candidates failed the reasoning quality gate and are intentionally not published.
|
| 37 |
|
| 38 |
## Architecture
|
| 39 |
|
| 40 |
+
Nanbeige4.2 is a looped Llama: **22 physical transformer blocks execute twice**, with RMS normalization after each pass and independent KV history. The bundle therefore has **44 executed/cache layers while storing and quantizing one copy of the 22 physical blocks**.
|
| 41 |
|
| 42 |
## Bundle
|
| 43 |
|
|
|
|
| 51 |
tokenizer/
|
| 52 |
```
|
| 53 |
|
| 54 |
+
The embedded tokenizer includes the vendor `chat_template.jinja` with thinking enabled and disabled modes. The `.aimodel` producer is `coreai-core 1.0.0b2`.
|
| 55 |
|
| 56 |
## Verification
|
| 57 |
|
|
|
|
| 59 |
- Quantized Core AI engine: token-exact vs fp32 on 24-token Paris, 16-token freezing-point, and 64-token decimal-reasoning gates.
|
| 60 |
- M4 Max Release benchmark, prompt 128 / generation 256 / 3 runs: **47.37 prefill / 46.35 decode tok/s**.
|
| 61 |
- 4,096-token boundary: **29.83 prefill / 32.80 decode tok/s**, 9.17 GiB peak RSS, zero swaps.
|
|
|
|
| 62 |
- iPhone `h18p` device acceptance is pending; no iPhone performance claim is made.
|
| 63 |
|
| 64 |
## License
|
gpu-pipelined/nanbeige4_2_3b_decode_int8hu_block32_sym_s1/metadata.json
CHANGED
|
@@ -21,9 +21,11 @@
|
|
| 21 |
"hf_model_id": "Nanbeige/Nanbeige4.2-3B",
|
| 22 |
"hf_revision": "5ff54fb7ed86ce8e216d78bff5417ab9981de3d4"
|
| 23 |
},
|
| 24 |
-
"compression":
|
|
|
|
|
|
|
| 25 |
"compilation": {
|
| 26 |
"date": "2026-07-22T15:06:28.708628+00:00",
|
| 27 |
"targets": []
|
| 28 |
}
|
| 29 |
-
}
|
|
|
|
| 21 |
"hf_model_id": "Nanbeige/Nanbeige4.2-3B",
|
| 22 |
"hf_revision": "5ff54fb7ed86ce8e216d78bff5417ab9981de3d4"
|
| 23 |
},
|
| 24 |
+
"compression": {
|
| 25 |
+
"scheme": "int8hu"
|
| 26 |
+
},
|
| 27 |
"compilation": {
|
| 28 |
"date": "2026-07-22T15:06:28.708628+00:00",
|
| 29 |
"targets": []
|
| 30 |
}
|
| 31 |
+
}
|