---
license: apache-2.0
language:
- en
library_name: pytorch
pipeline_tag: text-generation
tags:
- causal-lm
- base-model
- from-scratch
- muon
- gqa
---

# Zenyx-100M-Base

A **151M-parameter** dense decoder-only language model trained from scratch on web, code and math text (the "100M" in the name is the original plan; the trained model is 151M).

> **Status: pre-training in progress** (11.48B of a 50B-token budget at step 21,900; newer checkpoints may be in the repository). This is a **base model**: not instruction-tuned, no safety tuning, and it makes up facts. All numbers below are from mid-training checkpoints (the learning rate has not finished decaying); the benchmarks run with the segment memory off. From step 19,500 training also includes a *recall task* for the segment memory ([below](#segment-memory-experimental)). A post-training stage is planned ([below](#planned-post-training)).

## At a glance

| | |
|---|---|
| Parameters | 151.0M (152.2M with the experimental segment memory, checkpoints from step 17,900) |
| Architecture | 18-layer Llama-style decoder, grouped-query attention, SwiGLU, context 2,048; since step 17,860 an experimental segment memory in two blocks (see [Segment memory](#segment-memory-experimental)) |
| Vocabulary | 49,105 tokens (byte-level BPE) |
| Training | 2 × Tesla T4, fp16; Muon + AdamW; 524,288 tokens per step (about 11k tokens/s with the segment memory, 21-25k before it) |
| Progress | step 21,900 = 11.48B tokens of 50B; learning rate at 39% of peak, decaying until 15B tokens |
| Latest benchmarks | step 21,900 (core suite); step 20,200 (wider suite, MMLU, chat-memory test); below; step 10,900 (code, speed, letter-choice); step 10,800 (wide arithmetic) |
| License | Apache-2.0 |

## Usage

The weights are in `checkpoints/main-v1/step_N/model.pt` (PyTorch, fp16/fp32); `zenyx_interactive.py` in this repository is a self-contained loader with the model code embedded (PyTorch, `tokenizers`, `huggingface_hub`; a GPU is recommended, a T4 is enough). It loads the latest checkpoint automatically.

```bash
pip install torch tokenizers huggingface_hub
```

```python
from huggingface_hub import hf_hub_download
hf_hub_download("Arko007/Zenyx-100M-Base", "zenyx_interactive.py", local_dir=".")

from zenyx_interactive import Zenyx
z = Zenyx.load()                                    # latest checkpoint (about 600 MB); Zenyx.load(step=21900) for a specific one

print(z.ask("What is 17 * 23?", show=False))         # arithmetic: the model's native chat format, greedy
print(z.ask("Write a Python function that returns the square of a number.", show=False, max_new=150))
print(z.ask("The capital of France is", mode="plain", show=False, max_new=30))        # raw text continuation (base-model style)

# multi-turn chat: keep=True remembers the conversation; longer histories go through the segment memory
z.new_chat()
z.ask("Hi, my name is Priya and I live in Pune.", keep=True, show=False)
print(z.ask("What is my name?", keep=True, show=False))
```

Output of exactly this code on the step-20,500 checkpoint (T4): `391`, then a short Python `square` function in a code block. Defaults: sampling with temperature 0.8, top-k 50 and repetition penalty 1.15 (arithmetic is decoded greedily, code at temperature 0.2); override with `temperature=`, `top_k=`, `repetition_penalty=`, `max_new=`. This is a base model: expect invented facts and chat answers that ignore instructions; it is strongest at arithmetic in its own format and at short code.

## Results

Base model, no fine-tuning, 0-shot, all test items. Fact1000 is a 1,000-item four-way factual test written by the author (chance 25%); SciQ prompts include the supporting passage. Standard error is about 1.4 points on 1,000 items.

### Comparison at the latest checkpoints

| | **Zenyx-100M** (step 21,900) | BetterGPT-150M | Muse2-230M* | Muse2-125M* | Zenyx-V3 (MoE) |
|---|---|---|---|---|---|
| Parameters | 151.0M | 152.6M | 196.1M | 122.9M | 1.58B (0.38B active) |
| Training tokens | 11.5B | about 15B | not published | not published | about 80B |
| Fact1000 | 72.1% | 76.7% | 56.3% | 51.9% | **79.8%** |
| Fact1000, without the arithmetic items | 69.5% | 78.3% | 61.7% | 57.1% | **79.2%** |
| SciQ (acc / length-normalised) | **81.7 / 76.7** | 80.7 / 75.6 | 68.3 / 61.7 | 67.3 / 58.7 | 79.4 / 70.6 |
| ARC-Easy | 48.4 / 44.5 | **54.0 / 48.5** | 44.3 / 39.8 | 44.3 / 39.8 | 52.0 / 46.6 |
| ARC-Challenge | 21.7 / 24.4 | **25.5 / 27.2** | 18.9 / 24.1 | 17.3 / 22.6 | 21.6 / 26.1 |
| Pile bits per byte, 22 domains (lower is better) | 1.066 | 1.066 | 1.193 | 1.227 | **0.999** |
| Pile, mean of domains | 1.220 | 1.232 | 1.387 | 1.424 | **1.159** |
| Arithmetic, 14,000 problems, best prompt format | **97.8% (own chat format; about half of the items are in the training data, see Arithmetic)** | 9.7% (3-shot) | not run | not run | 16.0% (3-shot) |

\* Muse2: our own reimplementation of the architecture (no public code), same harness and items, not token-matched. V3 is step 110,000 (arithmetic: step 110,800). BetterGPT is its released checkpoint with its tokenizer's BOS+EOS prefix (the choice moves its scores by about 1 point; the author-default BOS prefix gives Fact1000 75.7%).

- **BetterGPT-150M leads on Fact1000, ARC-Easy and ARC-Challenge** by roughly 3 to 9 points and, by a hair, on the byte-weighted Pile (1.0662 against 1.0664; the gap is 0.01%), with about 1.5 times the training tokens (15B against 11.5B). **Zenyx-100M is level on SciQ (81.7 / 76.7 against 80.7 / 75.6; one standard error is 1.3 points, and SciQ prompts contain the supporting passage, so it partly measures reading the context) and ahead on the domain-mean Pile** (1.220 against 1.232). Zenyx-100M is ahead of both Muse2 models on Fact1000, SciQ, ARC-Easy and the Pile and level on ARC-Challenge (ARC-Challenge is close to the 25% chance level for every model in the table).
- **Zenyx-100M leads on arithmetic by a wide margin**: BetterGPT and V3 do 2-digit work and little more even with worked examples in the prompt, while Zenyx-100M was trained on arithmetic in its own chat format with a short scratchpad (see [Arithmetic](#arithmetic); part of this lead is memorisation, about half of the 14,000 items are in the training data).
- **Zenyx-V3** (about 10 times the parameters, 7 times the tokens) is ahead on Fact1000, ARC-Easy (accuracy) and the Pile; Zenyx-100M's SciQ is above V3's (79.4 / 70.6), in accuracy and length-normalised (the accuracy gap is about 1.8 standard errors).

### Wider suite (step 20,200)

| | Zenyx-100M | BetterGPT-150M | Zenyx-V3 (step 107,200) |
|---|---|---|---|
| HellaSwag (acc / norm) | 28.7 / 31.0 | 31.5 / 36.6 | 30.7 / 34.2 |
| PIQA | 61.4 / 61.6 | 65.2 / 65.5 | 61.4 / 60.7 |
| WinoGrande | 53.2 | 53.0 | 51.2 |
| OpenBookQA | 17.8 / 25.4 | 20.4 / 31.4 | 19.2 / 28.4 |
| BoolQ | 51.7 / 61.0 | 58.4 / 62.0 | 61.5 / 62.2 |
| COPA (100 items) | 59.0 | 63.0 | 55.0 |
| LAMBADA accuracy | 34.9 | 27.6 | 28.5 |
| WikiText bits per byte | 1.091 | 1.082 | 1.011 |

Most of these are close to chance for models of this size (HellaSwag 25%, PIQA/WinoGrande/COPA 50%). Zenyx-100M is ahead of BetterGPT-150M on LAMBADA; BetterGPT is ahead on HellaSwag, PIQA, OpenBookQA, BoolQ, COPA; level (within half a point) on WinoGrande, WikiText.  An independent lm-evaluation-harness scoring agreed with ours within about 1 point for both small models (checked at an earlier step).

**Further tasks of the author's 19-task notebook suite (Zenyx-100M step 20,200; BetterGPT-150M was not run on these):**

| | Zenyx-100M | Zenyx-V3 (step 107,200) | Chance |
|---|---|---|---|
| MMLU, 5-shot (letter logits, all 14,042 questions) | 25.3 ± 0.7 | 26.9 | 25 |
| MMLU, 0-shot | 25.4 ± 0.7 | not run | 25 |
| RACE (acc / norm) | 28.6 / 31.3 | 31.8 / 34.3 | 25 |
| CommonsenseQA (acc / norm) | 25.4 / 30.4 | 28.6 / 30.4 | 20 |
| LogiQA (acc / norm) | 20.3 / 25.3 | 20.9 / 24.9 | 25 |
| WSC273 | 53.8 | 56.4 | 50 |
| TruthfulQA (MC1 / MC2) | 19.5 / 36.0 | 18.7 / 35.4 | about 20 / 40 |

**MMLU is at chance with 0 and with 5 shots** (25.4 and 25.3 against 25.0): the model has no MMLU knowledge yet, and the worked examples in the 5-shot prompt do not help it (they only teach the answer format). 206 of the 14,042 5-shot prompts (1.5%) are longer than the 2,048-token window and lose their first examples; a 25-shot run was not made. The remaining tasks sit at or just above chance for this size, as the V3 column (a 1.58B MoE) mostly does too. The arithmetic stage of the notebook (plain "a + b =" prompts, no chat format) scores about 0%: the model computes in its chat format only (see [Arithmetic](#arithmetic)).

### Pile bits per byte by domain

Lower is better; bold = best of the four. Zenyx-100M step 21,900; Muse2-230M* and V3 as above. Bits per byte does not depend on the tokenizer.

| Domain | Zenyx-100M | BetterGPT-150M | Muse2-230M* | Zenyx-V3 |
|---|---|---|---|---|
| DM Mathematics | **1.299** | 1.348 | 1.723 | 1.326 |
| EuroParl | 1.994 | 2.155 | 2.444 | **1.964** |
| YouTube Subtitles | 1.850 | 2.011 | 2.290 | **1.816** |
| GitHub | 0.614 | 0.722 | 0.846 | **0.576** |
| Enron Emails | 1.262 | 1.249 | 1.416 | **1.222** |
| Wikipedia (en) | 1.038 | 1.080 | 1.123 | **0.987** |
| PhilPapers | 1.384 | 1.415 | 1.667 | **1.332** |
| HackerNews | 1.341 | 1.305 | 1.448 | **1.278** |
| Gutenberg (PG-19) | 1.334 | 1.298 | 1.430 | **1.268** |
| StackExchange | 0.985 | 1.025 | 1.207 | **0.917** |
| BookCorpus2 | 1.210 | 1.155 | 1.205 | **1.142** |
| Pile-CC | 1.166 | 1.124 | 1.199 | **1.098** |
| Books3 | 1.400 | 1.398 | 1.495 | **1.330** |
| OpenSubtitles | 1.404 | 1.358 | 1.399 | **1.333** |
| ArXiv | 0.926 | 0.911 | 1.131 | **0.855** |
| OpenWebText2 | 1.198 | 1.188 | 1.281 | **1.126** |
| PubMed Central | 0.814 | 0.797 | 0.922 | **0.741** |
| Ubuntu IRC | 1.788 | 1.852 | 2.121 | **1.714** |
| USPTO Backgrounds | 0.859 | 0.828 | 0.938 | **0.779** |
| NIH ExPorter | 0.947 | 0.906 | 1.018 | **0.861** |
| PubMed Abstracts | 0.968 | 0.932 | 1.072 | **0.873** |
| FreeLaw | 1.064 | 1.040 | 1.129 | **0.958** |
| **22 domains, byte-weighted** | 1.066 | 1.066 | 1.193 | **0.999** |
| **Mean of domains** | 1.220 | 1.232 | 1.387 | **1.159** |

Zenyx-100M beats BetterGPT in 8 of 22 domains (GitHub, YouTube Subtitles, EuroParl, StackExchange, Wikipedia (en), DM Mathematics, Ubuntu IRC, PhilPapers) and is 0 to 5% behind in the other 14 (overall 0.01% behind). It is ahead of Muse2-230M* in 20 of 22 domains.

### Over training

| Step | Tokens (B) | Fact1000 (no arithmetic) | SciQ acc / norm | ARC-Easy acc / norm | Pile bits/byte | Arithmetic (14,000) |
|---|---|---|---|---|---|---|
| 1,810 | 0.95 | 50.6% (53.6%) | not run | not run | not run | not run |
| 6,000 | 3.15 | 58.5% (61.7%) | 72.4 / 67.3 | 44.4 / 40.1 | 1.145 | 93.7% |
| 7,400 | 3.88 | 60.9% (63.8%) | 71.0 / 63.4 | 44.0 / 40.2 | not run | 72.1% |
| 8,910 | 4.67 | 60.1% (62.5%) | 72.3 / 65.8 | 42.3 / 39.6 | 1.120 | 95.6% |
| 9,500 | 4.98 | 61.9% (65.3%) | 74.2 / 69.2 | 44.5 / 41.2 | 1.120 | 95.2% |
| 9,700 | 5.09 | 61.7% (64.2%) | 72.7 / 67.9 | 45.6 / 41.6 | 1.119 | not run |
| 10,800 | 5.66 | 68.1% (66.0%) | 75.2 / 68.5 | 45.4 / 40.7 | 1.111 | 89.4% |
| 12,650 | 6.63 | 70.5% (68.0%) | 76.1 / 70.8 | 46.0 / 42.3 | 1.103 | 91.1% |
| 14,500 | 7.60 | 68.9% (66.9%) | 75.7 / 71.6 | 47.3 / 43.9 | 1.095 | 85.5% |
| 14,700 | 7.71 | 70.1% (67.2%) | 76.6 / 72.5 | 46.9 / 42.2 | 1.090 | 96.5% |
| 16,100 | 8.44 | 64.6% (67.3%) | 73.6 / 66.9 | 44.1 / 40.4 | 1.084 | 98.0% |
| 17,700 | 9.28 | 67.1% (69.2%) | 72.5 / 66.3 | 46.1 / 41.7 | 1.078 | 97.0% |
| 17,860 | 9.36 | 73.0% (70.0%) | 78.9 / 71.9 | 47.2 / 42.6 | 1.079 | 97.3% |
| 19,300 | 10.12 | 71.8% (69.2%) | 79.8 / 74.1 | 46.8 / 43.7 | 1.076 | 93.3% |
| 19,500 | 10.22 | 72.1% (69.5%) | 77.6 / 72.3 | 47.3 / 43.1 | 1.074 | 96.5% |
| 19,600 | 10.28 | 73.1% (70.1%) | 82.3 / 80.1 | 49.9 / 47.0 | 1.076 | 93.7% |
| 20,200 | 10.59 | 72.7% (69.9%) | 80.8 / 77.0 | 48.8 / 45.6 | 1.072 | 97.9% |
| 20,300 | 10.64 | 73.5% (70.7%) | 79.5 / 74.8 | 47.3 / 44.0 | 1.071 | 97.2% |
| 21,110 | 11.07 | 73.2% (71.1%) | 83.1 / 77.5 | 49.2 / 46.2 | 1.068 | 98.3% |
| 21,200 | 11.12 | 72.8% (69.5%) | 80.1 / 74.9 | 48.2 / 44.2 | 1.069 | 97.4% |
| **21,900** | **11.48** | **72.1% (69.5%)** | **81.7 / 76.7** | **48.4 / 44.5** | **1.066** | **97.8%** |

Compared with step 6,000, Fact1000 without the arithmetic items rose from 61.7% to 69.5%, SciQ from 72.4 to 81.7 and ARC-Easy from 44.4 to 48.4 (accuracy), the Pile from 1.145 to 1.066; ARC-Challenge stays at about chance. Since step 14,700 (one standard error is about 1.4 points on 1,000 items, 1.0 on ARC-Easy): Fact1000 without the arithmetic items +2.3, SciQ +5.1, ARC-Easy +1.5, ARC-Challenge +0.3, the Pile -0.0238 bits per byte. The curve is not monotone: SciQ and ARC-Easy dipped at step 16,100 and recovered by 21,900, and Fact1000 without the arithmetic items was 67.2, 67.3, 69.2 and 69.5 at 14,700, 16,100, 17,700 and 21,900, so differences of a point or two between checkpoints are within noise. The Pile figure was 1.0902, 1.0842, 1.0777 and 1.0664 bits per byte at the same four steps. The all-items Fact1000 score changed by +2.0 points since step 14,700 and by +4.0 since step 10,800 (the standard error of such a difference is about 2). **Arithmetic oscillates between checkpoints** (72% to 98% over the run). 2-digit multiplication: 92% at step 9,500, 58% at 10,800, 95% at 12,650, 23% at 14,500 (the first scratchpad line was right 95% of the time, but 76% of the replies then added a spurious second step), 87% at 14,700 and 95% at 21,900; its cause is not established.

### Arithmetic

The training data includes arithmetic reasoning examples. The model answers arithmetic in a fixed chat-style format with a one-line scratchpad (no step-by-step working; the answer is computed in one forward pass):

```
<|im_start|>user
What is 683 + 479?<|im_end|>
<|im_start|>assistant
<think>
683 + 479 = 1162
</think>

1162<|im_end|>
```

14,000 problems (7 types × 2,000, fixed seeds, exact match), step 21,900: 2-digit addition 100.0%, 2-digit subtraction 99.7%, 3-digit addition 99.7%, 3-digit subtraction 92.4%, 4-digit addition 99.4%, 5-digit addition 98.8%, 2-digit × 2-digit 94.7%; **all 97.8%** (13,691 of 14,000). Without the chat format the same model scores under 1%: this is a trained, narrow skill.

**The 14,000-problem suite overlaps the training data.** We streamed 4 million rows of the arithmetic dataset in the training mix (`SagheerLab/Arithmetic-Reasoning`, same "What is x + y?" format) and found 6,691 of the 14,000 items word for word (2-digit addition, subtraction and multiplication: essentially all; 3-digit addition 35%; 3-digit subtraction 0.2%; 4- and 5-digit addition 0%; the scan stopped at its row cap, so these are lower bounds). The headline therefore mixes memorised items with skill. The three types that are not in the training data (3-digit subtraction, 4-digit and 5-digit addition, 6,000 items) score **96.9%** at step 21,900; that is the number to compare across checkpoints.

**Our own suite** (`tests/arith_own.py`, 8,048 fresh items with seeds the old suite never used, every answer double-checked; step 19,300, native chat format): 3-digit subtraction 92.8%, 4-digit addition 99.5%, 4-digit subtraction 89.0%, 5-digit addition 98.2%, **5-digit subtraction 49.5%** (borrowing). **Sums of 3 to 10 numbers: 0.1%** (3 of 2,700), mixed add/subtract chains 2.9%, repeated numbers such as 2+2+2+2+2 0%, products of three or four small factors 3.8%. The model guesses a long sum in one step (`9 + 1 + 3 + 3 + 8 = 12`, the answer is 24) instead of adding term by term. Training data that adds one term at a time (3-10 terms, verified by evaluation) has been written for the planned fine-tuning; the pre-training mix has none.


**Wide suite** (7,636 further problems in 28 types, exact match, step 10,800): **7.3%** overall (13.0% at step 9,500; not rerun at the latest step). Best types: negative numbers, mul / div 31%; powers 30%; decimals, add / sub 30%; negative numbers, add / sub 23%. Near zero: 6-8 digit numbers, fractions, GCD / LCM, series sums, averages, unit conversion, linear equations and word problems. Between steps 9,500 and 10,800, 3-digit × 1-digit multiplication fell from 94.1% to 22.7% and 3-digit × 2-digit from 61.2% to 0.0%: replies now often follow a correct first scratchpad line with a spurious second step (for example `691 * 8 = 5528 | 691 + 5528 = 6219`). This is the same chaining that caused the 2-digit multiplication dip above, and the failure seen at step 7,400 coming back.

<details><summary>Wide suite by type</summary>

| Type | Items | Exact match | Reply finished |
|---|---|---|---|
| 3-digit × 1-digit | 286 | 22.7% | 100% |
| 3-digit × 2-digit | 286 | 0.0% | 100% |
| 3-digit × 3-digit | 286 | 10.5% | 100% |
| 6-8 digit addition | 286 | 0.7% | 100% |
| 6-8 digit subtraction | 286 | 0.0% | 100% |
| exact division | 286 | 16.4% | 100% |
| division rounded down | 286 | 2.1% | 100% |
| remainder / mod | 286 | 0.0% | 72% |
| negative numbers, add / sub | 286 | 23.1% | 100% |
| negative numbers, mul / div | 286 | 30.8% | 99% |
| decimals, add / sub | 286 | 29.7% | 100% |
| decimals, multiply | 286 | 2.1% | 100% |
| order of operations | 286 | 18.9% | 100% |
| powers | 186 | 30.1% | 90% |
| square / cube roots | 199 | 0.0% | 51% |
| percentages | 286 | 15.4% | 100% |
| fractions | 286 | 0.0% | 86% |
| greatest common divisor | 286 | 0.0% | 42% |
| least common multiple | 286 | 0.0% | 77% |
| factorial / choose / permutations | 101 | 1.0% | 89% |
| sums of series | 286 | 0.0% | 63% |
| rounding | 286 | 0.0% | 76% |
| linear equations in x | 286 | 1.0% | 100% |
| word problems, one step | 286 | 1.0% | 88% |
| word problems, two steps | 286 | 0.0% | 78% |
| average | 286 | 0.0% | 80% |
| median | 286 | 0.0% | 83% |
| unit conversion | 286 | 0.0% | 67% |

</details>

## Model details

| Component | Setting |
|---|---|
| Width / depth | d_model 768, 18 blocks |
| Attention | 12 query heads × 64, **4 key/value heads** (grouped-query), causal, per-head QK-norm |
| Positions | RoPE, θ = 500,000 |
| MLP | SwiGLU, hidden size 2,048 |
| Normalisation | RMSNorm, pre-norm; fp32 residual stream with fp16 sub-layers (the T4 has no bf16) |
| Embeddings | tied input/output, 49,152 rows |
| Bias / dropout | none |

Sixteen of the 18 layers are plain grouped-query attention (blocks 8 and 12 additionally carry the segment memory below): sliding-window, gated linear and DeltaNet-style alternatives were benchmarked on a T4 at 2k context and none beat flash-attention GQA on speed. KV cache: 18,432 bytes per token in fp16.


### Segment memory (experimental)

From step 17,860 two blocks (8 and 12, counting from 0) can also read the keys and values of the **earlier 2,048-token windows of the same text**, so a window may use up to 32,768 tokens of context although every forward pass stays 2,048 tokens. For each query head the layer looks up the 32 best-matching stored keys (exact top-32; the stored keys carry no position, so RoPE is untouched), and adds the weighted values to its normal attention output through a per-head `tanh` gate that **starts at exactly zero**: at the moment it was switched on the model computed the same function as the step-17,860 checkpoint. The memory has its own query projection (initialised as a copy of the layer's) and its own optimiser for the gates; no gradient flows into the stored keys and values. Design after Memorizing Transformers (Wu et al., 2022) and the lessons of Hugging Face's failed Infini-attention re-implementation.

| | |
|---|---|
| Memory blocks | 8 and 12; the cached decoder reads the memory (`zenyx100m.memattn.generate_long`, and `Zenyx.ask` in the notebook feeds long prompts window by window); the CUDA-graph fast decoder ignores it |
| New parameters | 1,179,800 (152.2M in total) |
| Layout | per GPU and step, 8 runs of 16 consecutive windows; memory reset at every optimiser step; 524,288 tokens per step as before |
| Cost | 2.2× slower training (49 s against 22 s per step on 2 × T4), 11.7 GB against 11.1 GB |
| Recall task | from step 19,500: 25% of the runs (falling to 12% over 1,500 steps) plant 3-20 invented facts in earlier windows and ask questions about them in a later window; loss on the answer tokens only |

**Status: the memory copies text and ranks facts well, free recall of a fact from far back was missing until step 19,500 and has just started to work.** Measured on step 18,670 and step 19,300 (one T4, `tests/colab/mem_eval_v3.py`; the memory is on or off with the same weights; the step-18,000 checkpoint, whose gates were still small, is the control):

| Passcode hidden in unrelated text, 8 candidate numbers (chance 12.5%, 24 trials per length) | 2k | 4k | 8k | 16k | 32k |
|---|---|---|---|---|---|
| Step 18,670, memory on | 83% | 96% | 88% | 92% | 92% |
| Step 18,670, memory off | 83% | 12% | 0% | 12% | 21% |
| Step 18,000, memory on | 79% | 67% | 29% | 21% | 29% |
| Step 19,300, memory on | 88% | 96% | 83% | 83% | 71% |
| Step 19,300, memory off | 88% | 8% | 8% | 8% | 12% |

At step 19,300, 30 long Wikipedia articles (first 16,384 tokens each): the memory lowers the loss of windows 1-7 by **0.037 nats** (95% interval 0.028-0.047, better in every article; 0.035 at step 18,670, 0.022 at 18,000), growing with distance; a different article's memory makes the loss slightly worse (+0.012 to +0.014), so the gain is retrieval and not extra context. In 20 long chats of 8 windows the loss is 0.028 nats lower. **Window 0 is identical with and without the memory** (3.4028 against 3.4028), so a short prompt behaves exactly like the memory-free model, which is why the benchmarks on this card (run with the memory off) still apply. Reading the memory in the cached decoder is exact (logits agree with the full forward, top-1 agreement 100%, greedy decode identical on 16 of 16 tokens).

**Repeated text is copied from the memory at any position**: teacher-forced loss of a repeated 2,048-token span 3.78 without the memory against 0.21 with it (top-1 accuracy 32% to 96%), 0.24 when the span starts 64 tokens later, 0.42 when it starts 1,024 tokens later and 0.33 after 200 unrelated tokens (step 19,600). **Questions about a fact far back were not answered before the recall task** (step 19,300-19,400, greedy): the hidden passcode was written correctly 5% of the time at 4k tokens and 0% at 8k-64k (0% without the memory); a passage repeated after filler was continued for 12% / 8% / 4% of the next 40 tokens at 4k / 16k / 32k (1-3% without); after 3k-64k tokens of filler a chat fact (name, city, colour) was ranked first among 8 candidates 85% / 55% / 50% / 45% / 25% of the time (3k / 8k / 16k / 32k / 64k; 5-20% without the memory) but never produced as free text; and a three-fact memo 8.5k tokens back, asked as a question, scored 0 of 3. The reason is the training task: until step 19,500 nothing in training asked a question whose answer lay in an earlier window and did not simply continue the text.

**Recall task (from step 19,500, `zenyx100m/recall.py`).** Invented subjects and values (project leaders, budgets, code words, passcodes, dates, cities, counts, colours, times) are written as one sentence each into earlier windows of the same run (1-14 windows back, random positions and wrappers); the end of a later window becomes a block of questions about them in four layouts (Question/Answer, Q:/A:, the chat format, an unfinished sentence), and only the answer tokens carry loss. First result, the step-19,600 checkpoint after about 100 steps of it (one checkpoint, small samples, to be repeated): the memo test now answers the project lead and the budget correctly in chat format (2 of 3; the code word is wrong) against 0 of 3 with the memory off; on 24 fresh trials worded differently from the training task, the right value is produced in 8 of 24 chat answers (33%) and 5 of 24 plain Q/A answers (21%) with the memory and in 0 of 24 and 0 of 24 without it. Still failing: code words and exact dates, "repeat the first sentence of the memo", summaries of a long document (they continue the article instead).

**Practical chat test (step 20,200, `tests/colab/mem_chat_own.py`, free greedy answers through the notebook's `ask(keep=True)`).** A real multi-turn chat: turn 1 plants a fact, then the user keeps pasting about 600-token texts ("summarize this") and the model answers them until the history is about 2.5k / 5k / 9k tokens long; then the question is asked with the memory on and off (off = only the last 2,048 tokens are visible). Each value is scored by exact match; 10 chats per condition.

| Fact | Question right after the fact (no memory needed) | ~2.5k tokens later, memory on / off | ~5k | ~9k |
|---|---|---|---|---|
| User says name, city and dog's name | 60% | **33% / 0%** | 13% / 0% | 10% / 0% |
| Assistant turn states restaurant and confirmation code | 15% | **65% / 0%** | 30% / 0% | 20% / 0% |
| The model's own free-text answer ("tell me a fact about octopuses", later "what fact did you tell me?") | 13 of 16 chats | 0 of 6 / 0 of 6 | 0 of 10 / 0 of 10 | 0 of 7 / 0 of 7 |

With the memory off the model recalls nothing once the fact has left the window; with it on it still finds specific values (names, codes) up to 9k tokens back, but accuracy falls with distance and is far below what the model does when the fact is inside the window. The last row counts only chats in which the original answer was no longer visible to the memory-off model (the base model tends to copy its earlier answers into later ones, which keeps a fact alive inside the window; 6 to 10 of 16 chats remain at each distance): there the memory did **not** help, so remembering what the model itself said in free text is not working yet. Why: recall training so far plants short facts as statements by the user; it never asks about something the assistant said. The model also often answers in the first person ("My name is ...") instead of "your name is ...", a chat-format weakness that post-training should fix.

**Not shown:** lengths beyond 32,768 tokens in the ranking test, any benchmark effect (all benchmarks on this card run with the memory off), messy real-world retrieval (the natural-text gain is about 1% of the loss). The memory costs 2.2x training time. Core benchmarks at the latest step (memory off) are in the tables above; the step-19,600 checkpoint, the first after the recall task started, scored SciQ 82.3 / 80.1, ARC-Easy 49.9 / 47.0 and Fact1000 73.1% (up from 77.6 / 72.3, 47.3 / 43.1 and 72.1% at 19,500): steps 20,200 and 20,300 then scored SciQ 80.8 / 77.0 and 79.5 / 74.8, ARC-Easy 48.8 / 45.6 and 47.3 / 44.0, Fact1000 72.7% and 73.5%: back inside the range of the earlier checkpoints (SciQ 77.6-82.3, ARC-Easy 46.8-49.9; one standard error is 1.3 points on SciQ and 1.0 on ARC-Easy), so the 19,600 values were mostly noise and **we cannot attribute a benchmark gain to the recall task**. Checkpoints from step 17,900 contain 6 extra tensors (`mem_gate`, `mem_q_proj.weight`, `mem_q_norm.weight` in blocks 8 and 12): load them with `strict=False` or through `zenyx100m.memattn`.

## Tokenizer

Byte-level BPE with a SuperBPE-style second stage (merges may cross spaces; digits stay single tokens; 248 special tokens). On 63 MB of text from the training sources it needs about 9% fewer tokens than the DeepSeek-V4-Pro tokenizer (4.56 against 4.14 bytes per token) with a vocabulary 2.6 times smaller; it ties on Wikipedia and is worse on markdown and GSM8K. Only English, code and math were measured. File: `tokenizer_candidates_v3/F-49152-f70/tokenizer.json`.

## Training

| | |
|---|---|
| Optimiser | Muon on hidden matrices (lr 0.012), AdamW on embeddings and norms (lr 2e-3) |
| Schedule | warmup, constant until 3.9B tokens, then linear decay to 10% of peak between 3.9B and 15B tokens, held to the 50B-token budget |
| Batch | 524,288 tokens per step; about 21-25k tokens/s on 2 × T4 |
| Data | English web, code, math and chat text; since the October 24 relaunch (step 18,670) also instruction-style data (OpenOrca GPT-4 part, UltraChat, MetaMathQA, oasst, text-to-SQL) as ordinary pre-training text |

## Other measurements

- **MMLU** (step 20,200), all 14,042 questions: **25.3% ± 0.7 with 5 shots, 25.4% ± 0.7 with 0 shots** (chance 25%; Zenyx-V3 26.9%). The step-12,650 value was 25.0%.
- **Python** (step 10,900), twelve small functions run against unit tests (primes, factorial, palindrome, gcd, Fibonacci, FizzBuzz and similar): signature + docstring as plain completion **8 of 12** with greedy decoding and 43% of sampled attempts pass (8 samples per task, T = 0.6); chat format 5 of 12 and 34%; forcing `<think>` first lowers it to 2 of 12 and 7%. (Step 6,000: 4 of 12 and 25%.) It still fails harder tasks.
- **Letter-choice multiple choice** (step 10,900) in chat format is at chance (SciQ 19%, ARC-Easy 23%, ARC-Challenge 25%, 200 questions each) although the standard scoring of the same questions gives 75%, 45% and 24%: the model has the knowledge but not the "read four lettered options and pick one" skill.
- **Decode speed** (step 10,900), one T4, fp16, CUDA graph + `torch.compile`: 349 tokens/s at batch 1, 2,572 tokens/s total at batch 8, 9,475 tokens/s at batch 64.
- **Long generation:** greedy decoding repeats itself within about 30 words; **suggested: sampling (temperature 0.8, top-k 50) with a repetition penalty of 1.1-1.2.**

## Planned post-training

A supervised fine-tuning stage is being prepared; **no instruct model has been released and no result exists yet.** The training set is built (19 shards = 149,571 windows of 2,049 tokens, 300M window tokens of which 208M are assistant tokens that carry loss; one planned source, our own earlier synthetic set at 4%, could not be read and is missing) from public chat, instruction and reasoning data plus synthetic data with exact answers (single operations and, new, sums of 3-10 numbers, tool-calculator calls, system-prompt-versus-user conflicts, identity), English only (a language filter drops other scripts and most other Latin-script languages). ChatML with the tokenizer's real special tokens (`<|im_start|>`, `<|im_end|>`, `<think>`, `</think>`, `<tool_call>`, `<tool_response>`), loss on assistant tokens only; system prompts that must win over conflicting user instructions; a `Reasoning effort: none | low | medium | high` line with reasoning budgets of 256 / 512 / 1,024 tokens. Weak points that motivate it, measured with a code-scored chat suite at step 18,900 (10 categories, 8-45 prompts each, macro average 54.6%): greetings 33% (it often answers "Hello" with an apology and "as a text-based AI"), instruction following with checkable constraints 20%, system-prompt obedience 22%, small code tasks 25%, free-text facts 27%, against arithmetic 96% and recall of a detail given earlier in the same chat 88%. The numbers in this card are the pre-SFT baseline.

## Intended use and limitations

- **Use:** research, and as a starting point for fine-tuning. Not an assistant: it ignores system prompts, answers greetings badly, and may continue a prompt with an unrelated document.
- **Facts:** expect errors and invented details; long generations repeat themselves.
- **Arithmetic** works in its own chat format only (see above), for two numbers of up to five digits (five-digit subtraction 50%), and about half of the old test suite is in the training data. It does **not** add a list of numbers (0.1% on 3-10 terms) and not negatives, decimals, fractions, powers, algebra or word problems.
- **Reasoning, MMLU, letter-choice multiple choice:** at chance. **Code (step 10,900):** passes 8 of 12 small functions, not harder ones. The `<think>` habit exists only for arithmetic.
- **Context** 2,048 tokens (up to 32,768 with the experimental segment memory, whose free recall of far-back facts is just starting to work); **English only**; not evaluated for bias or harmful content.
- **Evaluation:** one training run, mid-training checkpoints; Fact1000 and the question sets are written by the author; Muse2 numbers use an unofficial reimplementation.

## Repository contents

| Path | Contents |
|---|---|
| `checkpoints/<run>/step_N/` | model weights (`model.pt`), optimiser state, training state; only the two newest checkpoints are kept |
| `tokenizer_candidates_v3/` | tokenizers, including the one used |
| `zenyx_interactive.py` | self-contained loader and chat helper (see Usage) |
| `wheels/`, `runs/`, `data/` | flash-attention build, training manifests |

The model code is not published yet.