Zenyx-100M-Base
A 151M-parameter dense decoder-only language model trained from scratch on web, code and math text (the "100M" in the name is the original plan; the trained model is 151M).
Status: pre-training in progress (5.66B of a 35B-token budget at step 10,800; newer checkpoints may be in the repository). This is a base model: not instruction-tuned, no safety tuning, and it makes up facts. All numbers below are from mid-training checkpoints (the learning rate has not finished decaying). A post-training stage is planned (below).
At a glance
| Parameters | 151.0M |
| Architecture | 18-layer Llama-style decoder, grouped-query attention, SwiGLU, context 2,048 |
| Vocabulary | 49,105 tokens (byte-level BPE) |
| Training | 2 × Tesla T4, fp16; Muon + AdamW; 524,288 tokens per step |
| Progress | step 10,800 = 5.66B tokens of 35B; learning rate at 86% of peak, decaying until 15B tokens |
| Latest benchmarks | step 10,800 (core suite, below); step 10,900 (MMLU, code, speed); step 9,700 (wider suite) |
| License | Apache-2.0 |
Results
Base model, no fine-tuning, 0-shot, all test items. Fact1000 is a 1,000-item four-way factual test written by the author (chance 25%); SciQ prompts include the supporting passage. Standard error is about 1.4 points on 1,000 items.
Comparison at the latest checkpoints
| Zenyx-100M (step 10,800) | BetterGPT-150M | Muse2-230M* | Muse2-125M* | Zenyx-V3 (MoE) | |
|---|---|---|---|---|---|
| Parameters | 151.0M | 152.6M | 196.1M | 122.9M | 1.58B (0.38B active) |
| Training tokens | 5.7B | about 15B | not published | not published | about 80B |
| Fact1000 | 68.1% | 76.7% | 56.3% | 51.9% | 79.8% |
| Fact1000, without the arithmetic items | 66.0% | 78.3% | 61.7% | 57.1% | 79.2% |
| SciQ (acc / length-normalised) | 75.2 / 68.5 | 80.7 / 75.6 | 68.3 / 61.7 | 67.3 / 58.7 | 79.4 / 70.6 |
| ARC-Easy | 45.4 / 40.7 | 54.0 / 48.5 | 44.3 / 39.8 | 44.3 / 39.8 | 52.0 / 46.6 |
| ARC-Challenge | 22.4 / 23.9 | 25.5 / 27.2 | 18.9 / 24.1 | 17.3 / 22.6 | 21.6 / 26.1 |
| Pile bits per byte, 22 domains (lower is better) | 1.111 | 1.066 | 1.193 | 1.227 | 0.999 |
| Pile, mean of domains | 1.269 | 1.232 | 1.387 | 1.424 | 1.159 |
| Arithmetic, 14,000 problems, best prompt format | 89.4% (own chat format) | 9.7% (3-shot) | not run | not run | 16.0% (3-shot) |
* Muse2: our own reimplementation of the architecture (no public code), same harness and items, not token-matched. V3 is step 110,000 (arithmetic: step 110,800). BetterGPT is its released checkpoint with its tokenizer's BOS+EOS prefix (the choice moves its scores by about 1 point; the author-default BOS prefix gives Fact1000 75.7%).
- BetterGPT-150M leads on knowledge and language modelling by roughly 5 to 15 points, with about three times the training tokens; Zenyx-100M is ahead of both Muse2 models on Fact1000, SciQ and the Pile and level on ARC.
- Zenyx-100M leads on arithmetic by a wide margin: BetterGPT and V3 do 2-digit work and little more even with worked examples in the prompt, while Zenyx-100M was trained on arithmetic in its own chat format with a short scratchpad (see Arithmetic).
- Zenyx-V3 (about 10 times the parameters, 14 times the tokens) is ahead of everything else on knowledge and the Pile.
Wider suite (step 9,700)
| Zenyx-100M | BetterGPT-150M | Zenyx-V3 (step 107,200) | |
|---|---|---|---|
| HellaSwag (acc / norm) | 28.0 / 29.8 | 31.5 / 36.6 | 30.7 / 34.2 |
| PIQA | 59.6 / 57.7 | 65.2 / 65.5 | 61.4 / 60.7 |
| WinoGrande | 52.5 | 53.0 | 51.2 |
| OpenBookQA | 13.8 / 26.4 | 20.4 / 31.4 | 19.2 / 28.4 |
| BoolQ | 61.9 / 62.2 | 58.4 / 62.0 | 61.5 / 62.2 |
| COPA (100 items) | 53.0 | 63.0 | 55.0 |
| LAMBADA accuracy | 34.1 | 27.6 | 28.5 |
| WikiText bits per byte | 1.142 | 1.082 | 1.011 |
Most of these are close to chance for models of this size (HellaSwag 25%, PIQA/WinoGrande/COPA 50%); BetterGPT is ahead on all but LAMBADA and BoolQ. An independent lm-evaluation-harness scoring agrees with ours within about 1 point for both small models.
Pile bits per byte by domain
Lower is better; bold = best of the four. Zenyx-100M step 10,800; Muse2-230M* and V3 as above. Bits per byte does not depend on the tokenizer.
| Domain | Zenyx-100M | BetterGPT-150M | Muse2-230M* | Zenyx-V3 |
|---|---|---|---|---|
| Enron Emails | 1.297 | 1.249 | 1.416 | 1.222 |
| Wikipedia (en) | 1.067 | 1.080 | 1.123 | 0.987 |
| PhilPapers | 1.425 | 1.415 | 1.667 | 1.332 |
| Gutenberg (PG-19) | 1.368 | 1.298 | 1.430 | 1.268 |
| HackerNews | 1.378 | 1.305 | 1.448 | 1.278 |
| PubMed Central | 0.842 | 0.797 | 0.922 | 0.741 |
| EuroParl | 2.065 | 2.155 | 2.444 | 1.964 |
| USPTO Backgrounds | 0.881 | 0.828 | 0.938 | 0.779 |
| GitHub | 0.678 | 0.722 | 0.846 | 0.576 |
| Pile-CC | 1.201 | 1.124 | 1.199 | 1.098 |
| BookCorpus2 | 1.248 | 1.155 | 1.205 | 1.142 |
| StackExchange | 1.029 | 1.025 | 1.207 | 0.917 |
| OpenWebText2 | 1.240 | 1.188 | 1.281 | 1.126 |
| OpenSubtitles | 1.447 | 1.358 | 1.399 | 1.333 |
| DM Mathematics | 1.441 | 1.348 | 1.723 | 1.326 |
| ArXiv | 0.974 | 0.911 | 1.131 | 0.855 |
| YouTube Subtitles | 1.936 | 2.011 | 2.290 | 1.816 |
| NIH ExPorter | 0.982 | 0.906 | 1.018 | 0.861 |
| PubMed Abstracts | 1.001 | 0.932 | 1.072 | 0.873 |
| Books3 | 1.460 | 1.398 | 1.495 | 1.330 |
| Ubuntu IRC | 1.849 | 1.852 | 2.121 | 1.714 |
| FreeLaw | 1.099 | 1.040 | 1.129 | 0.958 |
| 22 domains, byte-weighted | 1.111 | 1.066 | 1.193 | 0.999 |
| Mean of domains | 1.269 | 1.232 | 1.387 | 1.159 |
Zenyx-100M beats BetterGPT in 5 of 22 domains (GitHub, EuroParl, YouTube Subtitles, Wikipedia (en), Ubuntu IRC) and is about 4 to 9% behind on long-form web, books and science text. It is ahead of Muse2-230M* in 19 of 22 domains.
Over training
| Step | Tokens (B) | Fact1000 (no arithmetic) | SciQ acc / norm | ARC-Easy acc / norm | Pile bits/byte | Arithmetic (14,000) |
|---|---|---|---|---|---|---|
| 1,810 | 0.95 | 50.6% (53.6%) | not run | not run | not run | not run |
| 6,000 | 3.15 | 58.5% (61.7%) | 72.4 / 67.3 | 44.4 / 40.1 | 1.145 | 93.7% |
| 7,400 | 3.88 | 60.9% (63.8%) | 71.0 / 63.4 | 44.0 / 40.2 | not run | 72.1% |
| 8,910 | 4.67 | 60.1% (62.5%) | 72.3 / 65.8 | 42.3 / 39.6 | 1.120 | 95.6% |
| 9,500 | 4.98 | 61.9% (65.3%) | 74.2 / 69.2 | 44.5 / 41.2 | 1.120 | 95.2% |
| 9,700 | 5.09 | 61.7% (64.2%) | 72.7 / 67.9 | 45.6 / 41.6 | 1.119 | not run |
| 10,800 | 5.66 | 68.1% (66.0%) | 75.2 / 68.5 | 45.4 / 40.7 | 1.111 | 89.4% |
Knowledge scores moved little since step 6,000 (within about two standard errors); the Pile score was flat from step 8,910 to 9,700 and fell at step 10,800. The Fact1000 gain at step 10,800 comes mostly from its 174 arithmetic items. Arithmetic oscillates between checkpoints (72% to 96%; 2-digit multiplication fell from 92% to 57.5% at step 10,800); its cause is not established.
Arithmetic
About 7-8% of the training mixture is arithmetic reasoning data. The model answers arithmetic in a fixed chat-style format with a one-line scratchpad (no step-by-step working; the answer is computed in one forward pass):
<|im_start|>user
What is 683 + 479?<|im_end|>
<|im_start|>assistant
<think>
683 + 479 = 1162
</think>
1162<|im_end|>
14,000 problems (7 types × 2,000, fixed seeds, exact match), step 10,800: 2-digit addition 100.0%, 2-digit subtraction 95.3%, 3-digit addition 99.7%, 3-digit subtraction 87.5%, 4-digit addition 98.9%, 5-digit addition 87.3%, 2-digit × 2-digit 57.5%; all 89.4% (12,521 of 14,000). Without the chat format the same model scores under 1%: this is a trained, narrow skill.
Wide suite (7,636 further problems in 28 types, exact match, step 10,800): 7.3% overall (13.0% at step 9,500). Best types: negative numbers, mul / div 31%; powers 30%; decimals, add / sub 30%; negative numbers, add / sub 23%. Near zero: 6-8 digit numbers, fractions, GCD / LCM, series sums, averages, unit conversion, linear equations and word problems. Between steps 9,500 and 10,800, 3-digit × 1-digit multiplication fell from 94.1% to 22.7% and 3-digit × 2-digit from 61.2% to 0.0%: replies now often follow a correct first scratchpad line with a spurious second step (for example 691 * 8 = 5528 | 691 + 5528 = 6219). This is the same chaining that caused the 2-digit multiplication dip above, and the failure seen at step 7,400 coming back.
Wide suite by type
| Type | Items | Exact match | Reply finished |
|---|---|---|---|
| 3-digit × 1-digit | 286 | 22.7% | 100% |
| 3-digit × 2-digit | 286 | 0.0% | 100% |
| 3-digit × 3-digit | 286 | 10.5% | 100% |
| 6-8 digit addition | 286 | 0.7% | 100% |
| 6-8 digit subtraction | 286 | 0.0% | 100% |
| exact division | 286 | 16.4% | 100% |
| division rounded down | 286 | 2.1% | 100% |
| remainder / mod | 286 | 0.0% | 72% |
| negative numbers, add / sub | 286 | 23.1% | 100% |
| negative numbers, mul / div | 286 | 30.8% | 99% |
| decimals, add / sub | 286 | 29.7% | 100% |
| decimals, multiply | 286 | 2.1% | 100% |
| order of operations | 286 | 18.9% | 100% |
| powers | 186 | 30.1% | 90% |
| square / cube roots | 199 | 0.0% | 51% |
| percentages | 286 | 15.4% | 100% |
| fractions | 286 | 0.0% | 86% |
| greatest common divisor | 286 | 0.0% | 42% |
| least common multiple | 286 | 0.0% | 77% |
| factorial / choose / permutations | 101 | 1.0% | 89% |
| sums of series | 286 | 0.0% | 63% |
| rounding | 286 | 0.0% | 76% |
| linear equations in x | 286 | 1.0% | 100% |
| word problems, one step | 286 | 1.0% | 88% |
| word problems, two steps | 286 | 0.0% | 78% |
| average | 286 | 0.0% | 80% |
| median | 286 | 0.0% | 83% |
| unit conversion | 286 | 0.0% | 67% |
Model details
| Component | Setting |
|---|---|
| Width / depth | d_model 768, 18 blocks |
| Attention | 12 query heads × 64, 4 key/value heads (grouped-query), causal, per-head QK-norm |
| Positions | RoPE, θ = 500,000 |
| MLP | SwiGLU, hidden size 2,048 |
| Normalisation | RMSNorm, pre-norm; fp32 residual stream with fp16 sub-layers (the T4 has no bf16) |
| Embeddings | tied input/output, 49,152 rows |
| Bias / dropout | none |
Every layer is plain grouped-query attention: sliding-window, gated linear and DeltaNet-style alternatives were benchmarked on a T4 at 2k context and none beat flash-attention GQA on speed. KV cache: 18,432 bytes per token in fp16.
Tokenizer
Byte-level BPE with a SuperBPE-style second stage (merges may cross spaces; digits stay single tokens; 248 special tokens). On 63 MB of text from the training sources it needs about 9% fewer tokens than the DeepSeek-V4-Pro tokenizer (4.56 against 4.14 bytes per token) with a vocabulary 2.6 times smaller; it ties on Wikipedia and is worse on markdown and GSM8K. Only English, code and math were measured. File: tokenizer_candidates_v3/F-49152-f70/tokenizer.json.
Training
| Optimiser | Muon on hidden matrices (lr 0.012), AdamW on embeddings and norms (lr 2e-3) |
| Schedule | warmup, constant until 3.9B tokens, then linear decay to 10% of peak between 3.9B and 15B tokens, held to the 35B-token budget |
| Batch | 524,288 tokens per step; about 21-25k tokens/s on 2 × T4 |
| Data (up to step 10,800) | code 37.5% (Python 12%, Java/JavaScript 13.5%, HTML/CSS 4.1%, Markdown/TeX 4.5%, other 3.4%), FineWeb-Edu 17%, Nemotron-CC-Math 10.5%, arithmetic reasoning 7%, FineMath 6%, InfiWebMath 5%, Wikipedia 6%, Ultra-FineWeb 4%, Nemotron fact-seeking 3%, SmolTalk chat 3%, Nemotron SFT-math 1% |
| Data (from step 10,800) | a revised mixture, ramped in gradually: less Nemotron-CC-Math (10.5 → 5.5%), FineMath and InfiWebMath; more math reasoning (IFM Math-Reasoning 3.0%), proof-pile-2 (2.6%), decimal arithmetic and Orca-Math word problems (2.3%); web (30%), chat (3%) and code (37.5%) unchanged. Every benchmark above was measured on the earlier mixture |
Other measurements (step 10,900)
- MMLU, 5-shot, all 14,042 questions: 24.7% ± 0.7 (chance 25%; Zenyx-V3 26.9%).
- Python, twelve small functions run against unit tests (primes, factorial, palindrome, gcd, Fibonacci, FizzBuzz and similar): signature + docstring as plain completion 8 of 12 with greedy decoding and 43% of sampled attempts pass (8 samples per task, T = 0.6); chat format 5 of 12 and 34%; forcing
<think>first lowers it to 2 of 12 and 7%. (Step 6,000: 4 of 12 and 25%.) It still fails harder tasks. - Letter-choice multiple choice in chat format is at chance (SciQ 19%, ARC-Easy 23%, ARC-Challenge 25%, 200 questions each) although the standard scoring of the same questions gives 75%, 45% and 24%: the model has the knowledge but not the "read four lettered options and pick one" skill.
- Decode speed, one T4, fp16, CUDA graph +
torch.compile: 349 tokens/s at batch 1, 2,572 tokens/s total at batch 8, 9,475 tokens/s at batch 64. - Long generation: greedy decoding repeats itself within about 30 words; suggested: sampling (temperature 0.8, top-k 50) with a repetition penalty of 1.1-1.2.
Planned post-training
A supervised fine-tuning stage is being prepared; no instruct model has been released and no result exists yet. ChatML with the tokenizer's real special tokens (<|im_start|>, <|im_end|>, <think>, </think>, <tool_call>, <tool_response>), loss on assistant tokens only; system prompts that must win over conflicting user instructions; a Reasoning effort: none | low | medium | high line with reasoning budgets of 256 / 512 / 1,024 tokens; English, code and math data only, audited before inclusion. The numbers in this card are the pre-SFT baseline.
Intended use and limitations
- Use: research, and as a starting point for fine-tuning. Not an assistant: it ignores system prompts and may continue a prompt with an unrelated document.
- Facts: expect errors and invented details; long generations repeat themselves.
- Arithmetic works in its own chat format only (see above) and not for big numbers, negatives, decimals, fractions, powers, algebra or word problems.
- Reasoning, MMLU, letter-choice multiple choice: at chance. Code: passes 8 of 12 small functions, not harder ones. The
<think>habit exists only for arithmetic. - Context 2,048 tokens; English only; not evaluated for bias or harmful content.
- Evaluation: one training run, mid-training checkpoints; Fact1000 and the question sets are written by the author; Muse2 numbers use an unofficial reimplementation.
Repository contents
| Path | Contents |
|---|---|
checkpoints/<run>/step_N/ |
model weights (model.pt), optimiser state, training state; only the two newest checkpoints are kept |
tokenizer_candidates_v3/ |
tokenizers, including the one used |
wheels/, runs/, data/ |
flash-attention build, training manifests |
The model code is not published yet.