PoliteChainsaw/Lewi-Surge-Preview-3B-Base

Lewi Surge Preview 3B (Base)

An open mixture-of-experts language model for programming, trained from scratch by one person in Ukraine on free and donated compute.

This is the base model after code-focused pretraining and annealing. It is not instruction-tuned: use it for completion, few-shot prompting, or as a starting point for your own fine-tuning. For a chat checkpoint, see Lewi-Surge-Preview-3B-Chat-Early.

💛 This model is built without funding. The bottleneck is GPU time: about $3 buys one hour on an NVIDIA GH200. If you find Lewi useful or just want to see where it goes, you can support the next training stages on Ko-fi. Just $50 would unlock ~120 hours on a GH200: the provider gives us access once we make a $50 deposit, and our free credits cover the rest.

Model

Parameters	~3.45B total, ~0.5B active per token
Layers	16 = 4 × (3 Gated DeltaNet linear attention + 1 full attention)
Experts	128 per layer, top-6 routed + 1 shared, expert size 512
Hidden size	1024, 8 query / 2 KV heads
Vocabulary	131,072 BPE + special tokens (Qwen3.6 vocabulary, pruned)
Context in training	4,096 tokens (128k long-context stage is in progress)
Precision	bfloat16

Only 4 of 16 layers keep a KV cache (2 KV heads each), about 4 KB per token.

How it was trained

1. Stage 1: a 1.4B model (12 layers, 64 experts) trained from scratch on ~3.2B tokens of FineWeb-Edu.
2. Growth: grown to 3.45B without restarting: experts split 64 → 128, plus one more block of 4 layers.
3. Code-first pretraining: ~1.6B tokens of curated code (with FIM), instructions, textbooks, math, and Ukrainian/Spanish/Portuguese/French/German web text.
4. Annealing (this checkpoint): ~1.4B tokens, more than half conversations, learning rate decayed to zero.

JAX + MaxText, Muon optimizer. Hardware: rented/free H100/H200 sessions. Code: github.com/prostos2162-maker/lewi-surge.

Results

Before → after code-focused annealing (greedy decoding, standard prompts, local runs):

Benchmark	Metric	Before	After (this model)
HumanEval	pass@1	18.9%	25.0%
MBPP (sanitized)	pass@1	23.7%	29.6%
HellaSwag	acc_norm	41.9%	43.4%
ARC-Challenge	acc_norm	26.5%	27.4%

Knowledge-heavy benchmarks barely move at this size and token count.

Against small open base models

Same script for every model: base checkpoints, 0-shot, greedy; HumanEval uses the better of two standard prompt formats per model. Lewi is a work in progress; the others are final releases.

Model	Training tokens	HumanEval	MBPP	ARC-C	HellaSwag
Qwen2.5-0.5B (Alibaba)	18T	29.9	37.7	32.3	49.5
Qwen2.5-Coder-0.5B (Alibaba)	23.5T	27.4	48.2	30.2	47.3
Lewi Surge Preview (this model)	7.3B	25.0	29.6	27.4	43.4
SmolLM2-1.7B (Hugging Face)	11T	23.2	44.0	47.9	61.3
Granite 3.0 1B-A400M (IBM)	10T	22.6	36.2	38.2	59.3
Pythia-1.4B (EleutherAI)	300B	5.5	7.0	28.6	49.3

On code, Lewi lands between models trained on 1,400 to 3,200 times more data. On knowledge-heavy tasks it is behind, which is what more data fixes.

Usage

Requires transformers >= 5.0 (architecture Qwen3_5MoeForCausalLM).

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "PoliteChainsaw/Lewi-Surge-Preview-3B-Base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")

prompt = 'def is_prime(n: int) -> bool:\n    """Return True if n is a prime number."""\n'
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Runs on a single consumer GPU (~7 GB in bf16).

Limitations

• A small model trained on well under 10B tokens in total: factual knowledge is weak and it can state wrong things confidently.
• Best at short Python functions; other languages and long programs are much weaker.
• Not safety-tuned. Do not use it for anything where mistakes matter.

Links

Project page · Code · Support on Ko-fi