roneneldan/TinyStories
Viewer • Updated • 2.14M • 183k • 1.25k
A small GPT-2-style language model trained from scratch on TinyStories, with causal attention: standard softmax attention: softmax(QKᵀ/√d)·V.
Part of a paired comparison of causal and page gauss attention: for each seed, both models start from the same initial weights and see the same batches. All attention is written by hand (no FlashAttention). One folder per seed (seed_0/, seed_1/, …).
| parameters | 42.0 M |
| layers × heads | 8 × 8 (head dim 64) |
| d_model / d_ff | 512 / 1024 |
| context | 512 tokens |
| tokenizer | byte-level BPE, 4096 tokens, trained on TinyStories |
| training | 20,000 steps × 32 × 512 tokens, AdamW, lr 0.0006 → 6e-05, bfloat16 |
| seed | loss | perplexity | training time (h) |
|---|---|---|---|
| 0 | 1.3289 | 3.777 | 1.28 |
Mean over 1 seed(s): 1.3289. Per-story losses are in seed_*/val_per_story.npz (sums and token counts per validation story).
import json, sys, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from tokenizers import Tokenizer
path = snapshot_download("analist/GPT-2-style-tinystories-8L512d-ctx512-causal-2026-10-28", allow_patterns=["seed_0/*", "modeling.py", "tokenizer.json"])
sys.path.insert(0, path)
from modeling import GPT
cfg = json.load(open(f"{path}/seed_0/config.json"))
model = GPT(cfg["variant"], cfg["vocab_size"], cfg["d_model"], cfg["d_ff"], cfg["n_heads"], cfg["n_layers"],
max_len=cfg["block_size"], hid_factor=cfg["hid_factor"])
model.load_state_dict(load_file(f"{path}/seed_0/model.safetensors"))
model.eval()
tok = Tokenizer.from_file(f"{path}/tokenizer.json")
torch.manual_seed(0)
ids = model.generate(torch.tensor([tok.encode("Once upon a time").ids]), max_new_tokens=200)[0].tolist()
if cfg["eot_id"] in ids:
ids = ids[:ids.index(cfg["eot_id"])]
print(tok.decode(ids))