GPT-2-style-tinystories-6L384d-ctx256-page-gauss-2026-10-28
A small GPT-2-style language model trained from scratch on TinyStories, with page gauss attention: page attention with a Gaussian kernel: each key is replaced by the average of its family, A = exp(−‖kᵢ − kⱼ‖² / 2h²) for j ≤ i, rows normalised, K_page = ÃK, then softmax(Q·K_pageᵀ/√d)·V; the width h is learned per head.
Part of a paired comparison of causal and page gauss attention: for each seed, both models start from the same initial weights and see the same batches. All attention is written by hand (no FlashAttention). One folder per seed (seed_0/, seed_1/, …).
| parameters | 19.1 M |
| layers × heads | 6 × 6 (head dim 64) |
| d_model / d_ff | 384 / 768 |
| context | 256 tokens |
| tokenizer | byte-level BPE, 4096 tokens, trained on TinyStories |
| training | 20,000 steps × 64 × 256 tokens, AdamW, lr 0.001 → 0.0001, bfloat16 |
Exact validation loss (whole TinyStories validation split)
| seed | loss | perplexity | training time A6000 (h) |
|---|---|---|---|
| 0 | 1.3974 | 4.045 | 0.37 |
| 1 | 1.4047 | 4.074 | 0.36 |
| 2 | 1.3984 | 4.049 | 0.36 |
| 3 | 1.4055 | 4.078 | 0.37 |
| 4 | 1.4033 | 4.068 | 0.75 |
| 5 | 1.4032 | 4.068 | 0.75 |
| 6 | 1.3963 | 4.040 | 0.36 |
| 7 | 1.3963 | 4.040 | 0.36 |
| 8 | 1.3982 | 4.048 | 0.36 |
| 9 | 1.4028 | 4.067 | 0.36 |
Mean over 10 seed(s): 1.4006 ± 0.0036. Per-story losses are in seed_*/val_per_story.npz (sums and token counts per validation story).
Usage
import json, sys, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from tokenizers import Tokenizer
path = snapshot_download("analist/GPT-2-style-tinystories-6L384d-ctx256-page-gauss-2026-10-28", allow_patterns=["seed_0/*", "modeling.py", "tokenizer.json"])
sys.path.insert(0, path)
from modeling import GPT
cfg = json.load(open(f"{path}/seed_0/config.json"))
model = GPT(cfg["variant"], cfg["vocab_size"], cfg["d_model"], cfg["d_ff"], cfg["n_heads"], cfg["n_layers"],
max_len=cfg["block_size"], hid_factor=cfg["hid_factor"])
model.load_state_dict(load_file(f"{path}/seed_0/model.safetensors"))
model.eval()
tok = Tokenizer.from_file(f"{path}/tokenizer.json")
torch.manual_seed(0)
ids = model.generate(torch.tensor([tok.encode("Once upon a time").ids]), max_new_tokens=200)[0].tolist()
if cfg["eot_id"] in ids:
ids = ids[:ids.index(cfg["eot_id"])]
print(tok.decode(ids))
Dataset used to train analist/GPT-2-style-tinystories-6L384d-ctx256-page-gauss-2026-10-28
Viewer • Updated • 2.14M • 181k • 1.25k