You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

GPT-2-style-tinystories-6L384d-ctx256-page-gauss-2026-10-28

A small GPT-2-style language model trained from scratch on TinyStories, with page gauss attention: page attention with a Gaussian kernel: each key is replaced by the average of its family, A = exp(−‖kᵢ − kⱼ‖² / 2h²) for j ≤ i, rows normalised, K_page = ÃK, then softmax(Q·K_pageᵀ/√d)·V; the width h is learned per head.

Part of a paired comparison of causal and page gauss attention: for each seed, both models start from the same initial weights and see the same batches. All attention is written by hand (no FlashAttention). One folder per seed (seed_0/, seed_1/, …).

parameters 19.1 M
layers × heads 6 × 6 (head dim 64)
d_model / d_ff 384 / 768
context 256 tokens
tokenizer byte-level BPE, 4096 tokens, trained on TinyStories
training 20,000 steps × 64 × 256 tokens, AdamW, lr 0.001 → 0.0001, bfloat16

Exact validation loss (whole TinyStories validation split)

seed loss perplexity training time A6000 (h)
0 1.3974 4.045 0.37
1 1.4047 4.074 0.36
2 1.3984 4.049 0.36
3 1.4055 4.078 0.37
4 1.4033 4.068 0.75
5 1.4032 4.068 0.75
6 1.3963 4.040 0.36
7 1.3963 4.040 0.36
8 1.3982 4.048 0.36
9 1.4028 4.067 0.36

Mean over 10 seed(s): 1.4006 ± 0.0036. Per-story losses are in seed_*/val_per_story.npz (sums and token counts per validation story).

Usage

import json, sys, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from tokenizers import Tokenizer

path = snapshot_download("analist/GPT-2-style-tinystories-6L384d-ctx256-page-gauss-2026-10-28", allow_patterns=["seed_0/*", "modeling.py", "tokenizer.json"])
sys.path.insert(0, path)
from modeling import GPT

cfg = json.load(open(f"{path}/seed_0/config.json"))
model = GPT(cfg["variant"], cfg["vocab_size"], cfg["d_model"], cfg["d_ff"], cfg["n_heads"], cfg["n_layers"],
            max_len=cfg["block_size"], hid_factor=cfg["hid_factor"])
model.load_state_dict(load_file(f"{path}/seed_0/model.safetensors"))
model.eval()

tok = Tokenizer.from_file(f"{path}/tokenizer.json")
torch.manual_seed(0)
ids = model.generate(torch.tensor([tok.encode("Once upon a time").ids]), max_new_tokens=200)[0].tolist()
if cfg["eot_id"] in ids:
    ids = ids[:ids.index(cfg["eot_id"])]
print(tok.decode(ids))
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train analist/GPT-2-style-tinystories-6L384d-ctx256-page-gauss-2026-10-28