You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

GPT-2-style-tinystories-8L512d-ctx512-causal-2026-10-28

A small GPT-2-style language model trained from scratch on TinyStories, with causal attention: standard softmax attention: softmax(QKᵀ/√d)·V.

Part of a paired comparison of causal and page gauss attention: for each seed, both models start from the same initial weights and see the same batches. All attention is written by hand (no FlashAttention). One folder per seed (seed_0/, seed_1/, …).

parameters 42.0 M
layers × heads 8 × 8 (head dim 64)
d_model / d_ff 512 / 1024
context 512 tokens
tokenizer byte-level BPE, 4096 tokens, trained on TinyStories
training 20,000 steps × 32 × 512 tokens, AdamW, lr 0.0006 → 6e-05, bfloat16

Exact validation loss (whole TinyStories validation split)

seed loss perplexity training time (h)
0 1.3289 3.777 1.28

Mean over 1 seed(s): 1.3289. Per-story losses are in seed_*/val_per_story.npz (sums and token counts per validation story).

Usage

import json, sys, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from tokenizers import Tokenizer

path = snapshot_download("analist/GPT-2-style-tinystories-8L512d-ctx512-causal-2026-10-28", allow_patterns=["seed_0/*", "modeling.py", "tokenizer.json"])
sys.path.insert(0, path)
from modeling import GPT

cfg = json.load(open(f"{path}/seed_0/config.json"))
model = GPT(cfg["variant"], cfg["vocab_size"], cfg["d_model"], cfg["d_ff"], cfg["n_heads"], cfg["n_layers"],
            max_len=cfg["block_size"], hid_factor=cfg["hid_factor"])
model.load_state_dict(load_file(f"{path}/seed_0/model.safetensors"))
model.eval()

tok = Tokenizer.from_file(f"{path}/tokenizer.json")
torch.manual_seed(0)
ids = model.generate(torch.tensor([tok.encode("Once upon a time").ids]), max_new_tokens=200)[0].tolist()
if cfg["eot_id"] in ids:
    ids = ids[:ids.index(cfg["eot_id"])]
print(tok.decode(ids))
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train analist/GPT-2-style-tinystories-8L512d-ctx512-causal-2026-10-28