SmolLM3-SWA-8K-pre-linear-p30-scaled

SmolLM3-3B continued pretraining at 8K context with RoPE-SWA + NoPE (RoPE layers local, window 4096; NoPE layers global), Pre-Linear dropout p=0.3, and input-dependent attention scaling — each attention head's query is scaled by a learned, per-position, per-head factor beta_h(p, x) = 1 + coef_h(x) * log2(p + 2) ** pow_h(x).

The scaler weights live inside the checkpoint, and the two bundled files (configuration_smollm3_scaled.py, modeling_smollm3_scaled.py) rebuild the wrapper at load time — so loading needs trust_remote_code=True, but nothing else:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "vapavlo/SmolLM3-SWA-8K-pre-linear-p30-scaled"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",   # sdpa / eager work too
).cuda().eval()

ids = tok("The main driver of climate change is", return_tensors="pt").input_ids.cuda()
out = model.generate(ids, max_new_tokens=32, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))