UOR-R4 Geometric Language Model, 214M checkpoints
Research checkpoints of the UOR-R4 Geometric Language Model, a Rust research model that replaces a transformer's run-time machinery with geometry: R4/S3/H4 quaternion state and transport, prime/UOR addressing, exact Z[phi] arithmetic and fixed zeta-zero phases, together with exact addressed memory and learned typed operators.
Status: pre-alpha. No checkpoint here holds a useful conversation. They are published as research artifacts so the training runs and the architecture can be inspected and reproduced.
- Code (MIT): https://github.com/UOR-Foundation/uor-r4
- Tracker: https://github.com/UOR-Foundation/uor-r4/issues/2028
- Data: https://huggingface.co/datasets/caseyallard/uor-r4-data
Architecture
All four checkpoints share one 214M-parameter geometric stack: vocabulary 4096 (tokenizer.json, learned BPE), context 384, width 1536, 24 heads, layer pattern rrarrarrarrarrar (r = geometric read/transport layer, a = addressed contextual attention layer), read = l2, rotation enabled. The chat checkpoints add a small pointer head (dim 32). Training is offline Rust (candle, CUDA) and may use floating point and matrix multiplication. The project's serving target (decision D11) is different: no floating point, no multiplier instruction and no transformer backbone in served kernels. These safetensors are training-side checkpoints, not that serving artifact; exported .lut artifacts serve through uor-chat --stack <ARTIFACT.lut> (crate uor-r4-integer).
Checkpoints
Each folder holds model/config.json, model/model.safetensors and a sanitized report.json (the training report; absolute local paths replaced by <local>/).
| Folder | Source run | Description |
|---|---|---|
base-tinystories/ |
geo-214m-e448de86 | Base model trained on TinyStories (weight 0.7), TinyDialogues (0.1) and chat-v0-p2 (0.2). Final NLL 1.032 on the TinyStories validation set; v4 memory panel 31/40. |
base-planA/ |
planA-214m-2b425ccd | Initialized from base-tinystories, then 150k steps (batch 32, context 384, about 1.84B tokens, 42,463 s on one RTX PRO 6000; 46.8k tokens/s early, 43.4k average) on SmolLM-Corpus shards (FineWeb-Edu and Cosmopedia-v2 components, weights 0.35/0.35/0.075x4). Dev NLL 2.903 to 2.073 on a FineWeb validation shard. |
chat-planA-b8/ |
chat-214m-planA-af-b8 | base-planA fine-tuned on the 173M-token "mixB" corpus for 8000 steps (1308 s). |
chat-smoltalk-b16/ |
chat-214m-smoltalk-b16 | Fine-tuned on the same mixB corpus for 16000 steps (3152 s). |
The mixB corpus is: project-generated memory dialogues (mw-balp, mw-orcp, published in the data repo), dialogue-recall-v2, and the SmolTalk magpie-ultra / UltraChat-derived chat-v0-p2 set. The mixing manifest is memory-dialogue/fine-tune-mix-manifest.json in the data repo. The third-party text itself is not republished.
Results and limitations
- The base models are small-corpus language models; the NLL figures above are the only quality numbers claimed.
- The chat fine-tunes can rewrite and summarize short text. Code and arithmetic answers are wrong. They are not instruction-safe and should not be used for anything but inspecting the research.
- General prose, general reasoning, frontier capability and complete-path energy savings are not established by this project. A passed authored panel is not a general capability claim.
- Training and evaluation numbers are bound to the exact artifact, data, operator and budget recorded in each
report.json.
Data licenses and attribution
Weights are released under the MIT license. The training data carries its own terms, which downstream users should check:
- SmolLM-Corpus (FineWeb-Edu, Cosmopedia-v2): odc-by. https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus
- SmolTalk: Apache-2.0 for new subsets, others follow their sources. https://huggingface.co/datasets/HuggingFaceTB/smoltalk
- TinyStories: cdla-sharing-1.0. https://huggingface.co/datasets/roneneldan/TinyStories
- TinyDialogues: MIT. https://huggingface.co/datasets/styfeng/TinyDialogues
- UltraChat 200k: MIT. https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k
Start here
git clone https://github.com/UOR-Foundation/uor-r4 && cd uor-r4
pip install -U "huggingface_hub[cli]"
hf download caseyallard/uor-r4-geometric-214m --include "base-planA/*" --include tokenizer.json --local-dir ckpt
cargo run --release -p uor-r4-training --example geometric-stack -- \
sample model=ckpt/base-planA/model tokenizer=ckpt/tokenizer.json out=ckpt/base-planA
One checkpoint is about 856 MB. On an NVIDIA GPU, build with --features cuda and pass device=cuda. To rent a GPU pod from your own Runpod account with the project's tooling, see pods-quickstart.md. For a small model you can train on a laptop CPU in seconds, see "Start here" on the dataset card.
How to run
From a checkout of the GitHub repository:
cargo run --release -p uor-r4-training --example geometric-stack -- \
sample model=<dir>/model tokenizer=tokenizer.json out=<dir>
cargo run --release -p uor-r4-training --example geometric-stack -- \
evaluate model=<dir>/model tokenizer=tokenizer.json out=<dir>
where <dir> is one checkpoint folder (for example base-planA) and tokenizer.json is the file at the root of this repository.
Citation
See CITATION.cff in https://github.com/UOR-Foundation/uor-r4.