--- license: mit language: - en library_name: safetensors pipeline_tag: text-generation tags: - rust - geometric - language-model - research - pre-alpha datasets: - roneneldan/TinyStories - HuggingFaceTB/smollm-corpus - HuggingFaceTB/smoltalk - HuggingFaceH4/ultrachat_200k - caseyallard/uor-r4-data base_model_relation: finetune --- # UOR-R4 Geometric Language Model, 214M checkpoints Research checkpoints of the **UOR-R4 Geometric Language Model**, a Rust research model that replaces a transformer's run-time machinery with geometry: R4/S3/H4 quaternion state and transport, prime/UOR addressing, exact Z[phi] arithmetic and fixed zeta-zero phases, together with exact addressed memory and learned typed operators. **Status: pre-alpha.** No checkpoint here holds a useful conversation. They are published as research artifacts so the training runs and the architecture can be inspected and reproduced. - Code (MIT): https://github.com/UOR-Foundation/uor-r4 - Tracker: https://github.com/UOR-Foundation/uor-r4/issues/2028 - Data: https://huggingface.co/datasets/caseyallard/uor-r4-data ## Architecture All four checkpoints share one 214M-parameter geometric stack: vocabulary 4096 (`tokenizer.json`, learned BPE), context 384, width 1536, 24 heads, layer pattern `rrarrarrarrarrar` (`r` = geometric read/transport layer, `a` = addressed contextual attention layer), `read = l2`, rotation enabled. The chat checkpoints add a small pointer head (dim 32). Training is offline Rust (candle, CUDA) and may use floating point and matrix multiplication. The project's serving target (decision D11) is different: no floating point, no multiplier instruction and no transformer backbone in served kernels. These safetensors are training-side checkpoints, not that serving artifact; exported `.lut` artifacts serve through `uor-chat --stack ` (crate `uor-r4-integer`). ## Checkpoints Each folder holds `model/config.json`, `model/model.safetensors` and a sanitized `report.json` (the training report; absolute local paths replaced by `/`). | Folder | Source run | Description | |---|---|---| | `base-tinystories/` | geo-214m-e448de86 | Base model trained on TinyStories (weight 0.7), TinyDialogues (0.1) and chat-v0-p2 (0.2). Final NLL 1.032 on the TinyStories validation set; v4 memory panel 31/40. | | `base-planA/` | planA-214m-2b425ccd | Initialized from `base-tinystories`, then 150k steps (batch 32, context 384, about 1.84B tokens, 42,463 s on one RTX PRO 6000; 46.8k tokens/s early, 43.4k average) on SmolLM-Corpus shards (FineWeb-Edu and Cosmopedia-v2 components, weights 0.35/0.35/0.075x4). Dev NLL 2.903 to 2.073 on a FineWeb validation shard. | | `chat-planA-b8/` | chat-214m-planA-af-b8 | `base-planA` fine-tuned on the 173M-token "mixB" corpus for 8000 steps (1308 s). | | `chat-smoltalk-b16/` | chat-214m-smoltalk-b16 | Fine-tuned on the same mixB corpus for 16000 steps (3152 s). | The mixB corpus is: project-generated memory dialogues (`mw-balp`, `mw-orcp`, published in the data repo), dialogue-recall-v2, and the SmolTalk magpie-ultra / UltraChat-derived `chat-v0-p2` set. The mixing manifest is `memory-dialogue/fine-tune-mix-manifest.json` in the data repo. The third-party text itself is not republished. ## Results and limitations - The base models are small-corpus language models; the NLL figures above are the only quality numbers claimed. - The chat fine-tunes can rewrite and summarize short text. Code and arithmetic answers are wrong. They are **not instruction-safe** and should not be used for anything but inspecting the research. - General prose, general reasoning, frontier capability and complete-path energy savings are not established by this project. A passed authored panel is not a general capability claim. - Training and evaluation numbers are bound to the exact artifact, data, operator and budget recorded in each `report.json`. ## Data licenses and attribution Weights are released under the MIT license. The training data carries its own terms, which downstream users should check: - SmolLM-Corpus (FineWeb-Edu, Cosmopedia-v2): odc-by. https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus - SmolTalk: Apache-2.0 for new subsets, others follow their sources. https://huggingface.co/datasets/HuggingFaceTB/smoltalk - TinyStories: cdla-sharing-1.0. https://huggingface.co/datasets/roneneldan/TinyStories - TinyDialogues: MIT. https://huggingface.co/datasets/styfeng/TinyDialogues - UltraChat 200k: MIT. https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k ## Start here ```bash git clone https://github.com/UOR-Foundation/uor-r4 && cd uor-r4 pip install -U "huggingface_hub[cli]" hf download caseyallard/uor-r4-geometric-214m --include "base-planA/*" --include tokenizer.json --local-dir ckpt cargo run --release -p uor-r4-training --example geometric-stack -- \ sample model=ckpt/base-planA/model tokenizer=ckpt/tokenizer.json out=ckpt/base-planA ``` One checkpoint is about 856 MB. On an NVIDIA GPU, build with `--features cuda` and pass `device=cuda`. To rent a GPU pod from your own Runpod account with the project's tooling, see [pods-quickstart.md](https://github.com/UOR-Foundation/uor-r4/blob/main/docs/compute/pods-quickstart.md). For a small model you can train on a laptop CPU in seconds, see "Start here" on the [dataset card](https://huggingface.co/datasets/caseyallard/uor-r4-data). ## How to run From a checkout of the GitHub repository: ```bash cargo run --release -p uor-r4-training --example geometric-stack -- \ sample model=/model tokenizer=tokenizer.json out= cargo run --release -p uor-r4-training --example geometric-stack -- \ evaluate model=/model tokenizer=tokenizer.json out= ``` where `` is one checkpoint folder (for example `base-planA`) and `tokenizer.json` is the file at the root of this repository. ## Citation See `CITATION.cff` in https://github.com/UOR-Foundation/uor-r4.