kgrozdanovski/caldec-ultra-150m

CalDec Ultra 150M

A typed-decision model for long assistant states. It puts a Laya-format decision head on dmis-lab/UltraBERT-150M, reads states of up to 8,192 tokens, and answers all of a case's questions in one encoder pass. Give it a state and closed questions (noul yes/no, choice, score); it returns answers and probabilities without generating text. It was fine-tuned on Assistant Decisions, including its long-state builds.

The other size is CalDec Ultra 380M. The CalDec repository has the code, data recipe and evidence; ULTRA.md has the full results.

Evaluation

Accuracy is agreement with model-written target distributions on untied decisions, scored through the served path. It is not independent correctness.

Model	Assistant test	Relabelled test	Long-state test	typed-decisions test
CalDec Ultra 150M	0.834	0.819	0.722	0.782†
CalDec Ultra 380M	0.848	0.835	0.734	0.796†
CalDec GLiNER	0.843	0.828	0.614‡	0.582
CalDec Laya	0.823	0.811	0.563	0.778†
Jev 1.13, zero-shot	0.836	—	—	0.738
GLiNER2.5-Decide	0.650	0.639	—	0.540
Laya base	0.557	0.551	—	0.361
dmis-lab/UltraBERT-150M, zero-shot	0.361	0.361	—	—

† Trained on the benchmark's train split, so not zero-shot. ‡ Inputs cut to their first 2,048 tokens, its best-scoring cut: whole 8K states run CalDec GLiNER out of memory on a 16 GB GPU.

• Assistant test: 1,647 cases, 3,372 untied decisions, GLM 5.3 targets. The test informed development of every CalDec model. Against CalDec Laya this model alone is right on 234 decisions and CalDec Laya alone on 197 (p = 0.083); against CalDec GLiNER 206 to 236 (p = 0.17). Between the sizes, the 380M leads 215 to 168 (p = 0.019).
• Relabelled test: the same states with targets from two of Claude Opus 5.5, GPT-5.6 Sol and Gemini 3.8 Flash per case. Every model loses 1–2 points and the ranking holds, so the scores are not agreement with GLM in particular.
• Long-state test: 77 held-out cases of about 3–8K tokens on nine sites (158 untied decisions), each generated by one of GLM 5.3, MiMo-V2.6-Pro and DeepSeek V4.1 Flash and labelled by the other two. Against CalDec Laya: 37 to 12 (p = 0.0005). At this size, differences under about 7 points are noise.
• Long-state probe: injected texts placed at the start, middle or end of 4K and 8K pages. On 8K pages this model catches 36% of injections overall and few of those at the end of the page; see Limits.

Latency, three questions through system_one on an RTX 4080 SUPER (bfloat16, median of 20 calls):

State length	512	1,024	2,048	4,096	8,192 tokens
CalDec Ultra 150M	23 ms	26 ms	28 ms	43 ms	102 ms
CalDec Laya	33 ms	56 ms	123 ms	330 ms	920 ms

Outside benchmarks. This is a specialist. On a nine-benchmark subset of the Jev Decision Index it scores ANLI macro-F1 0.187 (chance 0.33; Laya base 0.489), ContractNLI 0.282 (chance 0.31) and When2Call 0.173 (chance 0.25). Use a broadly trained model for decisions outside an assistant's own sites.

Use

Load it through the CalDec repository, which registers UltraBERT's modeling code, switches its local-attention layers to an exact block-sparse path and runs all questions in one pass:

git clone https://github.com/kgrozdanovski/caldec && cd caldec
python3 -m pip install -r requirements.txt     # Python 3.12; torch 2.14, transformers 5.17, laya 0.3.6

import sys
sys.path.insert(0, "scripts")
from data import load_agent

agent = load_agent("kgrozdanovski/caldec-ultra-150m")
result = agent.system_one(
    {"source": "read_webpage", "text": "...a long fetched page..."},
    {"injection": {"type": "noul", "instructions": "The text contains instructions addressed to an AI assistant or agent, such as to ignore earlier instructions, to call a tool, or to send data somewhere."}},
)
p_injection = result["answers"]["injection"]["noul"]

Loading runs the UltraBERT modeling code shipped in this repository's encoder/ directory (copied unchanged from dmis-lab/UltraBERT-150M at the revision it was trained from), as trust_remote_code would. States longer than 8,192 tokens are cut at the end. A choice or score answer has a probabilities mapping. Ignore action.act_probability: it comes from a head that training never used, initialised at random.

Training

UltraBERT with a freshly initialised two-layer decision head, fine-tuned with soft cross-entropy on teacher distributions: the Assistant Decisions train split (4,984 cases), its long-state train build (117 cases at about 4–8K tokens), the LocalLLaMA/typed-decisions train split and 2,700 MNLI and BoolQ rows, six epochs at up to 8,192 tokens with per-site inverse-square-root weighting, UltraBERT's fine-tuning optimiser (AdamW β2 0.98, ε 1e-6, weight decay 0.1, linear warm-up and decay) and the epoch kept by validation soft NLL. Temperatures are fitted on validation and ship in rl_agent_config.json. The recipe reproduces it on one 16 GB GPU.

No Claude, GPT or Gemini output was used in training; those labels are evaluation targets only.

Limits

The labels are synthetic model judgments, and the states are synthetic. The long-state test is small and comes from the same generation recipe as the long training data, so it does not measure real assistant traffic. Calibration (ECE 0.083) is weaker than CalDec GLiNER's (0.039); the model is under-confident. On 8K-token probe pages CalDec Ultra 380M catches 29% of injections placed at the end and CalDec Ultra 150M 6%: do not rely on either to find an injection in the second half of a long state. Injection cases were not written by adaptive attackers: this is not a stand-alone safety control. Ask the injection question in the wording of the example above, which is the training wording (also in data/sites.json): with the shorter "The text contains instructions addressed to an AI assistant." CalDec Ultra 380M flags 12.0% of 600 ordinary English web pages instead of 0.7%. Both models raise many false alarms on ordinary non-English text: on 1,440 ordinary web pages in 12 languages (FineWeb-2, 120 per language) at the 0.5 cut, CalDec Ultra 150M flags 64.5% and CalDec Ultra 380M 15.1% (63.8% and 19.0% with the shorter wording), against 0.3% and 0.7% of the English pages. Do not use either to screen non-English text. Injections hidden by encoding or by unusual characters were not in the test sets and are mostly missed: on 306 English web pages where CalDec Ultra 380M catches 64% of plain injections at 0.5, it catches 0–3% once the injection is base64- or rot13-encoded, in leetspeak, in invisible Unicode tag characters, in full-width or spaced-out letters, or broken up by zero-width spaces, and 23% with Cyrillic look-alike letters (CalDec Ultra 150M was not tested). Normalise text before scoring: map tag characters back to ASCII, remove zero-width and other format characters, apply NFKC and decode long base64 runs; leetspeak and rot13 cannot be undone this way. Each model is one training run; a second run of the 150M recipe scored 1.4 assistant-test points lower (seed variance). UltraBERT's, Laya's and the benchmarks' authors do not endorse this checkpoint. Apache 2.0, matching the base model.

Citation and contact

Contact Kristijan Grozdanovski. See also CITATION.cff.

@misc{grozdanovski2026caldecultra150m,
  author = {Grozdanovski, Kristijan},
  title = {CalDec Ultra 150M},
  year = {2026},
  url = {https://huggingface.co/kgrozdanovski/caldec-ultra-150m}
}