Model card: coverage over all 43 benchmarks (37 in the index)

ee410e2 verified 1 day ago

9.23 kB

language: en
license: apache-2.0
library_name: peft
base_model: Qwen/Qwen3.5-2B-Base
base_model_relation: adapter
pipeline_tag: text-classification
tags:
  - sieve
  - decision-model
  - calibration
  - lora
  - multiple-choice
  - qwen3.5
  - decision-index
model-index:
  - name: Sieve-2B-v2
    results:
      - task:
          type: text-classification
          name: typed decision (choice / noul)
        dataset:
          type: mixed
          name: >-
            Decision Index 0.3.2 leaderboard (public suite and the maintainers'
            private tests)
        metrics:
          - type: decision_index
            value: 30.84
            name: Decision Index 0.3.2, Full score (leaderboard)
      - task:
          type: text-classification
          name: typed decision (choice / noul)
        dataset:
          type: mixed
          name: >-
            Decision Index 0.3 public suite (140,178 scored requests; the index
            averages 37 benchmarks)
        metrics:
          - type: decision_index
            value: 37.26
            name: Decision Index 0.3, public index (chance-corrected)

Sieve-2B-v2

Sieve-2B-v2 is a decision model from the Sieve family. You give it a piece of text or JSON (the state) and typed questions, and it returns a calibrated probability for every option of every question. It reads the state once, scores the options directly, and never generates text.

It is a LoRA adapter (rank 64) and a scalar scorer head on the first 16 of 24 blocks of Qwen/Qwen3.5-2B-Base (revision b1485b2f). The code is at github.com/sthanika-ai/Sieve.

It was first published as Sieve-2B-Plus, and its Decision Index run files keep that name.

Summary


Backbone	Qwen/Qwen3.5-2B-Base (revision `b1485b2f`), blocks 1-16 of 24
Design	option fork
Trained weights	LoRA r=64, α=128 on the attention, MLP and Gated DeltaNet projections of blocks 1-16, and a scalar head
Question types	`choice` and `noul`
Decision Index	Full score 30.84 and public score 37.26 on the leaderboard (edition 0.3.2)
Temperature	T = 1.4785
Token limits	32,768 tokens for the state, the question and the longest option; trained on inputs up to 8,192
GPU memory	the weights take about 3 GB in bf16

How it works

The state and the question are encoded once. Every option is listed in the prompt, in an order fixed by a hash of each option's text. The encoded state is then copied once per option, and each option is scored as its own continuation of that copy. No option sees another, so reordering the options cannot change the answer: that holds by construction, not by training. A scalar head reads the last token of each option; a softmax over the options, divided by one built-in temperature, gives the probabilities.

	parameters
run at inference (embeddings + blocks 1-16 of the base, adapter, head)	1.47 B
trained (LoRA 44,851,200 + scorer head 1,053,697)	45.9 M
full base model (`Qwen/Qwen3.5-2B-Base` text model), for reference	1.88 B

type	options	returns
`choice`	the caller's keys, optionally described	a probability per key
`noul`	yes / no, scored as a 2-option choice	P(yes)

Use

pip install "sieve-decisions[cuda,serve] @ git+https://github.com/sthanika-ai/Sieve"
pip install --no-build-isolation causal-conv1d==1.7.0   # recommended: the fused convolution kernel

from sieve import load_sieve, decide

m = load_sieve("sthanika-ai/Sieve-2B-v2")
decide(m, {"subject": "Duplicate charge on invoice 4411", "body": "Billed twice. Refund today or we cancel."},
       {"department": {"type": "choice", "instructions": "Which team handles this?",
                       "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs and outages"}},
        "churn_risk": {"type": "noul", "instructions": "Does the customer threaten to cancel?"}})

Loading. load_sieve downloads Qwen3.5-2B-Base once and applies this adapter and head.
The model's own call. m.choice(state, question, options) takes a plain list of options and returns {"decision", "index", "confidence", "probs"}.
Server. sieve-serve --model sthanika-ai/Sieve-2B-v2.
Decision Index. python -m decision_index run --engine sieve.decision_index_engine:SieveEngine --option model=sthanika-ai/Sieve-2B-v2 --out runs/Sieve-2B-v2.
Token limits. A request longer than 32,768 tokens (state, question and longest option) is refused, never trimmed.

Results

Decision Index

On the Decision Index leaderboard (edition 0.3.2, board data of 2026-10-12), Sieve-2B-v2 has a Full score of 30.84 and a public score of 37.26. The Full score ranks the board: the public benchmarks count 20%, the maintainers' private tests of the same skills 50% and their private tasks from new domains 30%.

Our own run of the edition 0.3 public suite, with the official kit, gives a public index of 37.26.

Coverage. All 140,178 scoreable requests over 43 benchmarks were run: 140,176 answered and 2 refused (their options have identical text), with no truncation and no error. The index averages 37 of the benchmarks.
Results. sthanika-ai/Sieve-2B-v2-decision-index-results.

area	0.3 public	0.2.1
Knowledge & Reasoning	16.6	21.0
Language Understanding	35.9	37.2
Retrieval & Classification	57.4	57.4
Tools & Automation	56.6	56.6
Arts & Human Taste	18.5	17.8
Index	37.26	38.65

GSM8K. Under 0.2.1 this model scores 88 on GSM8K; under 0.3, where the wrong options are answers to other GSM8K problems, it scores 4 (accuracy at chance). Its GSM8K training rows used the 0.2.1 style of wrong options (fixed offsets of the right answer), and the model learned to recognise that pattern rather than to solve the problems. Treat it as having no GSM8K-level arithmetic.

Calibration

One temperature, T = 1.4785, fitted by NLL on a 9,976-question calibration split that covers every track and was never trained on. It never changes which option wins. On a separate 2,560-question validation split:

	NLL	ECE	confidence − accuracy	accuracy
raw (T = 1)	0.393	0.041	+0.039	85.08%
built in (T = 1.48)	0.371	0.012	−0.003	85.08%

Separate temperatures for yes/no questions or by option count did not do better, so the model ships one.

Latency

Single process, one request at a time, bf16, A100 80GB, 400 uniformly sampled suite requests: median 97 ms, mean 124 ms, 80th percentile 110 ms.

Training

Base. Qwen/Qwen3.5-2B-Base at revision b1485b2f, a hybrid of Gated DeltaNet and full-attention blocks; only blocks 1-16 of 24 run.
Adapter. LoRA r=64, α=128 on the attention, MLP and Gated DeltaNet projections of blocks 1-16, plus a scalar head (LayerNorm → 512 → 1).
Recipe. 1 epoch, AdamW, lr 5e-5 with 3% warmup and cosine decay to 10%, batch 8 questions, gradient-norm clip 1.0, trained through the same option fork that serves the model.
Data. 163,425 questions from 154 decision tracks, sampled at 40% per track from a public + synthetic mix and ordered so every part of the epoch sees every track. It covers intent and topic classification, NLI and fact-checking, commonsense and knowledge questions, safety and moderation, preference judging, tool selection and routing, retrieval and long documents, and rule-based synthetic reasoning (dates, numbers, policies, lookups).
- 36,984 questions (22.6%) come from the train or dev splits of datasets whose test splits appear in the Decision Index: HellaSwag 8,246, WinoGrande 8,246, iSarcasmEval 5,318, CLINC150 3,027, BANKING77 2,457, ESCI 2,374, GSM8K 2,024, SGD 1,408, RAGTruth 1,098, HoVer 968, ARC 906, When2Call 470, ContractNLI 442. No test split was used.
- Each public dataset keeps its own licence.

Limitations

No arithmetic beyond what is in the options (see GSM8K above). Knowledge-heavy benchmarks (MMLU-Pro, GPQA, HLE) remain weak.
Single training run, one seed.
The 0.2.1 score of 38.65 is inflated by GSM8K; the 0.3 public index is the better number.
Requires transformers==5.17 (qwen3_5 architecture) and peft>=0.21.

Files

file	contents
`adapter_model.safetensors`	the LoRA adapter (rank 64)
`head.safetensors`	the scalar scorer head
`config.json`	backbone, read layer, temperature, context limit and training settings
`tokenizer.json`, `tokenizer_config.json`, `vocab.json`, `merges.txt`	the Qwen3.5 tokenizer

License

Apache-2.0 for the adapter and head in this repository, and for the code at github.com/sthanika-ai/Sieve. The Qwen3.5 base model is Apache-2.0. The training datasets keep their own licences.