jasonjimnz/tyness-boe-2.6b-pre-alpha-1-instruct

Tyness BOE 2.6B Instruct — Pre-Alpha 1 (metadata block + adversarial correction pairs, 2.6B)

Fourth pre-alpha from Tyness Train, an agent-guided fine-tuning harness for small LLMs on low-VRAM GPUs — and the first at 2.6B. This model is RAG-style grounded (answers only from provided norm excerpts) with a structured METADATOS block in the system prompt (official identifiers, dates, rank, department, article-title index) and adversarial correction pairs (user asserts a false premise → the model corrects it with the exact citation). It is our best model: FINAL 0.480 on the definitive paired evaluation, up from 0.446 at 1.2B with the identical dataset and recipe — a controlled capacity experiment.

⚠️ Status: PRE-ALPHA. Grounded answering works well (0.625 focused / 0.700 full-precept; deepseek-v4-flash scores 0.989/0.986 on the same items), but sycophancy resistance and parametric knowledge remain weak. Always verify against boe.es.

Model description

Base model	LiquidAI/LFM2.5-2.6B (2.69B params, hybrid: 22 conv + 8 GQA layers)
Task	Context-grounded Spanish law Q&A with exact official citations (BOE-A-AAAA-NNNNN)
Method	QLoRA SFT (TRL, NF4, fp32 compute) — focused relevance-ORDERED contexts + METADATOS system-prompt block + 149 adversarial correction pairs + empty-<think> targets (see Training)
Hardware	NVIDIA GTX 1060 6GB (Pascal, sm_61) — 2016 consumer GPU
Training	5 epochs · seq 2048 · 410 steps · ~13.5h · final loss 0.704 · token accuracy 85.5%
Author	jasonjimnz

Training data

Dataset boe-focused-adv-v1 (1,297 entries), identical to our best 1.2B model (run19) — this release is a controlled capacity experiment: same data, same hyperparameters, only the base model doubles.

• 1,148 grounded entries over 10 BOE articles (both state-level and regional): contexts are relevance-ORDERED paragraphs of the cited sections (never relevance-filtered — filtering starves generic questions; measured at n=200), plus a METADATOS block in the system prompt (identifier, official number, rank, department, dates, status, article-title index).
• 149 adversarial correction pairs (~11.5%): the user asserts a false premise (misquoted article, digit-swapped identifier, invented deadline) that the context disproves; the target answer rejects it and corrects with the exact citation.
• Sources: consolidated Spanish legislation (BOE open data) via legalize-es, teacher-generated ChatML entries.
• Loss on assistant tokens only; evaluation items are never in any training split.

Training configuration

Parameter	Value
LoRA r / alpha / dropout	16 / 32 / 0.05
Target modules	q_proj, k_proj, v_proj, o_proj
Learning rate / scheduler	2e-4 / cosine
Epochs	5 (410 optimizer steps)
Effective batch	1 × 16
Sequence length	2048 (fp32 VRAM headroom on 6GB)
Quantization	NF4 4-bit, fp32 compute (Pascal: never bf16/fp16)
Optimizer	paged_adamw_32bit
prepend_empty_think	enabled — LFM2.5-2.6B's chat template forces a <think> tag at generation start; training targets were prefixed with an empty <think>\n\n</think>\n\n block so the trained token stream matches inference

Evaluation results

1. EVAL-SPEC v1.0.0 bank (es-eval-v1-pilot10, n=102, three context modes)

Questions generated by a teacher LLM directly from the norm text (never from the dataset), deterministic quality gates, same items for every system. Judge: glm5.3-flash.

System	closed-book	focused	full precept
LFM2.5-1.2B-Instruct (base, un-finetuned)	0.238	0.493	0.487
run19 — our best 1.2B (same dataset/recipe)	0.151	0.495	0.565
run24 — this model (2.6B)	0.179	0.625	0.700
deepseek-v4-flash (measured ceiling reference)	0.466	0.989	0.986

Per-archetype (focused mode), run19 → run24: applied cases 0.30 → 0.60, multi-precept synthesis 0.10 → 0.50, identifier traps 0.42 → 0.66, negatives 0.49 → 0.67, content 0.52 → 0.64. Only regression: temporal questions (0.86 → 0.52, under investigation — possible <think> interference with date arithmetic).

2. Historical paired bank (gen-eval-100, n=200 per set, uncontaminated)

Set	run19 (1.2B)	run24 (2.6B)
native (focused contexts)	0.452	0.484
hard (whole sections)	0.440	0.476
FINAL (average)	0.446 ± 0.352	0.480 ± 0.342

Positive paired deltas on both sets (+3.1 / +3.5 points) and on all five independent measurements. The controlled design (byte-identical dataset and recipe) attributes the gain to model capacity: the 1.2B's weakest archetypes — applied reasoning and cross-precept synthesis — were capacity-bound.

Score progression across the project (definitive FINAL): 0.005 → 0.035 → 0.267 → 0.3475 → 0.415 → 0.446 → 0.480 (96× over naive SFT).

Full details: eval-report.md.

What works: grounded reading of provided precepts, applied-case reasoning (doubled), multi-precept synthesis (5×), identifier-trap rejection, legal register. What fails: sycophancy resistance under adversarial pressure (improved to 0.66 but far from the 1.00 ceiling), parametric knowledge without context (0.179 — retrieval is still required), temporal questions regressed (0.86 → 0.52 at 1.2B → 2.6B).

How to run

llama.cpp (all quantizations included: F16 / Q8_0 / Q4_K_M)

llama-server -m tyness-boe-2.6b-pre-alpha-1-instruct-q4_k_m.gguf -ngl 99 -c 16384 --port 8080 --jinja
curl http://127.0.0.1:8080/v1/chat/completions -d '{
  "messages": [
    {"role":"system","content":"METADATOS: identificador BOE-A-2019-18746, Orden HAC/1270/2019, ...\\n\\nEres un jurista experto en el BOE. Responde ÚNICAMENTE con el CONTEXTO. Si un dato no consta: No consta en el contexto proporcionado. Si el usuario afirma algo incorrecto, corrígelo citando el precepto exacto."},
    {"role":"user","content":"CONTEXTO:\n<paste relevant norm paragraphs here>\n\nPREGUNTA: ¿…?"}
  ],
  "temperature": 0
}'

Transformers (note: thinking-capable base)

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("jasonjimnz/tyness-boe-2.6b-pre-alpha-1-instruct", dtype="auto")
tok = AutoTokenizer.from_pretrained("jasonjimnz/tyness-boe-2.6b-pre-alpha-1-instruct")
# This model was SFT'd with empty <think> blocks: it opens and immediately closes the
# thinking section, then answers directly from the provided context.
messages = [{"role": "user", "content": "CONTEXTO:\n...\n\nPREGUNTA: ¿…?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

Limitations

• Specialized in the 10-norm trained scope; not a general legal advisor.
• Sycophancy: still far from the ceiling on adversarial premises — verify corrections against boe.es.
• Temporal/date questions regressed vs the 1.2B (0.86 → 0.52): double-check dates.
• Parametric knowledge is weak by design (closed-book 0.179): always ground it with CONTEXTO + METADATOS.

Generated by Tyness Train on 2026-10-16. Base model © LiquidAI, LFM Open License v1.0.