--- license: other license_name: lfm-open-license-v1.0 license_link: https://huggingface.co/LiquidAI/LFM2.5-2.6B/blob/main/LICENSE base_model: - LiquidAI/LFM2.5-2.6B tags: - boe - spanish-law - legal - lfm2 - lfm2.5 - lora - qlora - sft - rag - grounded-generation - metadata-block - adversarial-correction - teacher-student - tyness-train - pre-alpha library_name: peft datasets: - boe-focused-adv-v4 language: - es pipeline_tag: question-answering --- # Tyness BOE 2.6B Instruct — Pre-Alpha 2 (teacher–student + adversarial replay) **Fifth pre-alpha from [Tyness Train](https://github.com/jasonjimnz/model_trainer)** — and our **best model: 0.704 focused / 0.718 full-precept / 0.181 closed-book** on the EVAL-SPEC v1.0.0 bank, up from 0.640/0.724/0.179 (pre-alpha-1). Built by combining two data signals that had only been tested separately: **teacher–student multi-turn dialogues** (which improved ficha, content, synthesis and closed-book but eroded defenses in isolation) and **fresh adversarial replay** (which restores them). Both together win. > ⚠️ **Status: PRE-ALPHA.** Grounded reading is strong (0.70–0.72 with context; deepseek-v4-flash > scores 0.99 on the same items), but parametric knowledge is weak by design (0.181 closed-book) > and sycophancy resistance, while improved, is not solved. Always verify against boe.es. ## What improved vs pre-alpha-1 (run24, same 102-item comparison) | Metric | pre-alpha-1 | **pre-alpha-2 (this)** | |---|---|---| | focused (mean) | 0.640 | **0.704** (+6.4) | | content (focused) | 0.64 | **0.70** | | multi-precept synthesis (focused) | 0.00 | **0.50** | | identifier-trap rejection (focused) | 0.86 → *0.54 with TS-only* | **0.67** (replay restores) | | negative "no-consta" (focused) | 0.63 → *0.47 with TS-only* | **0.72** (restored above baseline) | | temporal (n=10 items) | 0.70 | **0.81** | | full-precept mode | 0.724 | 0.718 (flat, within noise) | | closed-book | 0.179 | 0.181 (flat) | Zero empty answers (a generation bug in pre-alpha-1 — answers trapped inside the `` block on hard items — was diagnosed and fixed in the evaluation harness). ## Model description | | | |---|---| | Base model | [LiquidAI/LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B) (2.69B params, hybrid: 22 conv + 8 GQA layers) | | Task | **Context-grounded** Spanish law Q&A with exact official citations (BOE-A-AAAA-NNNNN) | | Method | QLoRA SFT (TRL, NF4, fp32 compute) — focused relevance-ORDERED contexts + METADATOS system-prompt block + adversarial correction pairs (original + fresh quote-to-cite) + teacher–student multi-turn dialogues + empty-`` targets | | Hardware | NVIDIA GTX 1060 6GB (Pascal, sm_61) — 2016 consumer GPU | | Training | 5 epochs · seq 2048 · 440 steps · ~14.5h · final loss 0.752 · token accuracy 84.9% | | Author | [jasonjimnz](https://huggingface.co/jasonjimnz) | ## Training data and methodology Dataset `boe-focused-adv-v4` (1,397 entries) — three signals, one mix: 1. **Base recipe (1,297 entries, byte-identical to our best 1.2B release)**: grounded entries over 10 BOE articles with relevance-ORDERED contexts (never relevance-filtered — measured at n=200), a **METADATOS block** in the system prompt (identifier, official number, rank, department, dates, status, article-title index), and **149 adversarial correction pairs** (~11.5%): the user asserts a false premise, the target answer corrects it with the exact citation. 2. **Teacher–student multi-turn dialogues (30 dialogues, 201 turns)**: progressive comprehension→application conversations per norm, including **11 temporal dialogues** with step-by-step date computation (entry into force, transitional provisions). Every teacher turn gate-checked for literal grounding in the norm. 3. **Fresh adversarial replay (70 quote-to-cite pairs)**: new adversarial entries whose targets must include a literal «quote» of the correct precept, each quote verified against the norm text. This replay is what restores identifier-trap and refusal behavior when teacher–student data is added — the two signals were only tested separately before (pre-alpha-1 without TS; an intermediate run with TS-only lost 0.32 on traps and 0.16 on refusals). **Empty-`` targets**: the LFM2.5-2.6B chat template forces a `` tag at generation start; targets were prefixed with an empty `\n\n\n\n` block so the trained token stream matches inference (without this, training/inference mismatch silently corrupts the model). Loss on **assistant tokens only**; evaluation items are never in any training split. Sources: consolidated Spanish legislation (BOE open data) via [legalize-es](https://github.com/legalize-dev/legalize-es), teacher-generated ChatML entries. ## Training configuration | Parameter | Value | |---|---| | LoRA r / alpha / dropout | 16 / 32 / 0.05 | | Target modules | q_proj, k_proj, v_proj, o_proj | | Learning rate / scheduler | 2e-4 / cosine | | Epochs | 5 (440 optimizer steps) | | Effective batch | 1 × 16 | | Sequence length | 2048 | | Quantization | NF4 4-bit, **fp32 compute** (Pascal: never bf16/fp16) | | Optimizer | paged_adamw_32bit | ## Evaluation results Bank: `es-eval-v1.1-pilot10` (EVAL-SPEC v1.0.0 + expanded temporal archetype, 107 items × 3 context modes). Questions generated by a teacher LLM from the norm text only, deterministic quality gates, same items for every system. Judge: glm5.3-flash. | Mode | pre-alpha-1 (run24) | **pre-alpha-2 (this)** | deepseek-v4-flash (ceiling ref.) | |---|---|---|---| | closed-book (no context) | 0.179 | 0.181 | 0.466 | | focused (production recipe) | 0.640 | **0.704** | 0.989 | | full precept | 0.724 | **0.718** | 0.986 | Per-archetype (focused): ficha 0.77 · content 0.70 · applied case 0.53 · multi-precept 0.50 · negative 0.72 · identifier-trap 0.67 · temporal 0.81. Closed-book stays at 0.18 — **parametric knowledge is weak by design**: on this rig (1.2B/2.6B + QLoRA) knowledge injection empirically fails and erodes other skills, so the architecture is hybrid by decision: this model is the *reader*; the corpus lives in retrieval. Score progression across the project (definitive FINAL): 0.005 → 0.035 → 0.267 → 0.3475 → 0.415 → 0.446 (1.2B best) → 0.480 (2.6B) → focused 0.704 (this, new bank). Full details: [eval-report.md](eval-report.md). **What works**: grounded reading of provided precepts, applied-case reasoning, multi-precept synthesis (0.50), identifier-trap rejection with quote-based correction, refusal calibration (restored), legal register. **What fails**: sycophancy under strong adversarial pressure (0.67, ceiling 1.00), parametric knowledge (closed-book 0.181), temporal/date arithmetic partially (0.81 with context). ## How to run ### llama.cpp (all quantizations included: F16 / Q8_0 / Q4_K_M) ```bash llama-server -m tyness-boe-2.6b-pre-alpha-2-instruct-q4_k_m.gguf -ngl 99 -c 16384 --port 8080 --jinja curl http://127.0.0.1:8080/v1/chat/completions -d '{ "messages": [ {"role":"system","content":"METADATOS: identificador BOE-A-2019-18746, Orden HAC/1270/2019, ...\\n\\nEres un jurista experto en el BOE. Responde ÚNICAMENTE con el CONTEXTO. Si un dato no consta: No consta en el contexto proporcionado. Si el usuario afirma algo incorrecto, corrígelo citando el precepto exacto."}, {"role":"user","content":"CONTEXTO:\n\n\nPREGUNTA: ¿…?"} ], "temperature": 0 }' ``` ### Transformers (note: thinking-capable base) ```python from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained("jasonjimnz/tyness-boe-2.6b-pre-alpha-2-instruct", dtype="auto") tok = AutoTokenizer.from_pretrained("jasonjimnz/tyness-boe-2.6b-pre-alpha-2-instruct") # This model was SFT'd with empty blocks: it opens and immediately closes the # thinking section, then answers directly from the provided context. messages = [{"role": "user", "content": "CONTEXTO:\n...\n\nPREGUNTA: ¿…?"}] inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt") ``` ## Limitations - Specialized in the 10-norm trained scope; not a general legal advisor. - **Sycophancy**: improved (0.67) but not solved — verify corrections against boe.es. - **Parametric knowledge is weak by design** (closed-book 0.181): always ground it with CONTEXTO + METADATOS. - Temporal/date questions improved (0.81) but date arithmetic can still err. *Generated by Tyness Train on 2026-10-20. Base model © LiquidAI, LFM Open License v1.0.*