jasonjimnz/tyness-boe-1.2b-pre-alpha

Tyness BOE 1.2B — Pre-Alpha (Tyness Train v0.4.x)

The first model trained end-to-end with Tyness Train, an agent-guided fine-tuning harness for small LLMs on low-VRAM GPUs. This is a pre-alpha release: it proves the full pipeline (dataset → QLoRA SFT on a GTX 1060 6GB → merge → GGUF → evaluation) and documents honest findings, including negative ones.

⚠️ Status: PRE-ALPHA. The model learned the format of Spanish legal Q&A but hallucinates specific citations. Do not use it for legal information. See Evaluation & findings.

Model description

Base model	LiquidAI/LFM2.5-1.2B-Base (1.2B, hybrid conv+attention, 32K context)
Task	Spanish law Q&A with official BOE citations (BOE-A-AAAA-NNNNN)
Method	QLoRA SFT (TRL SFTTrainer, 4-bit NF4 + double quantization, fp32 compute)
Hardware	NVIDIA GTX 1060 6GB (Pascal, sm_61) — a consumer GPU from 2016
Training time	2h 54m, 216 optimizer steps, 1.37M tokens
Author	jasonjimnz

Why this model exists

Tyness Train is a harness for fine-tuning 1B–3B models on 6GB VRAM GPUs. This first release validates the whole pipeline on a deliberately hard, knowledge-dense domain:

1. Dataset: ChatML entries from Spanish consolidated legislation (BOE).
2. Training: QLoRA on Pascal, where bf16/fp16 compute is not usable — fp32 only.
3. Export: merged safetensors (bf16) + GGUF f16/Q8_0/Q4_K_M.
4. Evaluation: LLM-judged (glm5.3-flash) against held-out questions with expected answers.

Training data

Source: consolidated Spanish legislation published by the Agencia Estatal Boletín Oficial del Estado (BOE), reused per its open-data conditions, via legalize-es (markdown per norm, YAML frontmatter).

Entries: teacher-generated ChatML conversations (question / grounded answer), 1,148 training samples + 20 held-out eval items, filtered to 10 BOE articles:

BOE id	Norm	Train	Eval
BOE-A-2026-14659	Ley 8/2026, de Ordenación del Transporte Marítimo de Cantabria	118	2
BOE-A-2020-7440	Decreto-ley 9/2020 (ayudas, Capítulo II)	142	2
BOE-A-2021-10163	Orden HAC/612/2021, modelo 179	118	2
BOE-A-2019-18746	Orden HAC/1270/2019	118	2
BOE-A-2016-9637	Orden INT/1676/2016	118	2
BOE-A-1995-19198	Acuerdo sobre transporte internacional por carretera (aplicación provisional)	119	2
BOE-A-1996-6400	Real Decreto 330/1996, traspaso de funciones	60	2
BOE-A-1975-9248	Ley 17/1975, Organismo autónomo «Registro de Matrícula…»	118	2
BOE-A-2014-10908	Orden HAP/1949/2014, Punto General de Entrada	119	2
BOE-A-1993-10089	Real Decreto 562/1993	118	2

Methodology:

• Seeded, diversity-aware article selection (≥40 entries, ≥4 question categories, mix of scopes).
• 2 held-out entries per article for evaluation (never seen in training).
• Each sample: system prompt (rigorous BOE jurist persona) + user question + assistant answer.
• Loss computed on assistant tokens only (completion_only_loss).
• Provenance metadata kept strictly separated — never part of training.

Training configuration

Parameter	Value
LoRA r / alpha / dropout	16 / 32 / 0.05
Target modules	q_proj, k_proj, v_proj, o_proj
Learning rate / scheduler	2e-4 / cosine, warmup 6 steps
Epochs	3
Effective batch	2 × 8 = 16
Sequence length	512
Quantization	NF4 4-bit + double quant, fp32 compute (Pascal has no usable bf16/fp16 ALUs)
Optimizer	paged_adamw_32bit
Loss curve	2.97 → 1.54 (ep.1) → 1.40 (ep.2) → 1.34 (ep.3), token accuracy 71.2%

Evaluation & findings

Judged by an LLM referee (glm5.3-flash) on 20 held-out questions, verdict per item: correct / partial / incorrect, against the expected answer and the key reason.

Result: 0/20 correct, mean score 0.005. Report: pre-alpha baseline.

Findings (this is the value of the pre-alpha):

1. Format learned, facts not. Answers are fluent, structured legal Spanish with plausible citation patterns — but the model confabulates identifiers ("Ley 6/2015", "Reglamento (UE) 1169/2009" for a subsidies question). A 1.2B model trained for 3 epochs does not reliably memorize norm-specific facts from ~115 samples/article.
2. Generation truncation (256 tokens) cut some answers mid-argument; part of the failures may be "partial" rather than "incorrect". Fixed for the next iteration (longer generation budget).
3. The judge pipeline works: strict verdicts, useful per-item reasons, reproducible greedy decoding. The eval infrastructure is trustworthy even though this checkpoint is not.
4. The hardware pipeline works: 3 epochs on 6GB VRAM without OOM, ~1.5GB peak, 100% GPU utilization, bf16-free fp32 compute on Pascal.

Next iteration (in progress): 6 epochs, LoRA r=32, 16K context ceiling, longer eval generations — targeting fact memorization, not just format.

How to run

llama.cpp (recommended — all quantizations included in this repo)

# 4-bit (fits in ~700MB of RAM/VRAM)
llama-cli -hf jasonjimnz/tyness-boe-1.2b-pre-alpha:Q4_K_M -p "¿Cuál es el plazo para…?" --jinja

# or a local file + OpenAI-compatible server
llama-server -m tyness-boe-1.2b-pre-alpha-q4_k_m.gguf -ngl 99 -c 4096 --port 8080 --jinja
curl http://127.0.0.1:8080/v1/chat/completions -d '{
  "messages": [{"role":"system","content":"Eres un jurista experto en el BOE. Responde con rigor y cita identificadores oficiales. Si un dato no consta, decláralo."},
               {"role":"user","content":"¿Qué es el modelo 179?"}],
  "temperature": 0
}'

transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("jasonjimnz/tyness-boe-1.2b-pre-alpha", dtype="auto")
tok = AutoTokenizer.from_pretrained("jasonjimnz/tyness-boe-1.2b-pre-alpha")
messages = [
    {"role": "system", "content": "Eres un jurista y documentalista experto en el ordenamiento jurídico español y en el BOE. Responde con rigor técnico y cita identificadores oficiales (BOE-A-AAAA-NNNNN). Si un dato no consta, decláralo explícitamente."},
    {"role": "user", "content": "¿Cuál es el identificador BOE del Decreto-ley 9/2020?"},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

Files

File	Format	Size	Use
model.safetensors	safetensors, bf16	2.2GB	transformers / further fine-tuning
tyness-boe-1.2b-pre-alpha-f16.gguf	GGUF F16	2.2GB	llama.cpp (full precision)
tyness-boe-1.2b-pre-alpha-q8_0.gguf	GGUF Q8_0	1.2GB	llama.cpp (8-bit)
tyness-boe-1.2b-pre-alpha-q4_k_m.gguf	GGUF Q4_K_M	698MB	llama.cpp (4-bit, recommended)
eval-report.md	—	—	pre-alpha evaluation findings

Limitations

• Hallucinates norm-specific facts and citations — verified in evaluation. Never use for real legal guidance.
• Trained on 10 norms only: outside that scope it answers in-style but without grounding.
• Base (non-instruct) LFM2.5 lineage: better at completion-style tasks than chat.
• Spanish only.

Attribution

• Base model: LiquidAI (LFM Open License v1.0).
• Data: Agencia Estatal Boletín Oficial del Estado, reused per its open-data conditions (Resolución de 27 de junio de 2024), via legalize-es.

Trained and published with Tyness Train v0.4.7 — Pre-Alpha, October 2026.