Qwen3.5-4B-Soyuz-GRPO-v3 (merged bf16)

GRPO (Group Relative Policy Optimization) fine-tune of Soyuz-4B, where the reward is a separate Claude Code instance grading each answer against an explicit per-task criteria checklist. This card documents how the GRPO actually affected the model — including a negative transfer result, reported honestly.

Setup

Policy: Soyuz-4B (Qwen3_5ForCausalLM) + LoRA r=32, on-policy REINFORCE with a group baseline (G=6 rollouts/task) + KL to the frozen base (LoRA-disabled reference). Custom bare-torch loop.
Reward: claude -p (headless Claude Code) scores each rollout passed/total against the task's explicit criteria → reward in [0,1]. Criteria-checklist grading has much lower variance than free-form quality scoring (this was the single biggest fix for a stable signal).
Per-turn recovery shaping: each task is a 2-turn rollout (answer → self-revise). Generation and judging are interleaved so the turn-2 critique is grounded in the judge's actual reason and is harsh when the answer is bad ("scored N/10, problems: …, scrap it and rewrite"). Turn-2 reward = score + 0.5·(score − score_prev) → rewards climbing out of a bad answer, penalises regression.
Data: 44 tasks × 2 epochs — algorithms/data-structures, library use (pandas, numpy, requests, asyncio, SQLAlchemy, pytest, click), real-code fixes (SQL-injection, async race, FastAPI pagination, vectorize, retry bug, refactor), and creative (micro-fiction, ELI5, limerick, haiku, dialogue, tone-rewrite).

How GRPO affected the model (all numbers file-backed)

On the trained distribution — clear gain:

metric	epoch 1	epoch 2	Δ
reward (criteria)	0.700	0.757	+0.057 (27/44 tasks up)
turn-0 score	0.652	0.723	+0.071
recovery (climb from a bad turn-0)	0.301	0.273	−0.028

The recovery skill internalised into the first turn. Recovery fell not because the model got worse at fixing — but because turn-0 got better, so there were fewer bad starts left to recover from. The model learned to do it right the first time (e.g. product-description turn-0 0.23→0.60, cover-letter 0.3→0.93, ELI5 0.0→0.73). Biggest gains were on the hard tail (semver-compare +0.35, weighted-choice +0.27, streaming-median +0.22, SQL-injection-fix +0.18).

Negative transfer to agentic terminal tasks. Tested on the terminal-bench-2 tasks the base model partially solved, GRPO-v3 regressed: tasks-with-any-pass dropped 11/11 → 4/11 (e.g. hf-model-inference 4/5→0/5, openssl 3/5→0/5, sqlite 3/5→0/5). General single-response coding/creative GRPO does not transfer to — and hurts — multi-step shell-agent behaviour. To improve agentic terminal performance you must run GRPO on the agentic rollouts themselves (verifier reward), not on single-shot coding questions.

App generation. Works well as a single-shot generator (one prompt → full index.html), which is the model's strength; it fails inside a heavy agent loop (drowns in planning). Suppress its <think> ("output only the HTML, do not plan") or it spends its token budget reasoning and never emits the code.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
m = AutoModelForCausalLM.from_pretrained("AlexWortega/qwen35-4b-soyuz-grpo-v3", dtype=torch.bfloat16, device_map="cuda")
tok = AutoTokenizer.from_pretrained("AlexWortega/qwen35-4b-soyuz-grpo-v3")

Best for single-response coding / writing / rewriting. For multi-step shell agents, prefer the base Soyuz-4B.