Label-free RLVR runs on Qwen2.5-Math-1.5B: checkpoints and results
Raw verl checkpoints, logs and evaluation results from two 300-step runs trained with labelfree_runs_kit_2026-11-07,
every setting as shipped, starting from Qwen/Qwen2.5-Math-1.5B at
revision 4a83ca6e4526a4f2da3aa259ec36c259f66b2ab2. Both runs finished on 2026-11-08 (UTC).
| Folder | Run | Code |
|---|---|---|
qwen_band_entropy_grad_s0/ (#6) |
Majority-vote GRPO without labels, plus a rank-1 off-band penalty around the entropy-gradient direction (Ξ» = 1) | the kit's repo/, unmodified |
qwen_intuitor_s0/ (#7) |
INTUITOR: self-certainty reward (Zhao et al., 2505.19590) | the kit's repo/ plus two fixes, in qwen_intuitor_s0/patches/ |
Recipe: 64 prompts Γ 16 samples per step, learning rate 5e-7 constant, no KL, temperature 1, 300 steps, a checkpoint every 50 steps, FSDP2 on one GPU.
Results at step 300
Scores in percent from the kit's frozen evaluator (few-GT bundle v5, called by eval_frozen.py). Each method ran with
one seed, so gaps of about 2 points or less may be noise. Ξ is #7 β #6, computed from the unrounded scores.
| Benchmark | #6 band_entropy_grad | #7 INTUITOR | Ξ |
|---|---|---|---|
| MATH-500 | 59.7 | 64.9 | +5.2 |
| AMC23 | 44.5 | 42.0 | β2.4 |
| AIME24 | 6.8 | 7.5 | +0.7 |
| AIME25 | 5.4 | 5.5 | +0.1 |
| Minerva Math | 12.8 | 15.4 | +2.6 |
| OlympiadBench | 28.3 | 30.4 | +2.2 |
Math average (id_avg) |
26.2 | 27.6 | +1.4 |
| ARC-Challenge | 45.9 | 51.0 | +5.1 |
| GPQA-Diamond | 23.5 | 18.4 | β5.1 |
| MMLU-Pro | 22.8 | 15.0 | β7.8 |
Out-of-domain average (ood_avg) |
30.7 | 28.1 | β2.6 |
| MMLU | 37.1 | 40.3 | +3.2 |
| HellaSwag | 29.6 | 34.2 | +4.6 |
| CommonsenseQA | 41.2 | 41.8 | +0.6 |
| TruthfulQA | 24.8 | 21.0 | β3.8 |
| IFEval | 19.2 | 10.6 | β8.6 |
| HumanEval | 9.1 | 4.3 | β4.7 |
id_avg is the mean of the six math benchmarks and ood_avg the mean of ARC-Challenge, GPQA-Diamond and MMLU-Pro.
Unrounded values are in results/final_eval_step300.csv and each run's eval_step300/results.csv.
How each benchmark was sampled (responses up to 3,072 tokens; top-p 0.95 when sampling):
| Suite | Benchmarks | Questions | Samples per question | Temperature |
|---|---|---|---|---|
math500_3k |
MATH-500 | 500 | 8 | 0.6 |
aimeamc32 |
AIME24, AIME25, AMC23 | 100 | 32 | 0.6 |
minerva8 |
Minerva Math | 272 | 8 | 0.6 |
olympiad8 |
OlympiadBench | 672 | 8 | 0.6 |
mmlu8 |
MMLU | 500 | 8 | 0.6 |
he8 |
HumanEval | 164 | 8 | 0.6 |
genmc |
HellaSwag, TruthfulQA, CommonsenseQA, IFEval | 2,000 | 1 | 0 (greedy) |
arc4 |
ARC-Challenge | 1,172 | 4 | 0.6 |
gpqa4 |
GPQA-Diamond | 197 | 4 | 0.6 |
mmlupro1 |
MMLU-Pro | 992 | 1 | 0 (greedy) |
Learned direction
lead_u_cosine_vs_true.json: for each of the 196 target weight matrices, the cosine between the leading left singular
vector of the update Wβββ β Wβ and the reference direction from the kit's ground-truth run
(bases/qwen_true_u1_reference.pt).
| #6 | #7 | |
|---|---|---|
| Mean |cos| | 0.334 | 0.473 |
| Mean weighted by βWβββ β Wββ_F | 0.353 | 0.451 |
| Max | 0.832 | 0.920 |
| Random-direction baseline | 0.025 | 0.025 |
| Sum of βWβββ β Wββ_F over the 196 matrices | 3.47 | 16.66 |
Also in results/lead_u_cosine_step300.csv.
MATH-500 during training
verl's validation every 25 steps: greedy decoding, one response per problem, up to 3,072 tokens. This is not the protocol of the final MATH-500 score above (8 samples at temperature 0.6). Step 0 is the base model, and 41% of its responses ran to the length limit.
| Step | #6 accuracy | #7 accuracy | #6 responses β₯ 3,000 tokens | #7 responses β₯ 3,000 tokens |
|---|---|---|---|---|
| 0 | 30.0 | 30.6 | 206 | 203 |
| 25 | 47.2 | 62.2 | 92 | 10 |
| 50 | 58.4 | 62.2 | 37 | 10 |
| 75 | 62.4 | 60.4 | 22 | 15 |
| 100 | 60.4 | 61.8 | 13 | 18 |
| 125 | 60.8 | 62.8 | 16 | 10 |
| 150 | 61.0 | 63.2 | 22 | 6 |
| 175 | 62.6 | 63.8 | 14 | 10 |
| 200 | 62.4 | 65.4 | 11 | 10 |
| 225 | 62.4 | 65.6 | 14 | 9 |
| 250 | 63.6 | 65.0 | 15 | 8 |
| 275 | 63.0 | 67.4 | 13 | 7 |
| 300 | 62.2 | 65.8 | 13 | 11 |
#7 is ahead at every step from 100 on. Paired on the same 500 problems (exact McNemar test), the gap between the runs is significant at p < 0.05 only at steps 25, 50 and 275. From step 75 to 300, #6 solved 23 problems it had missed and missed 24 it had solved (p = 1.0); #7 solved 65 new and lost 38 (p = 0.01).
Per-problem outputs: <run>/validation/<step>.jsonl. Summary: results/validation_math500_by_step.csv.
Training
results/training_metrics_<run>.csv has one row per step: verl's metrics from train.log, plus the label-free
estimator's own log line under estimator/. mode_share is the fraction of a batch's parseable answers that equal the
batch's most common answer (1.0 means one answer everywhere). The kit stops a majority-vote run once it stays above 0.5
for 5 consecutive steps; INTUITOR has no such stop.
| #6 | #7 | |
|---|---|---|
mode_share, step 1 β 300 (max) |
0.031 β 0.027 (0.062 at step 198) | 0.038 β 0.032 (0.068 at step 266) |
| Token entropy, step 1 β 300 | 0.82 β 0.20 | 0.80 β 0.04 |
| Mean response length, step 1 β 300 | 1,242 β 898 tokens | 1,252 β 866 tokens |
Loading the step-300 models
Each run's eval_step300/export/ holds its step-300 model in transformers format: the exact weights the evaluator
ran, in float32, with lm_head tied to the input embeddings as in the base model. export_manifest.json records the
SHA-256 of the source checkpoint and confirms that every tensor round-trips exactly.
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "JaeWooShin/labelfree-runs-2026-11-07"
sub = "qwen_intuitor_s0/eval_step300/export" # or "qwen_band_entropy_grad_s0/eval_step300/export"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=sub)
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=sub)
For vLLM, download the folder first and pass its local path:
from huggingface_hub import snapshot_download
path = snapshot_download(repo, allow_patterns=f"{sub}/*") + "/" + sub
Transformers 4.57 may warn that the tokenizer has "an incorrect regex pattern" and suggest fix_mistral_regex=True.
The warning is aimed at some Mistral tokenizers and does not apply here: tokenization matches the base model's.
The checkpoints at steps 50 to 250 are in verl's format only. To convert one, download its actor/ folder and run
verl's model merger from the kit's repo/:
python -m verl.model_merger merge --backend fsdp --local_dir <global_step_N>/actor --target_dir <output dir>.
Layout
Each run folder mirrors that run's output directory:
checkpoints/global_step_N/(N = 50, 100, β¦, 300), exactly as verl saved it:actor/model_world_size_1_rank_0.pt: model weights (a single-rank FSDP2 state dict, nottransformersformat)actor/optim_world_size_1_rank_0.pt: AdamW stateactor/extra_state_world_size_1_rank_0.pt,actor/fsdp_config.json,actor/huggingface/(config and tokenizer)data.pt: dataloader state
checkpoints/latest_checkpointed_iteration.txt: verl's record of the latest step (300), which its auto-resume reads.train.logandrun.json: the training log and the kit's run record.run_one.out: what the kit'srun_one.shprinted, including the training and evaluation exit codes.hydra/.hydra/: the resolved training configuration (config.yaml), the command-line overrides (overrides.yaml) and Hydra's own settings (hydra.yaml).validation/<step>.jsonl: the MATH-500 validation outputs, 500 records per step (prompt, response, reference answer, score, accuracy, whether the answer was boxed).eval_step300/: the step-300 evaluation. It hasresults.csvand, for each suite, every generation (<suite>.generations.jsonl), the per-question scores (<suite>.scores.json) and the generation statistics (<suite>.generation.json), plusquestions.json,scores.jsonand the logs.export/is the step-300 model intransformersformat (see "Loading the step-300 models").eval.log,lead_u.logandlead_u_cosine_vs_true.json: the evaluator's log and the learned-direction measurement.eval_fix/: how the evaluation was run (see below).- #6 only:
basis/entropy_grad_qwen.pt, a copy of the kit'sbases/entropy_grad_qwen.pt(the direction the band penalty is built around), and the logs and status files of the failed first evaluation attempt. - #7 only:
patches/.
results/ at the top level holds the tables above as CSV files.
preflight/ at the top level holds what ran before launch: the dry-run configs, and the 2-step smoke tests with
their GPU-memory samples. For INTUITOR they show the shipped code failing (qwen_intuitor_s0_smoke.train.log, the
AssertionError), the run with only the first fix failing (qwen_intuitor_s0_smoke.patch1.train.log, the
ValueError), and the run with both fixes completing (qwen_intuitor_s0_smoke.patch1+2.train.log).
Every checkpoint and evaluation file was hash-verified on the Hub against the local copy.
Deviations from the kit
No training setting differs from the kit.
INTUITOR: two fixes
As shipped, the kit's INTUITOR configuration stops at its first training step, so this run uses the kit's code plus two small fixes.
verl/workers/engine/fsdp/transformer_impl.py: withuse_remove_padding=false, the per-token mean log-probability that self-certainty needs kept its padding positions, and the trainer's length check failed (AssertionErrorinno_padding_2_padding). The fix trims it to each sequence's true length, as verl already does for the token entropy.verl/trainer/ppo/ray_trainer.py: thegrpo_selfcertaintyestimator is registered only whenrecipe/principal_subspaces/selfsup_rewards.pyis imported, which the kit's configuration never does (ValueError: Unknown advantage estimator). The fix imports it when one of its estimators is requested.
Evaluation: the kit deletes its own temporary directory
run_one.sh exports TMPDIR=${RAYTMP}_tmp, deletes that directory right after training (line 135), and only then calls
eval_one.sh. The evaluator's vLLM compiles FlashInfer's sampling kernels with nvcc the first time it runs, and nvcc
cannot write into the deleted TMPDIR (nvcc fatal: Could not open output file). Training is not affected.
- #6: the first evaluation failed this way after 78 seconds, before generating anything. Its files are
eval_step300.failed_20261108T212727Z/andeval.failed_20261108T212727Z.log; theevaluation exit 1line inrun_one.outrefers to that log. The evaluation step alone was then re-run the wayeval_one.shruns it, inrun_one.sh's environment, with aTMPDIRthat exists (eval_fix/rerun_eval6.sh). The checkpoint, the evaluator and its settings are unchanged.lead_u_cosine_vs_true.jsoncomes from the first pass; it reads only the checkpoint. - #7: the directory was recreated the moment
run_one.shdeleted it (eval_fix/guard_tmpdir7.shand its log), so its evaluation ran insiderun_one.shas usual.
For future runs, moving rm -rf "${RAYTMP}" "${RAYTMP}_tmp" in run_one.sh below the eval_one.sh call fixes this.
Environment
The few-GT bundle v5 lock (PyTorch 2.8.0+cu128, vLLM 0.11.0, Transformers 4.57.6) plus flash-attn 2.8.1, which the source environment also had: verl's padding utilities import it, and its Triton cross-entropy computes the log-probs. Hardware: one NVIDIA RTX PRO 5000 Blackwell (48 GB) per run.