Label-free RLVR runs on Qwen2.5-Math-1.5B: checkpoints and results

Raw verl checkpoints, logs and evaluation results from two 300-step runs trained with labelfree_runs_kit_2026-11-07, every setting as shipped, starting from Qwen/Qwen2.5-Math-1.5B at revision 4a83ca6e4526a4f2da3aa259ec36c259f66b2ab2. Both runs finished on 2026-11-08 (UTC).

Folder Run Code
qwen_band_entropy_grad_s0/ (#6) Majority-vote GRPO without labels, plus a rank-1 off-band penalty around the entropy-gradient direction (Ξ» = 1) the kit's repo/, unmodified
qwen_intuitor_s0/ (#7) INTUITOR: self-certainty reward (Zhao et al., 2505.19590) the kit's repo/ plus two fixes, in qwen_intuitor_s0/patches/

Recipe: 64 prompts Γ— 16 samples per step, learning rate 5e-7 constant, no KL, temperature 1, 300 steps, a checkpoint every 50 steps, FSDP2 on one GPU.

Results at step 300

Scores in percent from the kit's frozen evaluator (few-GT bundle v5, called by eval_frozen.py). Each method ran with one seed, so gaps of about 2 points or less may be noise. Ξ” is #7 βˆ’ #6, computed from the unrounded scores.

Benchmark #6 band_entropy_grad #7 INTUITOR Ξ”
MATH-500 59.7 64.9 +5.2
AMC23 44.5 42.0 βˆ’2.4
AIME24 6.8 7.5 +0.7
AIME25 5.4 5.5 +0.1
Minerva Math 12.8 15.4 +2.6
OlympiadBench 28.3 30.4 +2.2
Math average (id_avg) 26.2 27.6 +1.4
ARC-Challenge 45.9 51.0 +5.1
GPQA-Diamond 23.5 18.4 βˆ’5.1
MMLU-Pro 22.8 15.0 βˆ’7.8
Out-of-domain average (ood_avg) 30.7 28.1 βˆ’2.6
MMLU 37.1 40.3 +3.2
HellaSwag 29.6 34.2 +4.6
CommonsenseQA 41.2 41.8 +0.6
TruthfulQA 24.8 21.0 βˆ’3.8
IFEval 19.2 10.6 βˆ’8.6
HumanEval 9.1 4.3 βˆ’4.7

id_avg is the mean of the six math benchmarks and ood_avg the mean of ARC-Challenge, GPQA-Diamond and MMLU-Pro. Unrounded values are in results/final_eval_step300.csv and each run's eval_step300/results.csv.

How each benchmark was sampled (responses up to 3,072 tokens; top-p 0.95 when sampling):

Suite Benchmarks Questions Samples per question Temperature
math500_3k MATH-500 500 8 0.6
aimeamc32 AIME24, AIME25, AMC23 100 32 0.6
minerva8 Minerva Math 272 8 0.6
olympiad8 OlympiadBench 672 8 0.6
mmlu8 MMLU 500 8 0.6
he8 HumanEval 164 8 0.6
genmc HellaSwag, TruthfulQA, CommonsenseQA, IFEval 2,000 1 0 (greedy)
arc4 ARC-Challenge 1,172 4 0.6
gpqa4 GPQA-Diamond 197 4 0.6
mmlupro1 MMLU-Pro 992 1 0 (greedy)

Learned direction

lead_u_cosine_vs_true.json: for each of the 196 target weight matrices, the cosine between the leading left singular vector of the update W₃₀₀ βˆ’ Wβ‚€ and the reference direction from the kit's ground-truth run (bases/qwen_true_u1_reference.pt).

#6 #7
Mean |cos| 0.334 0.473
Mean weighted by β€–W₃₀₀ βˆ’ Wβ‚€β€–_F 0.353 0.451
Max 0.832 0.920
Random-direction baseline 0.025 0.025
Sum of β€–W₃₀₀ βˆ’ Wβ‚€β€–_F over the 196 matrices 3.47 16.66

Also in results/lead_u_cosine_step300.csv.

MATH-500 during training

verl's validation every 25 steps: greedy decoding, one response per problem, up to 3,072 tokens. This is not the protocol of the final MATH-500 score above (8 samples at temperature 0.6). Step 0 is the base model, and 41% of its responses ran to the length limit.

Step #6 accuracy #7 accuracy #6 responses β‰₯ 3,000 tokens #7 responses β‰₯ 3,000 tokens
0 30.0 30.6 206 203
25 47.2 62.2 92 10
50 58.4 62.2 37 10
75 62.4 60.4 22 15
100 60.4 61.8 13 18
125 60.8 62.8 16 10
150 61.0 63.2 22 6
175 62.6 63.8 14 10
200 62.4 65.4 11 10
225 62.4 65.6 14 9
250 63.6 65.0 15 8
275 63.0 67.4 13 7
300 62.2 65.8 13 11

#7 is ahead at every step from 100 on. Paired on the same 500 problems (exact McNemar test), the gap between the runs is significant at p < 0.05 only at steps 25, 50 and 275. From step 75 to 300, #6 solved 23 problems it had missed and missed 24 it had solved (p = 1.0); #7 solved 65 new and lost 38 (p = 0.01).

Per-problem outputs: <run>/validation/<step>.jsonl. Summary: results/validation_math500_by_step.csv.

Training

results/training_metrics_<run>.csv has one row per step: verl's metrics from train.log, plus the label-free estimator's own log line under estimator/. mode_share is the fraction of a batch's parseable answers that equal the batch's most common answer (1.0 means one answer everywhere). The kit stops a majority-vote run once it stays above 0.5 for 5 consecutive steps; INTUITOR has no such stop.

#6 #7
mode_share, step 1 β†’ 300 (max) 0.031 β†’ 0.027 (0.062 at step 198) 0.038 β†’ 0.032 (0.068 at step 266)
Token entropy, step 1 β†’ 300 0.82 β†’ 0.20 0.80 β†’ 0.04
Mean response length, step 1 β†’ 300 1,242 β†’ 898 tokens 1,252 β†’ 866 tokens

Loading the step-300 models

Each run's eval_step300/export/ holds its step-300 model in transformers format: the exact weights the evaluator ran, in float32, with lm_head tied to the input embeddings as in the base model. export_manifest.json records the SHA-256 of the source checkpoint and confirms that every tensor round-trips exactly.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "JaeWooShin/labelfree-runs-2026-11-07"
sub = "qwen_intuitor_s0/eval_step300/export"  # or "qwen_band_entropy_grad_s0/eval_step300/export"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=sub)
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=sub)

For vLLM, download the folder first and pass its local path:

from huggingface_hub import snapshot_download

path = snapshot_download(repo, allow_patterns=f"{sub}/*") + "/" + sub

Transformers 4.57 may warn that the tokenizer has "an incorrect regex pattern" and suggest fix_mistral_regex=True. The warning is aimed at some Mistral tokenizers and does not apply here: tokenization matches the base model's.

The checkpoints at steps 50 to 250 are in verl's format only. To convert one, download its actor/ folder and run verl's model merger from the kit's repo/: python -m verl.model_merger merge --backend fsdp --local_dir <global_step_N>/actor --target_dir <output dir>.

Layout

Each run folder mirrors that run's output directory:

  • checkpoints/global_step_N/ (N = 50, 100, …, 300), exactly as verl saved it:
    • actor/model_world_size_1_rank_0.pt: model weights (a single-rank FSDP2 state dict, not transformers format)
    • actor/optim_world_size_1_rank_0.pt: AdamW state
    • actor/extra_state_world_size_1_rank_0.pt, actor/fsdp_config.json, actor/huggingface/ (config and tokenizer)
    • data.pt: dataloader state
  • checkpoints/latest_checkpointed_iteration.txt: verl's record of the latest step (300), which its auto-resume reads.
  • train.log and run.json: the training log and the kit's run record.
  • run_one.out: what the kit's run_one.sh printed, including the training and evaluation exit codes.
  • hydra/.hydra/: the resolved training configuration (config.yaml), the command-line overrides (overrides.yaml) and Hydra's own settings (hydra.yaml).
  • validation/<step>.jsonl: the MATH-500 validation outputs, 500 records per step (prompt, response, reference answer, score, accuracy, whether the answer was boxed).
  • eval_step300/: the step-300 evaluation. It has results.csv and, for each suite, every generation (<suite>.generations.jsonl), the per-question scores (<suite>.scores.json) and the generation statistics (<suite>.generation.json), plus questions.json, scores.json and the logs. export/ is the step-300 model in transformers format (see "Loading the step-300 models").
  • eval.log, lead_u.log and lead_u_cosine_vs_true.json: the evaluator's log and the learned-direction measurement.
  • eval_fix/: how the evaluation was run (see below).
  • #6 only: basis/entropy_grad_qwen.pt, a copy of the kit's bases/entropy_grad_qwen.pt (the direction the band penalty is built around), and the logs and status files of the failed first evaluation attempt.
  • #7 only: patches/.

results/ at the top level holds the tables above as CSV files.

preflight/ at the top level holds what ran before launch: the dry-run configs, and the 2-step smoke tests with their GPU-memory samples. For INTUITOR they show the shipped code failing (qwen_intuitor_s0_smoke.train.log, the AssertionError), the run with only the first fix failing (qwen_intuitor_s0_smoke.patch1.train.log, the ValueError), and the run with both fixes completing (qwen_intuitor_s0_smoke.patch1+2.train.log).

Every checkpoint and evaluation file was hash-verified on the Hub against the local copy.

Deviations from the kit

No training setting differs from the kit.

INTUITOR: two fixes

As shipped, the kit's INTUITOR configuration stops at its first training step, so this run uses the kit's code plus two small fixes.

  1. verl/workers/engine/fsdp/transformer_impl.py: with use_remove_padding=false, the per-token mean log-probability that self-certainty needs kept its padding positions, and the trainer's length check failed (AssertionError in no_padding_2_padding). The fix trims it to each sequence's true length, as verl already does for the token entropy.
  2. verl/trainer/ppo/ray_trainer.py: the grpo_selfcertainty estimator is registered only when recipe/principal_subspaces/selfsup_rewards.py is imported, which the kit's configuration never does (ValueError: Unknown advantage estimator). The fix imports it when one of its estimators is requested.

Evaluation: the kit deletes its own temporary directory

run_one.sh exports TMPDIR=${RAYTMP}_tmp, deletes that directory right after training (line 135), and only then calls eval_one.sh. The evaluator's vLLM compiles FlashInfer's sampling kernels with nvcc the first time it runs, and nvcc cannot write into the deleted TMPDIR (nvcc fatal: Could not open output file). Training is not affected.

  • #6: the first evaluation failed this way after 78 seconds, before generating anything. Its files are eval_step300.failed_20261108T212727Z/ and eval.failed_20261108T212727Z.log; the evaluation exit 1 line in run_one.out refers to that log. The evaluation step alone was then re-run the way eval_one.sh runs it, in run_one.sh's environment, with a TMPDIR that exists (eval_fix/rerun_eval6.sh). The checkpoint, the evaluator and its settings are unchanged. lead_u_cosine_vs_true.json comes from the first pass; it reads only the checkpoint.
  • #7: the directory was recreated the moment run_one.sh deleted it (eval_fix/guard_tmpdir7.sh and its log), so its evaluation ran inside run_one.sh as usual.

For future runs, moving rm -rf "${RAYTMP}" "${RAYTMP}_tmp" in run_one.sh below the eval_one.sh call fixes this.

Environment

The few-GT bundle v5 lock (PyTorch 2.8.0+cu128, vLLM 0.11.0, Transformers 4.57.6) plus flash-attn 2.8.1, which the source environment also had: verl's padding utilities import it, and its Triton cross-entropy computes the log-probs. Hardware: one NVIDIA RTX PRO 5000 Blackwell (48 GB) per run.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for JaeWooShin/labelfree-runs-2026-11-07

Finetuned
(334)
this model

Paper for JaeWooShin/labelfree-runs-2026-11-07