ALFWorld 3B student: uncertainty-triggered on-policy distillation (final step 250)

Qwen2.5-3B-Instruct distilled on ALFWorld from the RL-trained teacher GiGPO-Qwen2.5-7B-Instruct-ALFWorld. The model is in global_step_250/huggingface/ (the final training step; weights, config and tokenizer).

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81"
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="global_step_250/huggingface", torch_dtype="bfloat16")
tok = AutoTokenizer.from_pretrained(repo, subfolder="global_step_250/huggingface")

Results

ALFWorld success rate (SR, %) and average turns of this one checkpoint, over three evaluation seeds (temperature 0.4, 50-step episode limit, 2,048 prompt tokens; 140 seen and 134 unseen tasks). The ± is the sample standard deviation over evaluation seeds.

Evaluation seed Seen SR Seen turns Unseen SR Unseen turns
42 90.00 (126/140) 13.32 88.06 (118/134) 15.01
123 90.71 (127/140) 13.70 88.06 (118/134) 15.52
2024 89.29 (125/140) 13.91 90.30 (121/134) 14.01
Mean ± SD 90.0 ± 0.7 13.6 88.8 ± 1.3 14.9

Training configuration

Setting Value
Student (initialisation) Qwen2.5-3B-Instruct
Teacher (frozen) GiGPO-Qwen2.5-7B-Instruct-ALFWorld
Training steps 250
Rollout batch 16 episodes per step
Training batch 64 (train_batch_size = ppo_mini_batch_size)
Seed 42
Episode limit 50 environment steps
Max prompt / response tokens 2,048 / 512
Rollout sampling temperature 1.0, top-p 1.0
Optimizer AdamW, lr 1e-6 constant, betas (0.9, 0.999), weight decay 0.01, gradient clip 1.0
Loss on non-triggered turns reverse KL to the teacher on the student's tokens (per-token OPD advantage, coefficient 1.0)
Loss on triggered turns SFT on the teacher-decoded turn, weight β = 1.0 (plain form)
Loss aggregation token mean
Uncertainty signal teacher confidence: mean teacher surprisal −log π_T over the student's tokens of the turn (temperature 1.0)
Trigger threshold per episode, the (1 − r) quantile of the last 1,000 turn scores; nothing triggers until 300 scores are collected; no per-episode cap
Target rate r linear 0.30 → 0.05 over the first 120 training steps, then 0.05
Intervention (teacher takeover) on: on a triggered turn the teacher's decoded action is executed in the environment
Teacher scheduling lazy: the teacher scores each turn and decodes a triggered turn only when needed (no speculative decoding); affects speed only, not what is trained
SFT warm-up / KL penalty / entropy loss none / none / none
Off-policy staleness at most 2 steps; weights synchronised every step
Parallelism FSDP trainer with Ulysses sequence parallel 2, dynamic batching (16,384 tokens per GPU); one student and one teacher vLLM engine, bf16

The tau1.28 in the repository name is the threshold of the fixed-threshold mode; this run uses the quantile threshold above, so that value is not used.

Resolved settings, as recorded in the training log:

algorithm:
  algorithm_type: on_policy_distill
  advantage_fn: multi_turn_opd
  advantage_fn_args: {kl_coef: 1.0}
  policy_loss_fn: said
  policy_loss_fn_args: {sft_coef: 1.0, sft_mode: plain, loss_agg_mode: token-mean}
  kl_penalty_fn: none
  kl_loss_fn: none
  entropy_loss_fn: none
  optimizer: {optimizer_type: adam, lr: 1.0e-06, lr_scheduler_type: constant, betas: [0.9, 0.999], weight_decay: 0.01, clip_grad: 1.0}
  sample_strategy: staleness_control
  sample_strategy_args: {max_staleness: 2}
workflow_args:
  detect_uq: confidence
  detect_temperature: 1.0
  tau_mode: quantile
  tau: 1.28            # used only when tau_mode is fixed
  window_size: 1000
  window_n_min: 300
  rate_schedule: linear
  r_start: 0.3
  r_end: 0.05
  r_decay_steps: 120
  trigger_cap: null
  teacher_takeover: true
  teacher_sched: lazy
  sft_warmup_steps: 0
  max_env_steps: 50
buffer: {batch_size: 16, train_batch_size: 64, total_steps: 250}
model: {max_prompt_tokens: 2048, max_response_tokens: 512}
rollout: {temperature: 1.0, top_p: 1.0}
trainer: {ulysses_sequence_parallel_size: 2, use_dynamic_bsz: true, ppo_max_token_len_per_gpu: 16384}
seed: 42
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81

Base model

Qwen/Qwen2.5-3B
Finetuned
(1,697)
this model