Instructions to use ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81
- SGLang
How to use ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81 with Docker Model Runner:
docker model run hf.co/ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81
ALFWorld 3B student: uncertainty-triggered on-policy distillation (final step 250)
Qwen2.5-3B-Instruct distilled on ALFWorld from the RL-trained teacher GiGPO-Qwen2.5-7B-Instruct-ALFWorld.
The model is in global_step_250/huggingface/ (the final training step; weights, config and tokenizer).
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "ryan-xman66/tcod-resume-alfworld_sacd_3b_confidence_tau1.28_beta1.0_sftplain_lazy_teache-61ce81"
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="global_step_250/huggingface", torch_dtype="bfloat16")
tok = AutoTokenizer.from_pretrained(repo, subfolder="global_step_250/huggingface")
Results
ALFWorld success rate (SR, %) and average turns of this one checkpoint, over three evaluation seeds (temperature 0.4, 50-step episode limit, 2,048 prompt tokens; 140 seen and 134 unseen tasks). The ± is the sample standard deviation over evaluation seeds.
| Evaluation seed | Seen SR | Seen turns | Unseen SR | Unseen turns |
|---|---|---|---|---|
| 42 | 90.00 (126/140) | 13.32 | 88.06 (118/134) | 15.01 |
| 123 | 90.71 (127/140) | 13.70 | 88.06 (118/134) | 15.52 |
| 2024 | 89.29 (125/140) | 13.91 | 90.30 (121/134) | 14.01 |
| Mean ± SD | 90.0 ± 0.7 | 13.6 | 88.8 ± 1.3 | 14.9 |
Training configuration
| Setting | Value |
|---|---|
| Student (initialisation) | Qwen2.5-3B-Instruct |
| Teacher (frozen) | GiGPO-Qwen2.5-7B-Instruct-ALFWorld |
| Training steps | 250 |
| Rollout batch | 16 episodes per step |
| Training batch | 64 (train_batch_size = ppo_mini_batch_size) |
| Seed | 42 |
| Episode limit | 50 environment steps |
| Max prompt / response tokens | 2,048 / 512 |
| Rollout sampling | temperature 1.0, top-p 1.0 |
| Optimizer | AdamW, lr 1e-6 constant, betas (0.9, 0.999), weight decay 0.01, gradient clip 1.0 |
| Loss on non-triggered turns | reverse KL to the teacher on the student's tokens (per-token OPD advantage, coefficient 1.0) |
| Loss on triggered turns | SFT on the teacher-decoded turn, weight β = 1.0 (plain form) |
| Loss aggregation | token mean |
| Uncertainty signal | teacher confidence: mean teacher surprisal −log π_T over the student's tokens of the turn (temperature 1.0) |
| Trigger threshold | per episode, the (1 − r) quantile of the last 1,000 turn scores; nothing triggers until 300 scores are collected; no per-episode cap |
| Target rate r | linear 0.30 → 0.05 over the first 120 training steps, then 0.05 |
| Intervention (teacher takeover) | on: on a triggered turn the teacher's decoded action is executed in the environment |
| Teacher scheduling | lazy: the teacher scores each turn and decodes a triggered turn only when needed (no speculative decoding); affects speed only, not what is trained |
| SFT warm-up / KL penalty / entropy loss | none / none / none |
| Off-policy staleness | at most 2 steps; weights synchronised every step |
| Parallelism | FSDP trainer with Ulysses sequence parallel 2, dynamic batching (16,384 tokens per GPU); one student and one teacher vLLM engine, bf16 |
The tau1.28 in the repository name is the threshold of the fixed-threshold mode; this run uses the
quantile threshold above, so that value is not used.
Resolved settings, as recorded in the training log:
algorithm:
algorithm_type: on_policy_distill
advantage_fn: multi_turn_opd
advantage_fn_args: {kl_coef: 1.0}
policy_loss_fn: said
policy_loss_fn_args: {sft_coef: 1.0, sft_mode: plain, loss_agg_mode: token-mean}
kl_penalty_fn: none
kl_loss_fn: none
entropy_loss_fn: none
optimizer: {optimizer_type: adam, lr: 1.0e-06, lr_scheduler_type: constant, betas: [0.9, 0.999], weight_decay: 0.01, clip_grad: 1.0}
sample_strategy: staleness_control
sample_strategy_args: {max_staleness: 2}
workflow_args:
detect_uq: confidence
detect_temperature: 1.0
tau_mode: quantile
tau: 1.28 # used only when tau_mode is fixed
window_size: 1000
window_n_min: 300
rate_schedule: linear
r_start: 0.3
r_end: 0.05
r_decay_steps: 120
trigger_cap: null
teacher_takeover: true
teacher_sched: lazy
sft_warmup_steps: 0
max_env_steps: 50
buffer: {batch_size: 16, train_batch_size: 64, total_steps: 250}
model: {max_prompt_tokens: 2048, max_response_tokens: 512}
rollout: {temperature: 1.0, top_p: 1.0}
trainer: {ulysses_sequence_parallel_size: 2, use_dynamic_bsz: true, ppo_max_token_len_per_gpu: 16384}
seed: 42