Qwen2.5-1.5B-Instruct Β· Sokoban (depth-8) Β· GRPO Β· step 150

Qwen/Qwen2.5-1.5B-Instruct fine-tuned with multi-turn RL on text Sokoban using GRPO (episode-level group-relative advantages, std-normalized), 8 rollouts per puzzle, trained from scratch (env seed 0).

This is the step-150 (final) checkpoint, which is also the best of the saved checkpoints by measured eval (checkpoints were saved every 25 steps; evals ran every 10 steps). Sibling models from the same comparison: GiGPO Β· Tree-GRPO Β· RegPO.

Results

Eval: 512 fixed puzzles, every 10 steps, sampling temperature 0.4. Metric: row-level val/regret_sokoban/success_rate from the training logs (fraction of eval rows marked solved; not a per-episode win rate). Untrained Qwen2.5-1.5B-Instruct: 6.2.

Model step 50 step 100 step 150 (this checkpoint) mean of last 5 evals (110–150) best eval (step)
GRPO 9.0 12.1 14.6 15.6 18.1 (140)
GiGPO 15.9 15.1 20.7 18.1 20.7 (150)
Tree-GRPO 9.2 17.9 25.8 22.1 25.8 (150)
RegPO 9.2 19.6 29.3 28.9 35.8 (140)

Eval curve of this model:

step 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150
eval success (%) 6.2 7.3 6.3 7.7 9.1 9.0 11.4 11.3 9.6 10.7 12.1 15.1 12.6 17.6 18.1 14.6

Caveats: one seed per method; 150 training steps; Tree-GRPO and RegPO were warm-started from the GRPO model's step-50 checkpoint, while GRPO and GiGPO were trained from scratch, so the cleanest comparison is RegPO / Tree-GRPO vs GRPO over steps 51–150. Peak evals between checkpoints were not saved.

Environment and training setup

  • Game: 6Γ—6 room, 2 boxes, rooms generated by reverse play with search depth 8 (no solution-length filter); 10 turns per episode, at most 2 moves per turn (20 moves in total).
  • Observation / prompt: RAGEN-style prompt with a symbol grid plus zero-indexed coordinates of player, boxes and targets; last 2 turns of history; no thinking (the assistant turn is prefilled with <answer>).
  • Action format: Up, Down, Left, Right, separated by ||, inside <answer>…</answer>. Strict format: text after </answer> or more than 2 moves makes the turn invalid (no move, βˆ’0.5 penalty).
  • Reward: gym-sokoban: βˆ’0.1 per move, +1 box on target, βˆ’1 box off target, +10 solved.
  • Optimization: 16 puzzles per step, PPO mini-batch 256, lr 1e-6, KL loss 0.05, clip 0.2/0.2, token-mean loss aggregation, no entropy bonus, max response 512 tokens, max prompt 3072 tokens.
  • Framework: verl 0.6.1 + verl-agent (multi-turn agent loop), vLLM rollouts.
  • Weights: bf16 safetensors, converted from the FSDP fp32 checkpoint (bf16 is also the precision vLLM used for all training rollouts and evals).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="bfloat16", device_map="auto")

prompt = open("prompt.txt").read()  # a turn prompt in the format below, ending with the "<answer>" prefill
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=16, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True))  # e.g. " Up </answer>"

Example turn prompt (as used in training; the model completes after the <answer> prefill):

<|im_start|>system
You're a helpful assistant. You always respond by giving your answer in <answer>...</answer>. Max response length: 200 words (tokens).<|im_end|>
<|im_start|>user
You are solving the Sokoban puzzle. You are the player and you need to push all boxes to targets.
You are provided with a symbol grid and the zero-indexed coordinates of the player, each box, and each target. 
Coordinates range from the top-left corner (0, 0) to the bottom-right corner (5, 5). 
When you are exactly next to a box, you can push it by moving in the same direction. 
You cannot push a box through a wall, and you cannot pull a box.
The answer should be a sequence of actions, like <answer>Right || Up</answer>.
Your available actions are:
Up, Down, Left, Right
You can make up to 2 actions per turn (20 in total), separated by the action separator " || "

Turn 1:
State:
Coordinates: 
Board size: 6 rows x 6 cols (zero-indexed).
Targets: (2, 1), (3, 4)
Boxes: (3, 1), (3, 2)
Player: (4, 3)
Grid Map: 
######
######
#O####
#XX_O#
#__P_#
######
You have 20 actions left. Always output: <answer> [your answer] </answer> with no extra text. Strictly follow this format. Max response length: 200 words (tokens).
<|im_end|>
<|im_start|>assistant
<answer>
Downloads last month
38
Safetensors
Model size
2B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo

Finetuned
(2,160)
this model