Instructions to use Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo") model = AutoModelForCausalLM.from_pretrained("Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo
- SGLang
How to use Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo with Docker Model Runner:
docker model run hf.co/Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo
Qwen2.5-1.5B-Instruct Β· Sokoban (depth-8) Β· GRPO Β· step 150
Qwen/Qwen2.5-1.5B-Instruct fine-tuned with multi-turn RL on text Sokoban using GRPO (episode-level group-relative advantages, std-normalized), 8 rollouts per puzzle, trained from scratch (env seed 0).
This is the step-150 (final) checkpoint, which is also the best of the saved checkpoints by measured eval (checkpoints were saved every 25 steps; evals ran every 10 steps). Sibling models from the same comparison: GiGPO Β· Tree-GRPO Β· RegPO.
Results
Eval: 512 fixed puzzles, every 10 steps, sampling temperature 0.4. Metric: row-level val/regret_sokoban/success_rate from the training logs (fraction of eval rows marked solved; not a per-episode win rate). Untrained Qwen2.5-1.5B-Instruct: 6.2.
| Model | step 50 | step 100 | step 150 (this checkpoint) | mean of last 5 evals (110β150) | best eval (step) |
|---|---|---|---|---|---|
| GRPO | 9.0 | 12.1 | 14.6 | 15.6 | 18.1 (140) |
| GiGPO | 15.9 | 15.1 | 20.7 | 18.1 | 20.7 (150) |
| Tree-GRPO | 9.2 | 17.9 | 25.8 | 22.1 | 25.8 (150) |
| RegPO | 9.2 | 19.6 | 29.3 | 28.9 | 35.8 (140) |
Eval curve of this model:
| step | 0 | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90 | 100 | 110 | 120 | 130 | 140 | 150 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| eval success (%) | 6.2 | 7.3 | 6.3 | 7.7 | 9.1 | 9.0 | 11.4 | 11.3 | 9.6 | 10.7 | 12.1 | 15.1 | 12.6 | 17.6 | 18.1 | 14.6 |
Caveats: one seed per method; 150 training steps; Tree-GRPO and RegPO were warm-started from the GRPO model's step-50 checkpoint, while GRPO and GiGPO were trained from scratch, so the cleanest comparison is RegPO / Tree-GRPO vs GRPO over steps 51β150. Peak evals between checkpoints were not saved.
Environment and training setup
- Game: 6Γ6 room, 2 boxes, rooms generated by reverse play with search depth 8 (no solution-length filter); 10 turns per episode, at most 2 moves per turn (20 moves in total).
- Observation / prompt: RAGEN-style prompt with a symbol grid plus zero-indexed coordinates of player, boxes and targets; last 2 turns of history; no thinking (the assistant turn is prefilled with
<answer>). - Action format:
Up,Down,Left,Right, separated by||, inside<answer>β¦</answer>. Strict format: text after</answer>or more than 2 moves makes the turn invalid (no move, β0.5 penalty). - Reward: gym-sokoban: β0.1 per move, +1 box on target, β1 box off target, +10 solved.
- Optimization: 16 puzzles per step, PPO mini-batch 256, lr 1e-6, KL loss 0.05, clip 0.2/0.2, token-mean loss aggregation, no entropy bonus, max response 512 tokens, max prompt 3072 tokens.
- Framework: verl 0.6.1 + verl-agent (multi-turn agent loop), vLLM rollouts.
- Weights: bf16
safetensors, converted from the FSDP fp32 checkpoint (bf16 is also the precision vLLM used for all training rollouts and evals).
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "Elrond1701/qwen2.5-1.5b-sokoban-d8-grpo"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="bfloat16", device_map="auto")
prompt = open("prompt.txt").read() # a turn prompt in the format below, ending with the "<answer>" prefill
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=16, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True)) # e.g. " Up </answer>"
Example turn prompt (as used in training; the model completes after the <answer> prefill):
<|im_start|>system
You're a helpful assistant. You always respond by giving your answer in <answer>...</answer>. Max response length: 200 words (tokens).<|im_end|>
<|im_start|>user
You are solving the Sokoban puzzle. You are the player and you need to push all boxes to targets.
You are provided with a symbol grid and the zero-indexed coordinates of the player, each box, and each target.
Coordinates range from the top-left corner (0, 0) to the bottom-right corner (5, 5).
When you are exactly next to a box, you can push it by moving in the same direction.
You cannot push a box through a wall, and you cannot pull a box.
The answer should be a sequence of actions, like <answer>Right || Up</answer>.
Your available actions are:
Up, Down, Left, Right
You can make up to 2 actions per turn (20 in total), separated by the action separator " || "
Turn 1:
State:
Coordinates:
Board size: 6 rows x 6 cols (zero-indexed).
Targets: (2, 1), (3, 4)
Boxes: (3, 1), (3, 2)
Player: (4, 3)
Grid Map:
######
######
#O####
#XX_O#
#__P_#
######
You have 20 actions left. Always output: <answer> [your answer] </answer> with no extra text. Strictly follow this format. Max response length: 200 words (tokens).
<|im_end|>
<|im_start|>assistant
<answer>
- Downloads last month
- 38