Instructions to use 3l3ktr4/donorsim-qwen3-8b-REINFORCE-VERL with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use 3l3ktr4/donorsim-qwen3-8b-REINFORCE-VERL with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("3l3ktr4/donorsim-qwen3-8b-REINFORCE-VERL") model = AutoModelForCausalLM.from_pretrained("3l3ktr4/donorsim-qwen3-8b-REINFORCE-VERL", device_map="auto") - Notebooks
- Google Colab
- Kaggle
donorsim-qwen3-8b-REINFORCE-VERL
RL-finetuned Qwen3-8B on the iterated Donor's Game, trained with REINFORCE++ (verl) + per-turn reward-to-go + KL-loss regularization.
Goal: teach the model to cooperate conditionally along Nowak's evolutionary axes (direct reciprocity, indirect reciprocity, network reciprocity, group selection) โ not just cooperate blindly.
Training setup
- Algorithm: REINFORCE++ (global batch normalization),
adv_estimator=reinforce_plus_plus - Clipping: symmetric PPO,
clip_ratio_low=clip_ratio_high=0.2,clip_ratio_c=10.0 - KL regularization:
kl_loss_coef=0.001(weak) - LoRA: rank 16, alpha 16 (full-strength, no delta scaling needed at this step count)
- Target modules:
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj - LR: 1e-4
- Batch / rollout:
train_batch_size=8,ppo_mini=8,rollout_n=4 - Sequence length:
max_prompt=2048,max_response=1024 - Thinking: disabled via
/no_thinktag in every prompt - Scaffold: verl v0.x,
tool_agent_loopmulti-turn interaction - Interaction class:
donor_sim.verl_integration.donor_game_interaction.DonorGameInteraction
What's in this checkpoint
Weights from ~step 30 of a planned 50-step stage-1 run on the
direct_indirect parquet. Training crashed at step 32/50 due to an
SGLang worker OOM (system error, not a policy problem). The
in-memory merge patch in
verl/workers/engine/fsdp/transformer_impl.py exports merged bf16
weights on every weight-sync step, so the model.safetensors in this
repo reflects the state right before the crash.
Weight deltas from base Qwen3-8B are tiny (~0.1% relative Frobenius norm per layer), yet already produce measurably different cooperation behavior (see benchmarks below).
Opponent pool used in training
Rebalanced from an earlier "too easy" pool. Current distribution in the parquet:
| Strategy | Share |
|---|---|
| always_defect | 29.2% |
| tit_for_tat | 21.6% |
| tit_for_two_tats | 18.4% |
| always_cooperate | 15.2% |
| random | 9.8% |
| grim_trigger | 5.8% |
The ~30% always_defect share is critical โ an earlier pool with only 15% always_defect let the model win with a trivial "mostly cooperate" policy regardless of the Nowak mechanism.
Benchmark (partial โ 8 of 12 Nowak axes)
Mean cooperation rate, 8B /no_think ฮฑ=16 (this model) vs ฮฑ=8 scaled-25%
baseline, --quick mode (2 seeds per point, 5-round games):
| Axis | ฮฑ=16 (this model) | ฮฑ=8 scaled (prev) | Nowak direction |
|---|---|---|---|
| Direct โ w sweep | 30 โ 40 โ 70 โ 75% โ | 45 โ 40 โ 55 โ 50% | โ with w |
| Direct โ c/b sweep | 70 โ 65 โ 60 โ 60 โ 60% | 60 โ 60 โ 60 โ 65 โ 60% | โ with c/b |
| Direct โ n sweep | 100 โ 60 โ 60 โ 49% | 100 โ 60 โ 60 โ 48% | โ with n (w falls) |
| Indirect โ q sweep | 30 โ 30 โ 35 โ 60% โ | 35 โ 35 โ 45 โ 30% | โ with q |
| Indirect โ c/b sweep | 30 โ 30 โ 30 โ 30 โ 35% | 30 โ 30 โ 30 โ 30 โ 40% | โ with c/b |
| Indirect โ n sweep | 100 โ 30 โ 55 โ 70% | 100 โ 30 โ 55 โ 58% | non-monotonic |
| Network โ k sweep | 64 โ 63 โ 55% โ | 63 โ 68 โ 69% | โ with k |
| Network โ c/b sweep | (partial, trending flat) | 63 โ 66 โ 65 โ 64 โ 45% | โ with c/b |
Group selection (axes 9-12) skipped โ benchmark has a separate auth
bug in the group engine (new agents spawned without api_key). Fix
is trivial but not yet applied.
Headline: on the three pure mechanism sweeps (w, q, k), ฮฑ=16 produces monotonic responses in the Nowak-predicted direction. ฮฑ=8 scaled was flat or backwards on the same axes.
Intended use
- Research probe for LLM alignment with evolutionary game theory.
- Not a general-purpose chat model โ prompts are expected to
follow the
DECISION: COOPERATE/DECISION: DEFECT /no_thinkschema of the Donor's Game harness.
Inference
Matches base Qwen3 chat template. Use /no_think in every user
prompt to suppress the thinking block (trained with this token).
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("3l3ktr4/donorsim-qwen3-8b-REINFORCE-VERL")
model = AutoModelForCausalLM.from_pretrained(
"3l3ktr4/donorsim-qwen3-8b-REINFORCE-VERL", dtype="bfloat16", device_map="auto")
msgs = [{"role": "user",
"content": "Round 1 of a 5-round Donor's Game ... "
"Reply with DECISION: COOPERATE or DECISION: DEFECT /no_think"}]
p = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(p, return_tensors="pt").to(model.device),
max_new_tokens=32, do_sample=False)
print(tok.decode(out[0]))
Related checkpoints
3l3ktr4/donorsim-qwen3-8b-DAPO-VERLโ earlier DAPO/GRPO run, collapsed into a trivial strategy before the algorithm switch.
License
Inherits base: Apache 2.0 (Qwen3).
- Downloads last month
- 6