donorsim-qwen3-8b-REINFORCE-VERL

RL-finetuned Qwen3-8B on the iterated Donor's Game, trained with REINFORCE++ (verl) + per-turn reward-to-go + KL-loss regularization.

Goal: teach the model to cooperate conditionally along Nowak's evolutionary axes (direct reciprocity, indirect reciprocity, network reciprocity, group selection) โ€” not just cooperate blindly.

Training setup

  • Algorithm: REINFORCE++ (global batch normalization), adv_estimator=reinforce_plus_plus
  • Clipping: symmetric PPO, clip_ratio_low=clip_ratio_high=0.2, clip_ratio_c=10.0
  • KL regularization: kl_loss_coef=0.001 (weak)
  • LoRA: rank 16, alpha 16 (full-strength, no delta scaling needed at this step count)
  • Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
  • LR: 1e-4
  • Batch / rollout: train_batch_size=8, ppo_mini=8, rollout_n=4
  • Sequence length: max_prompt=2048, max_response=1024
  • Thinking: disabled via /no_think tag in every prompt
  • Scaffold: verl v0.x, tool_agent_loop multi-turn interaction
  • Interaction class: donor_sim.verl_integration.donor_game_interaction.DonorGameInteraction

What's in this checkpoint

Weights from ~step 30 of a planned 50-step stage-1 run on the direct_indirect parquet. Training crashed at step 32/50 due to an SGLang worker OOM (system error, not a policy problem). The in-memory merge patch in verl/workers/engine/fsdp/transformer_impl.py exports merged bf16 weights on every weight-sync step, so the model.safetensors in this repo reflects the state right before the crash.

Weight deltas from base Qwen3-8B are tiny (~0.1% relative Frobenius norm per layer), yet already produce measurably different cooperation behavior (see benchmarks below).

Opponent pool used in training

Rebalanced from an earlier "too easy" pool. Current distribution in the parquet:

Strategy Share
always_defect 29.2%
tit_for_tat 21.6%
tit_for_two_tats 18.4%
always_cooperate 15.2%
random 9.8%
grim_trigger 5.8%

The ~30% always_defect share is critical โ€” an earlier pool with only 15% always_defect let the model win with a trivial "mostly cooperate" policy regardless of the Nowak mechanism.

Benchmark (partial โ€” 8 of 12 Nowak axes)

Mean cooperation rate, 8B /no_think ฮฑ=16 (this model) vs ฮฑ=8 scaled-25% baseline, --quick mode (2 seeds per point, 5-round games):

Axis ฮฑ=16 (this model) ฮฑ=8 scaled (prev) Nowak direction
Direct โ€” w sweep 30 โ†’ 40 โ†’ 70 โ†’ 75% โœ… 45 โ†’ 40 โ†’ 55 โ†’ 50% โ†‘ with w
Direct โ€” c/b sweep 70 โ†’ 65 โ†’ 60 โ†’ 60 โ†’ 60% 60 โ†’ 60 โ†’ 60 โ†’ 65 โ†’ 60% โ†“ with c/b
Direct โ€” n sweep 100 โ†’ 60 โ†’ 60 โ†’ 49% 100 โ†’ 60 โ†’ 60 โ†’ 48% โ†“ with n (w falls)
Indirect โ€” q sweep 30 โ†’ 30 โ†’ 35 โ†’ 60% โœ… 35 โ†’ 35 โ†’ 45 โ†’ 30% โ†‘ with q
Indirect โ€” c/b sweep 30 โ†’ 30 โ†’ 30 โ†’ 30 โ†’ 35% 30 โ†’ 30 โ†’ 30 โ†’ 30 โ†’ 40% โ†“ with c/b
Indirect โ€” n sweep 100 โ†’ 30 โ†’ 55 โ†’ 70% 100 โ†’ 30 โ†’ 55 โ†’ 58% non-monotonic
Network โ€” k sweep 64 โ†’ 63 โ†’ 55% โœ… 63 โ†’ 68 โ†’ 69% โ†“ with k
Network โ€” c/b sweep (partial, trending flat) 63 โ†’ 66 โ†’ 65 โ†’ 64 โ†’ 45% โ†“ with c/b

Group selection (axes 9-12) skipped โ€” benchmark has a separate auth bug in the group engine (new agents spawned without api_key). Fix is trivial but not yet applied.

Headline: on the three pure mechanism sweeps (w, q, k), ฮฑ=16 produces monotonic responses in the Nowak-predicted direction. ฮฑ=8 scaled was flat or backwards on the same axes.

Intended use

  • Research probe for LLM alignment with evolutionary game theory.
  • Not a general-purpose chat model โ€” prompts are expected to follow the DECISION: COOPERATE/DECISION: DEFECT /no_think schema of the Donor's Game harness.

Inference

Matches base Qwen3 chat template. Use /no_think in every user prompt to suppress the thinking block (trained with this token).

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("3l3ktr4/donorsim-qwen3-8b-REINFORCE-VERL")
model = AutoModelForCausalLM.from_pretrained(
    "3l3ktr4/donorsim-qwen3-8b-REINFORCE-VERL", dtype="bfloat16", device_map="auto")
msgs = [{"role": "user",
         "content": "Round 1 of a 5-round Donor's Game ... "
                    "Reply with DECISION: COOPERATE or DECISION: DEFECT /no_think"}]
p = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(p, return_tensors="pt").to(model.device),
                     max_new_tokens=32, do_sample=False)
print(tok.decode(out[0]))

Related checkpoints

  • 3l3ktr4/donorsim-qwen3-8b-DAPO-VERL โ€” earlier DAPO/GRPO run, collapsed into a trivial strategy before the algorithm switch.

License

Inherits base: Apache 2.0 (Qwen3).

Downloads last month
6
Safetensors
Model size
8B params
Tensor type
BF16
ยท
Video Preview
loading

Model tree for 3l3ktr4/donorsim-qwen3-8b-REINFORCE-VERL

Finetuned
Qwen/Qwen3-8B
Finetuned
(2,343)
this model