ChristophSchuhmann commited on
Commit
b913ad1
·
verified ·
1 Parent(s): 4aec74a

Update TRAINING.md

Browse files
Files changed (1) hide show
  1. TRAINING.md +96 -0
TRAINING.md ADDED
@@ -0,0 +1,96 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Reproducing the 24,000-group Chatterbox S3Gen VC follow-ups
2
+
3
+ ## Model and data lineage
4
+
5
+ The frozen upstream decoder is
6
+ `ResembleAI/chatterbox@5bb1f6ee58e50c3b8d408bc82a6d3740c2db6e18/s3gen.safetensors`.
7
+ The trainable starting adapters are the completed v1 step-4,000 checkpoints
8
+ in [`laion/chatterbox-s3gen-vc-grow`](https://huggingface.co/laion/chatterbox-s3gen-vc-grow):
9
+ `continue` for the 50/50 control and 40/40/20 speaker arm, `merged_new` for
10
+ the remerged 50/50 arm. The v1 release explains the preceding 300-hour S6
11
+ and synthetic-sidecar supervised fine-tune, 99-axis score projector, packed
12
+ audio preparation, model versions and checkpoint structure. These follow-ups
13
+ **do not start from a fresh Chatterbox model**.
14
+
15
+ The frozen source/target condition table has 4,000 rows selected from that
16
+ 300-hour prepared pool, with 100 top-ranked eligible sources for each of 40
17
+ emotion axes. The sources are 3–8 seconds; target references are 5–10
18
+ seconds and their CAMPPlus cosine to the respective source is below 0.30.
19
+ There are 3,357 distinct source UIDs and 477 target UIDs. Each run makes six
20
+ deterministic, differently shuffled passes over the **same 4,000 pairs**.
21
+ The same pair table and random seeds are used across the three arms for a
22
+ controlled comparison. The prepared `pool.parquet`, `selection.parquet` and
23
+ four `input_ears_*.parquet` shards are required for exact reproduction but
24
+ are not included in these model repositories; original record locators and
25
+ licensed audio must be staged separately. Do not unpack samples into
26
+ thousands of loose files.
27
+
28
+ ## Rewards and optimizer
29
+
30
+ All three arms generate 16 on-policy candidates with S3Gen's 10-step Euler
31
+ flow sampler for a single source/target condition. Humaneness Ears **Medium**
32
+ at revision `818506970d93c809a3295a0a02c40dee8ff2bfc3` predicts the
33
+ `audiobox:CE` output (AudioBox Content Enjoyment) and 40 emotion scores on
34
+ each generated output. The two strongest Ears emotion axes of the **source
35
+ audio** define the emotion fidelity MSE. The third source emotion is omitted.
36
+ The speaker-aware arm additionally runs the frozen S3Gen CAMPPlus encoder on
37
+ each output and computes cosine to the target-reference CAMPPlus embedding.
38
+
39
+ Each raw component is independently standardized across all 16 candidates
40
+ in the group. The control arms use `0.50*z(CE)+0.50*z(-top2_MSE)`; the
41
+ speaker arm uses `0.40*z(CE)+0.40*z(-top2_MSE)+0.20*z(target_CAMP_cosine)`.
42
+ The weighted score is clipped to `[-2.5, 2.5]` and centered to zero group
43
+ mean. Signed advantages weight the flow-matching velocity loss on the
44
+ generated mel portion, plus `0.025` times the squared velocity difference
45
+ from the respective frozen v1 step-4,000 policy. This is GROW-inspired
46
+ group-relative flow optimization, not token-level GRPO. Ears is a **training
47
+ reward model**, so its scores are not an independent evaluation.
48
+
49
+ The only trainable parameters are the rank-128 q/k/v/out LoRA and the
50
+ previously trained 99-score projection. All other Chatterbox modules remain
51
+ frozen. Each arm restarts AdamW (`weight_decay=0.01`) at the v1 step-4,000
52
+ model, warms up over 1,200 of 24,000 groups, peaks at `2e-6`, then follows
53
+ one cosine decay to `2e-7`. The 50/50 remerged arm merges the completed v1
54
+ `merged_new` linear LoRA into its effective base a second time, starts a new
55
+ zero-output rank-128 LoRA, and retains the v1 score projector. Four GPUs
56
+ process four candidates each, all-gather reward components, and average
57
+ gradients before every synchronized optimizer update. The frozen anchor is
58
+ the respective starting policy. Checkpoints every 1,000 groups include model
59
+ trainables, full AdamW state, code/reward/schedule metadata and an atomic
60
+ completion marker.
61
+
62
+ ## Jupiter execution and resume
63
+
64
+ The recorded Jupiter Slurm scripts request one exclusive Booster node, four
65
+ GH200 GPUs, four tasks, one GPU per task, project `reformo`, and 11h50
66
+ walltime. An initial 20-group smoke checks identical initial trainables
67
+ across ranks, finite rewards and gradients, and nonzero updates. Production
68
+ must not begin without the matching smoke PASS marker. Logs, metrics,
69
+ checkpoints and canary audio stay in Scratch; metrics use one JSONL and
70
+ canaries one TAR per arm. This avoids large inode counts. If a job reaches
71
+ the walltime, its dependent fallback can resume from the latest complete
72
+ optimizer-bearing checkpoint. The actual `run.sbatch` paths are Jupiter
73
+ specific; change account, storage, environment and paths before reuse.
74
+
75
+ The repositories contain `code/grow_v2/train_ddp.py` and
76
+ `code/grow_v3/train_speaker.py`, plus their imported v1/base helpers,
77
+ `code/grow_followup/infer_followup.py`, the Slurm scripts, requirements and
78
+ the fixed 99-axis score order. For continuation, provide the same packed
79
+ pool and selection order, set `CHATTERBOX_RUN_ROOT`,
80
+ `CHATTERBOX_GROW_TRAIN_ROOT`, and the arm's `CHATTERBOX_GROW_V2_ROOT` or
81
+ `CHATTERBOX_GROW_V3_ROOT` as appropriate. The new schedule and objective
82
+ must match the checkpoint; changing either is a new experiment, not an
83
+ exact resume.
84
+
85
+ ## Why three arms, and what is not yet known
86
+
87
+ The two 50/50 arms test whether continued training of the existing LoRA or
88
+ merging it into the base and starting a fresh LoRA is better. The 40/40/20
89
+ arm tests whether an explicit speaker-identity reward prevents the slight
90
+ speaker-similarity regression observed in the earlier CE-free v1 pilot.
91
+ No default recommendation should be made from training reward alone.
92
+ Held-out, matched-seed evaluation must report a speaker cosine, source
93
+ emotion fidelity, an independent quality/naturalness metric, content
94
+ accuracy (e.g. ASR/word error rate), and listening examples. The holdout
95
+ should ideally be speaker-disjoint, which the previous UID-only split was
96
+ not guaranteed to be.