Instructions to use laion/chatterbox-s3gen-vc-grow-ce-remerged with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Chatterbox
How to use laion/chatterbox-s3gen-vc-grow-ce-remerged with Chatterbox:
# pip install chatterbox-tts import torchaudio as ta from chatterbox.tts import ChatterboxTTS model = ChatterboxTTS.from_pretrained(device="cuda") text = "Ezreal and Jinx teamed up with Ahri, Yasuo, and Teemo to take down the enemy's Nexus in an epic late-game pentakill." wav = model.generate(text) ta.save("test-1.wav", wav, model.sr) # If you want to synthesize with a different voice, specify the audio prompt AUDIO_PROMPT_PATH="YOUR_FILE.wav" wav = model.generate(text, audio_prompt_path=AUDIO_PROMPT_PATH) ta.save("test-2.wav", wav, model.sr) - Notebooks
- Google Colab
- Kaggle
Update TRAINING.md
Browse files- TRAINING.md +96 -0
TRAINING.md
ADDED
|
@@ -0,0 +1,96 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Reproducing the 24,000-group Chatterbox S3Gen VC follow-ups
|
| 2 |
+
|
| 3 |
+
## Model and data lineage
|
| 4 |
+
|
| 5 |
+
The frozen upstream decoder is
|
| 6 |
+
`ResembleAI/chatterbox@5bb1f6ee58e50c3b8d408bc82a6d3740c2db6e18/s3gen.safetensors`.
|
| 7 |
+
The trainable starting adapters are the completed v1 step-4,000 checkpoints
|
| 8 |
+
in [`laion/chatterbox-s3gen-vc-grow`](https://huggingface.co/laion/chatterbox-s3gen-vc-grow):
|
| 9 |
+
`continue` for the 50/50 control and 40/40/20 speaker arm, `merged_new` for
|
| 10 |
+
the remerged 50/50 arm. The v1 release explains the preceding 300-hour S6
|
| 11 |
+
and synthetic-sidecar supervised fine-tune, 99-axis score projector, packed
|
| 12 |
+
audio preparation, model versions and checkpoint structure. These follow-ups
|
| 13 |
+
**do not start from a fresh Chatterbox model**.
|
| 14 |
+
|
| 15 |
+
The frozen source/target condition table has 4,000 rows selected from that
|
| 16 |
+
300-hour prepared pool, with 100 top-ranked eligible sources for each of 40
|
| 17 |
+
emotion axes. The sources are 3–8 seconds; target references are 5–10
|
| 18 |
+
seconds and their CAMPPlus cosine to the respective source is below 0.30.
|
| 19 |
+
There are 3,357 distinct source UIDs and 477 target UIDs. Each run makes six
|
| 20 |
+
deterministic, differently shuffled passes over the **same 4,000 pairs**.
|
| 21 |
+
The same pair table and random seeds are used across the three arms for a
|
| 22 |
+
controlled comparison. The prepared `pool.parquet`, `selection.parquet` and
|
| 23 |
+
four `input_ears_*.parquet` shards are required for exact reproduction but
|
| 24 |
+
are not included in these model repositories; original record locators and
|
| 25 |
+
licensed audio must be staged separately. Do not unpack samples into
|
| 26 |
+
thousands of loose files.
|
| 27 |
+
|
| 28 |
+
## Rewards and optimizer
|
| 29 |
+
|
| 30 |
+
All three arms generate 16 on-policy candidates with S3Gen's 10-step Euler
|
| 31 |
+
flow sampler for a single source/target condition. Humaneness Ears **Medium**
|
| 32 |
+
at revision `818506970d93c809a3295a0a02c40dee8ff2bfc3` predicts the
|
| 33 |
+
`audiobox:CE` output (AudioBox Content Enjoyment) and 40 emotion scores on
|
| 34 |
+
each generated output. The two strongest Ears emotion axes of the **source
|
| 35 |
+
audio** define the emotion fidelity MSE. The third source emotion is omitted.
|
| 36 |
+
The speaker-aware arm additionally runs the frozen S3Gen CAMPPlus encoder on
|
| 37 |
+
each output and computes cosine to the target-reference CAMPPlus embedding.
|
| 38 |
+
|
| 39 |
+
Each raw component is independently standardized across all 16 candidates
|
| 40 |
+
in the group. The control arms use `0.50*z(CE)+0.50*z(-top2_MSE)`; the
|
| 41 |
+
speaker arm uses `0.40*z(CE)+0.40*z(-top2_MSE)+0.20*z(target_CAMP_cosine)`.
|
| 42 |
+
The weighted score is clipped to `[-2.5, 2.5]` and centered to zero group
|
| 43 |
+
mean. Signed advantages weight the flow-matching velocity loss on the
|
| 44 |
+
generated mel portion, plus `0.025` times the squared velocity difference
|
| 45 |
+
from the respective frozen v1 step-4,000 policy. This is GROW-inspired
|
| 46 |
+
group-relative flow optimization, not token-level GRPO. Ears is a **training
|
| 47 |
+
reward model**, so its scores are not an independent evaluation.
|
| 48 |
+
|
| 49 |
+
The only trainable parameters are the rank-128 q/k/v/out LoRA and the
|
| 50 |
+
previously trained 99-score projection. All other Chatterbox modules remain
|
| 51 |
+
frozen. Each arm restarts AdamW (`weight_decay=0.01`) at the v1 step-4,000
|
| 52 |
+
model, warms up over 1,200 of 24,000 groups, peaks at `2e-6`, then follows
|
| 53 |
+
one cosine decay to `2e-7`. The 50/50 remerged arm merges the completed v1
|
| 54 |
+
`merged_new` linear LoRA into its effective base a second time, starts a new
|
| 55 |
+
zero-output rank-128 LoRA, and retains the v1 score projector. Four GPUs
|
| 56 |
+
process four candidates each, all-gather reward components, and average
|
| 57 |
+
gradients before every synchronized optimizer update. The frozen anchor is
|
| 58 |
+
the respective starting policy. Checkpoints every 1,000 groups include model
|
| 59 |
+
trainables, full AdamW state, code/reward/schedule metadata and an atomic
|
| 60 |
+
completion marker.
|
| 61 |
+
|
| 62 |
+
## Jupiter execution and resume
|
| 63 |
+
|
| 64 |
+
The recorded Jupiter Slurm scripts request one exclusive Booster node, four
|
| 65 |
+
GH200 GPUs, four tasks, one GPU per task, project `reformo`, and 11h50
|
| 66 |
+
walltime. An initial 20-group smoke checks identical initial trainables
|
| 67 |
+
across ranks, finite rewards and gradients, and nonzero updates. Production
|
| 68 |
+
must not begin without the matching smoke PASS marker. Logs, metrics,
|
| 69 |
+
checkpoints and canary audio stay in Scratch; metrics use one JSONL and
|
| 70 |
+
canaries one TAR per arm. This avoids large inode counts. If a job reaches
|
| 71 |
+
the walltime, its dependent fallback can resume from the latest complete
|
| 72 |
+
optimizer-bearing checkpoint. The actual `run.sbatch` paths are Jupiter
|
| 73 |
+
specific; change account, storage, environment and paths before reuse.
|
| 74 |
+
|
| 75 |
+
The repositories contain `code/grow_v2/train_ddp.py` and
|
| 76 |
+
`code/grow_v3/train_speaker.py`, plus their imported v1/base helpers,
|
| 77 |
+
`code/grow_followup/infer_followup.py`, the Slurm scripts, requirements and
|
| 78 |
+
the fixed 99-axis score order. For continuation, provide the same packed
|
| 79 |
+
pool and selection order, set `CHATTERBOX_RUN_ROOT`,
|
| 80 |
+
`CHATTERBOX_GROW_TRAIN_ROOT`, and the arm's `CHATTERBOX_GROW_V2_ROOT` or
|
| 81 |
+
`CHATTERBOX_GROW_V3_ROOT` as appropriate. The new schedule and objective
|
| 82 |
+
must match the checkpoint; changing either is a new experiment, not an
|
| 83 |
+
exact resume.
|
| 84 |
+
|
| 85 |
+
## Why three arms, and what is not yet known
|
| 86 |
+
|
| 87 |
+
The two 50/50 arms test whether continued training of the existing LoRA or
|
| 88 |
+
merging it into the base and starting a fresh LoRA is better. The 40/40/20
|
| 89 |
+
arm tests whether an explicit speaker-identity reward prevents the slight
|
| 90 |
+
speaker-similarity regression observed in the earlier CE-free v1 pilot.
|
| 91 |
+
No default recommendation should be made from training reward alone.
|
| 92 |
+
Held-out, matched-seed evaluation must report a speaker cosine, source
|
| 93 |
+
emotion fidelity, an independent quality/naturalness metric, content
|
| 94 |
+
accuracy (e.g. ASR/word error rate), and listening examples. The holdout
|
| 95 |
+
should ideally be speaker-disjoint, which the previous UID-only split was
|
| 96 |
+
not guaranteed to be.
|