Instructions to use laion/chatterbox-s3gen-vc-grow-ce-remerged with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Chatterbox
How to use laion/chatterbox-s3gen-vc-grow-ce-remerged with Chatterbox:
# pip install chatterbox-tts import torchaudio as ta from chatterbox.tts import ChatterboxTTS model = ChatterboxTTS.from_pretrained(device="cuda") text = "Ezreal and Jinx teamed up with Ahri, Yasuo, and Teemo to take down the enemy's Nexus in an epic late-game pentakill." wav = model.generate(text) ta.save("test-1.wav", wav, model.sr) # If you want to synthesize with a different voice, specify the audio prompt AUDIO_PROMPT_PATH="YOUR_FILE.wav" wav = model.generate(text, audio_prompt_path=AUDIO_PROMPT_PATH) ta.save("test-2.wav", wav, model.sr) - Notebooks
- Google Colab
- Kaggle
Download TRAINING.md from laion/chatterbox-s3gen-vc-grow-ce-remerged: direct link, hf CLI and curl.
- Browser
- Download file 5.74 kB
-
https://huggingface.co/laion/chatterbox-s3gen-vc-grow-ce-remerged/resolve/main/TRAINING.md
- Command line
-
hf download hf://laion/chatterbox-s3gen-vc-grow-ce-remerged/TRAINING.md
-
curl -L -o TRAINING.md https://huggingface.co/laion/chatterbox-s3gen-vc-grow-ce-remerged/resolve/main/TRAINING.md
Reproducing the 24,000-group Chatterbox S3Gen VC follow-ups
Model and data lineage
The frozen upstream decoder is
ResembleAI/chatterbox@5bb1f6ee58e50c3b8d408bc82a6d3740c2db6e18/s3gen.safetensors.
The trainable starting adapters are the completed v1 step-4,000 checkpoints
in laion/chatterbox-s3gen-vc-grow:
continue for the 50/50 control and 40/40/20 speaker arm, merged_new for
the remerged 50/50 arm. The v1 release explains the preceding 300-hour S6
and synthetic-sidecar supervised fine-tune, 99-axis score projector, packed
audio preparation, model versions and checkpoint structure. These follow-ups
do not start from a fresh Chatterbox model.
The frozen source/target condition table has 4,000 rows selected from that
300-hour prepared pool, with 100 top-ranked eligible sources for each of 40
emotion axes. The sources are 3–8 seconds; target references are 5–10
seconds and their CAMPPlus cosine to the respective source is below 0.30.
There are 3,357 distinct source UIDs and 477 target UIDs. Each run makes six
deterministic, differently shuffled passes over the same 4,000 pairs.
The same pair table and random seeds are used across the three arms for a
controlled comparison. The prepared pool.parquet, selection.parquet and
four input_ears_*.parquet shards are required for exact reproduction but
are not included in these model repositories; original record locators and
licensed audio must be staged separately. Do not unpack samples into
thousands of loose files.
Rewards and optimizer
All three arms generate 16 on-policy candidates with S3Gen's 10-step Euler
flow sampler for a single source/target condition. Humaneness Ears Medium
at revision 818506970d93c809a3295a0a02c40dee8ff2bfc3 predicts the
audiobox:CE output (AudioBox Content Enjoyment) and 40 emotion scores on
each generated output. The two strongest Ears emotion axes of the source
audio define the emotion fidelity MSE. The third source emotion is omitted.
The speaker-aware arm additionally runs the frozen S3Gen CAMPPlus encoder on
each output and computes cosine to the target-reference CAMPPlus embedding.
Each raw component is independently standardized across all 16 candidates
in the group. The control arms use 0.50*z(CE)+0.50*z(-top2_MSE); the
speaker arm uses 0.40*z(CE)+0.40*z(-top2_MSE)+0.20*z(target_CAMP_cosine).
The weighted score is clipped to [-2.5, 2.5] and centered to zero group
mean. Signed advantages weight the flow-matching velocity loss on the
generated mel portion, plus 0.025 times the squared velocity difference
from the respective frozen v1 step-4,000 policy. This is GROW-inspired
group-relative flow optimization, not token-level GRPO. Ears is a training
reward model, so its scores are not an independent evaluation.
The only trainable parameters are the rank-128 q/k/v/out LoRA and the
previously trained 99-score projection. All other Chatterbox modules remain
frozen. Each arm restarts AdamW (weight_decay=0.01) at the v1 step-4,000
model, warms up over 1,200 of 24,000 groups, peaks at 2e-6, then follows
one cosine decay to 2e-7. The 50/50 remerged arm merges the completed v1
merged_new linear LoRA into its effective base a second time, starts a new
zero-output rank-128 LoRA, and retains the v1 score projector. Four GPUs
process four candidates each, all-gather reward components, and average
gradients before every synchronized optimizer update. The frozen anchor is
the respective starting policy. Checkpoints every 1,000 groups include model
trainables, full AdamW state, code/reward/schedule metadata and an atomic
completion marker.
Jupiter execution and resume
The recorded Jupiter Slurm scripts request one exclusive Booster node, four
GH200 GPUs, four tasks, one GPU per task, project reformo, and 11h50
walltime. An initial 20-group smoke checks identical initial trainables
across ranks, finite rewards and gradients, and nonzero updates. Production
must not begin without the matching smoke PASS marker. Logs, metrics,
checkpoints and canary audio stay in Scratch; metrics use one JSONL and
canaries one TAR per arm. This avoids large inode counts. If a job reaches
the walltime, its dependent fallback can resume from the latest complete
optimizer-bearing checkpoint. The actual run.sbatch paths are Jupiter
specific; change account, storage, environment and paths before reuse.
The repositories contain code/grow_v2/train_ddp.py and
code/grow_v3/train_speaker.py, plus their imported v1/base helpers,
code/grow_followup/infer_followup.py, the Slurm scripts, requirements and
the fixed 99-axis score order. For continuation, provide the same packed
pool and selection order, set CHATTERBOX_RUN_ROOT,
CHATTERBOX_GROW_TRAIN_ROOT, and the arm's CHATTERBOX_GROW_V2_ROOT or
CHATTERBOX_GROW_V3_ROOT as appropriate. The new schedule and objective
must match the checkpoint; changing either is a new experiment, not an
exact resume.
Why three arms, and what is not yet known
The two 50/50 arms test whether continued training of the existing LoRA or merging it into the base and starting a fresh LoRA is better. The 40/40/20 arm tests whether an explicit speaker-identity reward prevents the slight speaker-similarity regression observed in the earlier CE-free v1 pilot. No default recommendation should be made from training reward alone. Held-out, matched-seed evaluation must report a speaker cosine, source emotion fidelity, an independent quality/naturalness metric, content accuracy (e.g. ASR/word error rate), and listening examples. The holdout should ideally be speaker-disjoint, which the previous UID-only split was not guaranteed to be.