ajju3513/whisper-medium-ala-hinglish-noise-robust

Whisper-Medium + Adaptive Layer Attention — Noise-Robust Hinglish ASR

A noise-robust Hinglish (Hindi–English code-switched) speech recognition model for adult and children's speech. It adds an Adaptive Layer Attention (ALA) fusion head to Whisper-medium and trains it with parallel clean/noisy knowledge distillation: a frozen teacher hears clean audio, the student hears the noisy version of the same utterance, and the student is pulled toward the teacher across four different signals.

Relative to the fine-tuned baseline this recovers a substantial amount of the accuracy lost to additive noise, at the cost of ~5.2 M extra parameters (0.7% of the model).

Why layer fusion

Whisper's decoder only ever sees the final encoder layer. But lower encoder layers carry acoustic and phonetic detail that survives noise better than the top layer's semantic abstraction. ALA lets the decoder attend over both.

The 24 encoder layers are split into two blocks (1–12, 13–23) and mean-pooled into two summary representations. A multi-head attention layer uses the final layer as its query and the two block summaries as keys/values, then adds the result back as a residual. The output projection is zero-initialised, so at step 0 the module is an exact identity over base Whisper — training only ever moves away from a known-good starting point, which is what makes it stable to bolt onto an already fine-tuned model.

Training objective

Distillation from a clean-audio teacher into a noisy-audio student, over four terms:

Term	Weight	What it aligns
Cross-entropy	1.0	Ground-truth transcript
KL divergence (T=2)	0.5	Output token distribution
Encoder cosine distance	0.5	Fused encoder representations
Decoder hidden MSE	0.5	Final decoder hidden states
Cross-attention MSE	0.2	Encoder–decoder alignment, upper layers weighted 2×

Utterances that are pure noise with no speech are mapped to an empty target and excluded from the distillation terms via a mask, so the model learns to emit nothing rather than hallucinate text on non-speech input — a common Whisper failure mode.

Setup: batch 2 × grad-accum 2, AdamW, LR 1e-5, 100 warmup steps with linear decay, fp16 AMP, grad-norm clip 1.0, 15 epochs. Base model is openai/whisper-medium fine-tuned on the clean Hinglish corpus first, then given the ALA head.

Results

Word Error Rate (%), lower is better. This release is epoch 7, selected on combined noisy WER — the best of all 15 epochs under both decode languages.

Test condition	Adult	Children	Combined
Noisy, language="hi"	29.90	24.99	28.21
Noisy, language="en"	30.22	24.87	28.38
Clean, language="hi"	11.46	13.15	12.04
Clean, language="en"	11.37	13.00	11.93

Children's speech is consistently harder on clean audio (13.00% vs 11.37%) but easier under noise (24.87% vs 30.22%), which suggests the adult recordings in this corpus carry more channel variability than the children's.

On clean audio, epoch 8 is marginally better (11.88% combined vs 11.93%). Epoch 7 was released because noise robustness is the objective this model is built for.

Usage

The ALA head is a custom module, so trust_remote_code=True is required.

import torch, torchaudio
from transformers import AutoModelForSpeechSeq2Seq, WhisperProcessor

model_id = "ajju3513/whisper-medium-ala-hinglish-noise-robust"
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, trust_remote_code=True, attn_implementation="eager"
).eval()
processor = WhisperProcessor.from_pretrained(model_id)

waveform, sr = torchaudio.load("audio.wav")
if sr != 16000:
    waveform = torchaudio.functional.resample(waveform, sr, 16000)
waveform = waveform.mean(dim=0)  # to mono

features = processor(
    waveform.numpy(), sampling_rate=16000, return_tensors="pt"
).input_features

with torch.no_grad():
    ids = model.generate(features, max_length=225, task="transcribe", language="hi")

print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Use language="hi" for Devanagari output and language="en" for romanised output. Both were evaluated; pick whichever matches your downstream text pipeline.

pipeline("automatic-speech-recognition") is not supported, because the fused encoder pass has to run before generate. Use the explicit call above.

Limitations

• Not a general-purpose ASR model. Fine-tuned on a specific Hinglish corpus with a specific microphone and noise profile. Expect degradation on other domains.
• Short-form only. Trained and evaluated on utterances under 30 s. Whisper's long-form chunking path is untested here.
• Noise conditions are synthetic. Noise was mixed in offline; real-world reverberant or overlapping-speaker audio is out of distribution.
• Absolute WER is high (~28% noisy) because code-switched speech under additive noise is genuinely hard. Read these numbers against the baseline, not against clean English benchmarks.
• Evaluated in fp32. fp16 inference should be close but was not measured.

Citation

@misc{velpula2026whisperala,
  author = {Velpula, Ajay Kumar},
  title  = {Adaptive Layer Attention for Noise-Robust Hinglish Speech Recognition},
  year   = {2026},
  url    = {https://huggingface.co/ajju3513/whisper-medium-ala-hinglish-noise-robust}
}

Built on openai/whisper-medium (Radford et al., 2022).