A few weeks ago I announced that my Malayalam Whisper model beat the previous best open Malayalam ASR model by 24 WER points. That number is still true. But while getting ready to train a v2 of the model, I decided to actually check something I’d never checked before: was my train/test split clean?
It wasn’t. Here’s what I found, how I found it, and what I’m doing about it.
The original claim, and its blind spot
My reported 37.64% WER was measured on the CommonVoice Malayalam test split only. CommonVoice is clean, read, studio-quality speech — a relatively easy condition for ASR. My training corpus actually pulls from six sources (IMaSC, SMC, IndicTTS, OpenSLR 63, CommonVoice, and Shrutilipi), but I’d never evaluated against the other five. That gap had been sitting in the back of my mind as an obvious “should fix eventually.”
What I hadn’t considered was a more basic question: was the official train/test split of my own corpus even trustworthy in the first place?
Building a leakage checker
Before touching any v2 training, I built a small evaluation harness with two things: an eval script (WER/CER with bootstrapped confidence intervals, not just a bare point estimate) and a leakage checker — exact transcript hash matching, plus fuzzy near-duplicate matching for text that isn’t byte-identical but is functionally the same sentence.
I ran it against my own corpus’s official train (86,911 rows) and test (4,828 rows) splits, expecting a clean result. It wasn’t.
What I found
- 41.6% of test rows have an exact-duplicate transcript sitting in train
- 44.4% are flagged once you include fuzzy near-duplicates
Any WER I’d computed against that raw test split would have been measuring memorization on nearly half the data, not real generalization. That’s not a rounding error — it’s the kind of thing that can make a mediocre model look great.
The more interesting finding: leakage isn’t random
I expected leakage, if it existed, to be roughly evenly spread — some copy-paste artifact affecting the whole corpus similarly. It wasn’t. Here’s the kept rate per source after filtering out flagged rows:
| Source | Test rows | Kept after filtering | Kept % |
|---|---|---|---|
| IMaSC | 1,656 | 44 | 2.7% |
| SMC | 87 | 5 | 5.7% |
| OpenSLR 63 | 230 | 32 | 13.9% |
| CommonVoice | 174 | 123 | 70.7% |
| IndicTTS | 149 | 126 | 84.6% |
| Shrutilipi | 2,532 | 2,354 | 93.0% |
IMaSC and SMC are almost entirely duplicated between train and test — only 2.7% and 5.7% of their test rows are genuinely unseen by the model. My best guess at why: both are small corpora with a bounded, fixed set of sentences read by a handful of speakers. A random row-level split looks correctly randomized by row count, but if the same underlying sentence gets read by multiple speakers, that sentence can easily land in both train and test even though the split “looks” fine on paper.
What surprised me more: Shrutilipi — scraped from All India Radio bulletins at document scale, and the source I expected to be messiest — turned out to be the cleanest by this measure, at 93% kept.
What this means for the numbers going forward
After filtering, I have a genuinely clean held-out set of 2,684 rows. But it’s now skewed: 87.6% of it is Shrutilipi. That’s a real limitation worth stating plainly — any WER I report on this clean set is much better powered to tell you how the model does on radio-news-style audio than on read/studio speech, simply because there isn’t much clean data left in the smaller sources to test against.
I also ran a separate benchmark of GGML/GGUF-quantized versions of the model (for whisper.cpp deployment) and got WER numbers in the 51–54% range — notably worse than my CommonVoice figure. I haven’t fully root-caused this yet. It’s likely some mix of a harder/broader sample and the leakage issue above cutting in different directions on a small sample, but I’m not going to claim more precision than I actually have here.
Why I’m publishing this instead of quietly fixing it
It would have been easy to just filter the test set, retrain, and post a new clean number without mentioning any of this. I think that would be a mistake. The whole reason benchmark numbers are useful is that people can trust what they represent — and “trust” has to include the boring, unglamorous parts, like admitting your first split had a hole in it.
The fix is straightforward and already done: I built a filtering script that removes the leaked rows, and that clean set is what I’ll use for evaluating the v2 model going forward. The train split likely has some of the same internal duplication (if a sentence leaked into test, it probably has near-duplicates elsewhere in train too), which is next on my list to check before v2 training finishes.
What’s next
v2 training is running now — more data, a corrected learning rate (a huge thanks to the Vividh-ASR paper for flagging that my original learning rate was likely too conservative), and evaluation against the actual clean held-out set from the start this time. I’ll post the v2 results, source-by-source, once training finishes — including on the sources this post shows I have the least clean data for.
If you’re building your own speech corpus by aggregating multiple sources, my one piece of advice: check your split before you trust your numbers. It took an afternoon and completely changed how I read my own results.