How I built a speech recognizer from scratch, layer by layer

Someone says "cat". You compute features and get 400 frames. Your transcript is 3 characters. Your training data contains exactly those two things, the audio and the string, and nothing else. Nobody has told you that frames 51 to 63 are the c.

So how do you compute a loss? A network needs to know the right answer at each output position. You have 400 output positions and 3 labels.

This post is the listening half of a voice agent built without calling anyone's API: features, voice activity detection, endpointing, and the recognizer. I care most about the middle part, because once you see what CTC actually does the rest of the field's architecture choices stop being arbitrary. Numbers throughout are from my own runs.

Two obvious answers, and why both fail

"Just spread the labels evenly." Assign frames 1 to 133 to c, 134 to 266 to a, 267 to 400 to t. This is wrong, and confidently so. The t in "cat" is a 20 ms burst while the a might run 200 ms, so you would be training the model to call silence "a". Worse, the error is different for every utterance, so it never averages out.

"Label the data properly, then." This is what the field actually did for thirty years. A forced aligner, built from a pronunciation dictionary and an HMM, produced frame labels and you trained on those. It works. It also means you need a lexicon for every word, a phoneme set, and a separate alignment model that itself has to be trained. That's the entire pre-2015 pipeline you're trying not to build.

Which leaves the question the whole field is organised around: can you train on the (audio, text) pair directly, never producing an alignment at all?

The move: don't choose an alignment, use all of them

Let the model emit one symbol per frame, and define a rule that squashes any 400-symbol string down to text. Then say: the probability of "cat" is the summed probability of every 400-symbol string that squashes to "cat".

Nobody tells the model where the c is. It's only told that whatever it emitted had better collapse to "cat". It discovers the alignment itself, as a side effect of maximising that sum. That's CTC, and everything else is detail.

Piece one: the collapse rule creates a problem

Start with the obvious rule, merge adjacent repeats, since identical neighbours are just the same sound held longer. That works for "cat" and breaks immediately on any word with a genuine double letter.

So the blank has two jobs. First is the one above: it survives the merge step and breaks a genuine repeat apart, then gets deleted. Second is more obvious once you see it. Your model has 400 frames and maybe three things to say, so most frames it has nothing new to contribute. It's mid-vowel, or in a pause, or unsure. A blank lets it say "nothing new this frame" without being wrong.

Piece two: summing something you can't enumerate

Your loss is the negative log of the total probability of all collapsing paths. For 400 frames over a 29-symbol vocabulary, the number of valid alignments is astronomically large. You couldn't write them down, let alone add them up.

You never have to, though, because alignments share prefixes. Every path that has emitted "c a" by frame 200 is, from frame 201 onward, facing exactly the same remaining problem. How it got there doesn't matter. So you stop tracking individual paths and instead track how much total probability has reached each (frame, position-in-target) cell, sweeping forward one frame at a time.

In code this is one line, torch.nn.CTCLoss. It's worth understanding anyway, because the edge cases bite. If the target is longer than the number of frames there's no valid path at all, and with zero_infinity=True PyTorch substitutes a zero gradient instead of raising. Pick a bad vocabulary and you'll silently train on a fraction of your data while everything appears to work.

Piece three: decoding, where the same sum comes back

At inference you want the most likely text, but the model gives you per-frame symbol probabilities. Greedy decoding takes the argmax at each frame and collapses it, which commits to a single alignment. Probably not what you want: the most likely alignment isn't the most likely text, because a text's probability is spread across many alignments.

Prefix beam search fixes that by scoring text prefixes instead of paths. Its one subtlety is exactly the one you would predict: each prefix has to track two probabilities, one for ending in a blank and one for ending in a real symbol. Without that split you can't distinguish "the model repeated the o" from "the model wrote a second o". It's the "too" versus "to" question again, and it will bite you in the decoder if you skipped it in the loss.

The flaw that defines everything after CTC

Look at how the loss factorises: the model emits each frame's symbol independently, given the audio. Frame 200 doesn't know what frame 199 emitted. Nothing in the model knows that a q is followed by a u, or that "shold" isn't a word.

Here's my Module 2 recognizer, a BiGRU with a CTC head trained on real LibriSpeech:

hyp: he inl san er mianthes tlen kis roint the stieia sord meai wen
ref: but in less than five minutes the staircase groaned beneath an
     extraordinary weight

Notice what kind of wrong this is, because it matters. The rhythm is right. Many vowels are right. It's not acoustically stupid, it's orthographically impossible — no English word looks like "mianthes". That's not a bug to fix with more data. It's structural, and it's what acoustics sound like with no language model attached.

There are exactly two ways out, and they're the next two architectures: bolt a language model onto the beam search from outside, or put one inside the network by letting it condition on what it has already emitted.

That table is more or less the whole map. Whisper is the third row, and that's why its output has correct spelling and punctuation, and also why it hallucinates fluent sentences over silence: a language model with a microphone keeps talking. Two of my pipeline choices exist only to manage that. The endpointer never sends blips under 250 ms, and condition_on_previous_text=False stops hallucination loops compounding across turns.

Building the recognizer that follows from this

Given the map, the design writes itself: put a CTC head and an attention decoder on one shared encoder. CTC's monotonic alignment stops the attention wandering, and the decoder supplies the language model CTC can't have.

Two things there matter when you build it. The ×4 subsampling before the blocks makes self-attention 16× cheaper (it's O(T²)) and costs nothing, so do it first. And two thirds of each block is feed-forward, which means attention is the smallest learned part of a Conformer, the opposite of what the name suggests.

Swapping the 2015 architecture for this one, on identical audio, is the single biggest win available:

Streaming costs three percent

Two changes, and they only work together. Attention gets chunked masking so a frame attends within its chunk plus some history. Your depthwise convolution has to become causal, left-padded only, because otherwise it leaks about 15 frames of future context and the attention mask you carefully wrote is a fiction.

Resample the chunk size every batch during training and one set of weights then serves every latency operating point.

The failure that no metric could see

My model discovered that emitting blank on every frame is a safe answer with a finite loss, and converged into it. 100% of frames decoding to blank, mean P(blank) 0.89, output entropy 0.82 nats out of a possible 6.93. Greedy decoding returned the empty string.

The literature calls this blank-dominated convergence, and it's a property of local minima in CTC, not something unique to my code. What made it expensive is that both instruments I was watching read as "be patient".

Then I nearly learned the wrong lesson from it. My theory was that the subword vocabulary of 1024 was too large, and the fix was restarting at 256. I was about to spend three days on that. Instead I spent an hour reading reference recipes, and found ESPnet uses 5,000 subwords on this same corpus for 8.4% word error. The vocabulary was never the problem.

Had I "fixed" the vocabulary and had it happened to work, I would have learned something false and kept it permanently. That's the real hazard of hyperparameter archaeology by guessing.

The two stages before the recognizer

Briefly, because they're where the easy wins are.

Your VAD is probably a data problem wearing a model's clothes. A frame-level detector needs about 55,000 parameters, small enough that you can stop blaming architecture. Mine reported P(speech) = 1.0 on room noise with nobody talking. Same architecture, same training loop, corrected data, and the same input scores 0.04. Write your evaluation set before you train anything, and make it the specific things you expect to break: digital silence, mains hum, babble, speech recorded too quietly. Mine took ten minutes and then chose which checkpoint shipped, because the best hard-case score was 0.806 at step 2750 while the final step scored 0.699. Last is rarely best.

Endpointing should contain no model. It's a state machine over VAD output plus a hangover timer, and all the content is in the policy. Use two layers: the acoustic one tells you a pause happened, and a linguistic one decides whether the turn ended, because people pause mid-sentence constantly. If the utterance sounds finished, commit after 300 ms; if it trails off on "and", "so", "because", "um", wait 950 ms. A redemption window of 320 ms stops a breath ending a turn, and a 220 ms pre-speech pad keeps the onset consonant the VAD would otherwise clip.

Endpointing also decides your threading model, which surprised me. Barge-in means detecting speech while the agent is talking. VAD is sub-millisecond so it can run inline, but recognition and synthesis take hundreds of milliseconds, and if either runs on the event loop the agent can't hear you while it speaks.

Other things that will cost you a day

• Hold out speakers, not utterances. My held-out set shared 39 of 39 speakers with training, so every number I'd quoted was speaker-dependent and not comparable with anything published.
• Checkpoint averaging has a precondition. It assumes the checkpoints orbit one basin. Mine spanned 0.86 to 0.61, so averaging the last five gave 0.659 against a best single of 0.612. Average the best K by held-out error, and keep it only if it wins.
• Don't regularise a model that hasn't learned the task. With SpecAugment on from step 0, CTC reached 1.06 by step 400. With it off, 0.13. Augmentation is worth more than data below about 1,000 hours, but only after the model has something to regularise.
• Verify the GPU by using it. torch.cuda.is_available() returned True twice on machines where a convolution then failed, once on a CUDA 13 wheel against a CUDA 12 driver and once on cuDNN 9.20 against a 9.10 driver.

What you get at the end

0.607 word error, against Whisper base.en at 0.033. You aren't going to win, and that's not the point of building it.

What you get instead is that "the gap is data" turns out to be false as a general claim, and you can only find that out by moving one thing at a time. For the VAD the gap really was data, and fixing the data fixed the model. For synthesis it's architecture. For recognition it's data, architecture, recipe and inference all at once. After doing this you can tell which one you're looking at, and that transfers to problems that have nothing to do with speech.

The module-by-module code, the training runs and the full failure catalogue are in the repo under docs/.