If you've built a speech recognizer you've met the alignment problem: 400 audio frames, three characters, and nothing saying which frames belong to which character. CTC handles it by refusing to choose, summing over every possible alignment.
Run the same problem backwards and that escape route disappears.
| Recognition | Synthesis | |
|---|---|---|
| you're given | 400 frames, 3 characters | 19 characters |
| you produce | the text | 250 frames, and you decide it's 250 |
| the alignment is | hidden in the audio, waiting | not there at all, you invent it |
| sum over all of them? | yes, that's CTC | no, you emit one sequence |
That last row is the difficulty. CTC can stay agnostic because it only ever scores a target you already hold. A synthesiser produces the output, so at every step it has to commit: is the "o" in "go" finished, or does it hold for another 40 ms? Get that wrong repeatedly and you don't get a slightly wrong transcript. You get speech that stutters, skips a syllable, or never stops.
Text to mel is a sequence problem: durations, prosody, pronunciation. Mel to waveform is a signal problem: inverting a transform that threw information away. Modern systems fuse them for quality, but you can't see why fusing helps until you've felt the seam.
The acoustic model, and the four tricks it needs
Tacotron's answer is to generate mel frames one step at a time, using attention over the text at each step to answer "which character am I on?". The attention is the alignment, built as it goes instead of discovered after the fact.
Four things make that trainable. You probably need all four, and I know because I removed one.
| Trick | What it does | Without it |
|---|---|---|
| Teacher forcing | feed the true previous frame during training | early training compounds its own garbage |
| Prenet dropout, on at inference too | stops the decoder copying the previous frame forward | fluent, confident mumble, uncorrelated with the text |
| Reduction factor r=2 | predict 2 frames per step | attention takes far longer to form |
| Location-sensitive attention | previous alignment through a conv into the scorer | stutters and skips characters |
Number two deserves a paragraph, because it looks like a bug you'd fix in review. Leaving dropout active at inference is deliberate. The previous-frame path is so informative that the decoder will happily copy it forward and ignore the text entirely, and what comes out is speech that sounds completely fluent and has nothing to do with what you asked for. It's a genuinely unsettling thing to listen to.
Watch the alignment map, not the loss
This is the most useful thing I can tell you about training a synthesiser.
Mel L1 loss falls early just by predicting blurry average speech. Mine went 0.81 to 0.24, smoothly, over 3,000 steps. That's the shape of curve you screenshot and feel good about. The model had learned essentially nothing: the free-running attention map was a smear with no diagonal in it, and generation cut out at 2.2 seconds on an 82-character sentence.
A diagonal forming means it's working. A smear means it's not, and maybe never will. Your loss can't distinguish those two states, so plot the alignment every few hundred steps and actually look at it.
If the diagonal won't form, the fix usually isn't more data. Guided
attention adds a penalty mean(A · W) where
W[t,n] = 1 − exp(−(n/N − t/T)² / 2g²), taxing attention mass far
from the diagonal and leaving mass near it free. It's a prior saying "time
moves forward", not a constraint, and with enough data you would drop it. With
19 training utterances it took alignment from never forming to forming in
hundreds of steps.
Judge it with the recognizer you already built
Mel L1 says nothing about intelligibility, and I don't have a listening panel. A generative model can hit a good training loss while being unusable, because the loss measures frame-level regression and not the property you care about. So transcribe the synthesiser's own output and measure character error against the text it was asked to say.
Build that judge early. Free-running quality oscillates between checkpoints while the training loss sits flat: mine swung between 0.96 and 0.61 character error with three separate regressions, and the judge picked step 8,000 of 10,000. Production TTS teams do exactly this, ASR-based intelligibility evals alongside human ratings. Last isn't best for generative models.
Two clever ideas that measured worse
When attention stalls at inference, the obvious fix is restricting it to a window around wherever it was last. I implemented that.
Your model trained on a full softmax over the whole text, so renormalising inside a window at decode time hands it context vectors of a kind it has never seen. Exposure-bias fixes belong in training, not decoding.
What I shipped instead were stop-head guards: require two consecutive stop votes and about five frames per input character before stopping. That matters more than it sounds, because padded batches make "done" frames outnumber "still talking" frames for short utterances, and one false positive truncates mid-word.
My second bad idea was hardware. That decoder is a Python loop over timesteps, each doing small tensor operations, and on Apple's MPS backend it ran 13× slower than on CPU. Per-dispatch overhead dominates when the ops are tiny and strictly sequential. Accelerators buy throughput, not latency on small sequential work, and a voice agent is mostly small sequential work.
What the mel threw away
Your vocoder has to reinvent two things the feature pipeline destroyed.
Frequency detail. 257 FFT bins were pooled into 80 triangles. Least-squares undo: multiply by the filterbank's pseudo-inverse and clamp the negatives.
Phase. Your power spectrum kept magnitudes and threw away every phase angle, but an inverse FFT needs complex numbers. Griffin-Lim (1984) exploits the redundancy of overlapping frames, 400-sample frames at a 160 hop giving 60% overlap: iterate inverse STFT, forward STFT, keep your magnitude, adopt the re-measured phase, and the phases get squeezed toward mutual consistency. Measured here, spectral convergence goes 0.235 to 0.112 over 60 iterations.
Self-consistent isn't correct, though. Many phase fields are consistent with the same magnitude spectrogram, and the one Griffin-Lim lands on generally is not the one a speaker produced. That mismatch is the phasey, metallic, slightly underwater quality of everything it makes.
Choosing a vocoder
What the spectral family noticed is that the inverse STFT already does the upsampling, exactly and for free, and has since 1977. So predict magnitude and phase per frame, run the exact inverse, and stop paying a network to rediscover arithmetic.
Match its STFT grid to the one your features already use, n_fft 512, window 400, hop 160 in my case, and the mels your acoustic model emits drop straight in with no retraining. That agreement isn't a detail: reusing a 512/256 overlap-add shell for something trained at 512/128 looked harmless and took a 19 dB signal-to-noise ratio down to 6.
One implementation note, since this is the detail everyone gets wrong first. If you hand-build the inverse STFT as overlap-add you need the synthesis-window normalisation, dividing by the sum of squared windows. Skip it and you get amplitude ripple at the frame rate that sounds like tremolo.
Pick a metric that can see what you care about
I scored my neural vocoder against Griffin-Lim on multi-resolution STFT distance. Griffin-Lim won.
Both numbers were correct. Only one was measuring whether the output sounds like a person.
Where your model sits in the lineage
Once you've built Tacotron and felt its two specific pain points, every generation of TTS research becomes legible as "which of these does this kill?"
| Era | Model | The move |
|---|---|---|
| 2017 | Tacotron 2 | what you just built, scaled: LSTM, location attention, WaveNet vocoder |
| 2019 | FastSpeech 1/2 | kill autoregression. Predict every phoneme's duration explicitly and generate all frames in parallel: no stop token, no stalls, 100× faster, at the cost of needing alignments from a teacher |
| 2021 | VITS | fuse the stages, text to waveform end to end, learns its own alignment and removes the mel seam entirely |
| 2023 | StyleTTS 2 | style diffusion plus adversarial training with a speech-LM judge |
| 2024 | Kokoro-82M | that recipe distilled to 82M params, CPU-realtime, near-SOTA naturalness at hobbyist cost |
| frontier | audio LMs | treat speech as codec tokens and let an LLM generate them: zero-shot cloning, expressive, heavier, less controllable |
Trace your own two pain points through that table. Stop-token and stall problems die with explicit durations at FastSpeech. The Griffin-Lim robotic residue dies with learned vocoders and end-to-end training. That's exactly why building Tacotron first is the right way to learn this field, even though you would never ship it.
Streaming, briefly
Both my engines yield audio per sentence, so time-to-first-audio is one sentence's synthesis, later sentences render while earlier ones play, and barge-in abandons the generator wasting at most one sentence. Production APIs stream at finer grain with chunked vocoding inside a sentence, around 100 ms of audio at a time. Per-sentence is probably enough to feel responsive, and it's a lot simpler.
If you want it in your own voice
Cloning is mostly a data-collection problem, and it's the one step here you can't parallelise, buy or download. An hour of speech is around 900 takes. At ten seconds of friction per take that's two and a half hours of pure overhead, and probably how most voice datasets end up abandoned.
So the recording tool is the artifact worth building, designed to remove decisions:
- Hands free. Press return, read the sentence, stop talking, and your VAD ends the take. Your hands never leave the return key for 900 sentences.
- Check every take before it counts. Clipping, noise floor, silence padding, and whether a transcription matches the prompt. Catching a bad take costs five seconds now and a day after training.
- Resumable at any instant, because nobody records 900 sentences in one sitting.
- Balance the corpus by greedy n-gram coverage when choosing prompts, so an hour of speech covers the sounds you need instead of the sentences you happened to pick.
Verifying a voice is a voice
If you go further into speaker embeddings, the design decision that matters is that the training objective and the reported metric are deliberately different tasks. You train a classifier over known speakers, but you report equal error rate on speakers the model has never heard, scored by cosine distance on a fixed trial list. Training accuracy hitting 100% only means it memorised 32 voices.
Then check your confidence interval before believing any A/B. A held-out split of 8 speakers gives an equal error rate with a bootstrap interval several points wide, wider than most of the differences you would want to test, like angular margin against plain softmax. Print the interval, the median across evaluations and the best checkpoint separately, so they can't be quietly confused.
On a split that small the honest answer to "does the angular margin help?" is that the experiment can't tell you. Knowing which of your results are real is the part that transfers to the next project.