An assistant you talk to has one hard requirement: it has to answer fast enough that you don't feel the gap. Everything else is negotiable. This is how to build one that runs on a single laptop, hits about two seconds a turn, and keeps running after you close the window.
All the numbers here come from an M4 MacBook Air with 16 GB, measured, not estimated, so you can size your own expectations against real hardware.
The budget is the architecture
Start by writing down what you can spend, because every component choice follows from it. Two seconds is roughly the limit before a spoken reply feels broken. Here's where mine goes:
| Stage | Choice | Where | Cost |
|---|---|---|---|
| Voice activity | Silero v5, 512-sample frames @16k | torch, CPU | negligible |
| Endpointing | acoustic + linguistic policy | no model | 300 to 950 ms |
| Transcription | whisper-base.en | MLX / Metal | ~0.3 s |
| Brain | ministral-3b | remote | 0.67 s |
| Speech | Kokoro-82M | torch / MPS | 13.2× realtime |
The thing to notice is that endpointing, which contains no model at all, is the second most expensive stage in the loop. If you're trying to get a voice agent under two seconds, that's where to look first — not at a smaller transcription model. Which model did you expect to be the bottleneck?
Turn detection needs two layers
Your VAD tells you a pause happened. It doesn't tell you the turn ended, and treating those as the same thing builds something that interrupts people constantly.
Put a linguistic layer on top of the acoustic one:
- utterance sounds finished, commit after 300 ms
- it trails off on "and", "so", "because", "um", wait 950 ms for more
- more speech arrives, append it and re-evaluate
Two smaller constants matter as much as those. A redemption window of ~320 ms stops a breath from ending someone's turn. A pre-speech pad of ~220 ms keeps the onset consonant your VAD would otherwise clip — without it, "sixty" arrives as "ixty" often enough to be a real problem.
Plan for the microphone hearing the speaker
Design for this up front, because every component behaves correctly while the system as a whole fails.
Speakers leak into the mic. So: the assistant speaks, the leak reaches the mic, VAD fires on her own voice, barge-in kills playback about 300 ms in, transcription runs on the leak, and the model answers it. Now she's talking again. That loop has no exit, and every individual piece did exactly what you asked.
(tts 1.97s for 2.4s of audio)
(cut off)
(cut off after 0.3s)
you: ONE <- her own voice
vela: One.The fix is to ignore the mic while speaking, plus a tail of around 350 ms for the room to stop ringing. Re-enable barge-in when the user is on headphones, because then the premise is gone. The general lesson is that this class of bug only exists at the orchestrator level, so that's the layer you need to be able to reason about — it's also the layer with no paper to copy.
Match commands before you reach for the model
Opening an app, pausing music, setting a timer, asking the time — match all of these with regular expressions before any model call. It looks primitive next to routing everything through an LLM. Is it? I think so, for three reasons: they answer instantly, they answer identically every time, and nothing can talk them out of it. Anything unmatched falls through to conversation.
The general version: wherever correct behaviour is fully specified, a model converts a guarantee into a probability. Sometimes that's a good trade. Make it deliberately.
Tell the model what it can't do
Small models fail in a specific way that will undermine trust in your assistant faster than anything else: ask about a capability they don't have and they answer anyway. Ask a 3B model what's in your calendar and it'll invent a meeting, with a time.
Put the absences in the prompt explicitly — no calendar, no email, no files, no battery, no location, no browser history, no screen access. That single addition moved my small models from 1 out of 4 to 4 out of 5 on a set of hallucination traps. They probably weren't lying so much as under-instructed: nothing in the context said those capabilities were absent, so the likeliest continuation was the world where they exist.
Memory: read it fresh every turn
SQLite is enough. Four tables carry different lifetimes: episodic (what happened), semantic (what's true), narrative (what it adds up to), and a ledger of predictions written down before they were tested.
The design rule that makes it work is that the situation summary gets read from the database on every turn instead of living in the context window. Ask mine what training step a model grokked at and you get 5600, because that number is a row in the ledger. If it isn't in the database, she says she doesn't know.
Adopt that constraint early. Every companion-style assistant I've seen dies the same way: it has nothing real to report, so it fabricates an inner life, and you feel the lie inside ninety seconds.
Voice cloning, and why it probably won't be fast
If you want the assistant to answer in your own voice, this is the part to understand before you spend a weekend on it.
Profile by stage, and don't skip that step. My clone pipeline took 34.9 seconds to produce 4.9 seconds of audio. The vocoder was 4.8 s of that and is fine. The token generator was 28.4 s — 81.3% of the time, one stage, 5.7× too slow. Every optimisation I'd been considering was aimed somewhere else.
Why that stage is slow is architectural, not a tuning problem. It generates speech autoregressively: a 0.5B transformer emitting discrete speech tokens at 25 per second of audio, one at a time, each conditioned on every token before it. A five second reply is ~125 sequential forward passes that can't be parallelised. Kokoro is fast because it isn't autoregressive — it predicts an entire mel spectrogram in essentially one shot. That difference is the entire 90× gap.
The runtime matters more than the model
Here's the comparison that should change how you pick components on Apple silicon, both measured on the same machine:
Three times the parameters, four and a half times the speed. Per parameter, that's MLX being roughly 13× more efficient.
Why? Because of what autoregressive decoding actually is: memory-bandwidth bound, not compute bound. Every token means reading the entire weight matrix out of memory, and the arithmetic done with it's trivial by comparison. MLX uses unified memory properly and avoids per-kernel launch overhead; MPS does neither well. If your model also runs classifier-free guidance that's two forward passes per token, so double the bandwidth again — dropping the guidance weight to 0.3 bought me about 2×.
So if you're choosing components for on-device inference on a Mac, check what runtime they ship for before you compare model sizes. It's a bigger lever.
Always discard the first run
Practical notes
- Report thermal state. On a fanless machine an hour into work the numbers move meaningfully. Say which run, how warm, how many.
- Report thermal state. On a fanless machine an hour into work, numbers move meaningfully. Say which run, how warm, how many.
- Keep a model warm in a separate process. Mine pays 11.5 s load and 5.3 s warmup once instead of per reply.
- Degrade loudly, not silently. If the clone service is down, the assistant uses the fast voice and logs why, instead of going quiet.
- Cache your common phrases. "I'm here", "Opening Chrome", "Paused", "It's 11:40 p.m." repeat constantly. Pre-synthesise the top fifty in the good voice, hash-key them, fall back to the fast model for novel sentences. Cheapest win available, and I'd do this before porting anything.
If you build one of these, the part that'll surprise you is how little of the work is model selection. It's a latency budget, a turn-taking policy, and a feedback loop between your speaker and your microphone.