My goal was never voice control on its own. It was an assistant that answers quickly enough to feel like a conversation, so nobody has to stand in the kitchen waiting to pull up a recipe while watching a spinner for twenty seconds.
This is the second part of my series about the same machine. The first was about how far 16 GB of VRAM gets you when a local model is meant to replace cloud AI as a family assistant. This one is about what happened when the same model had to speak.
Once Gemma 4 12B had become the family assistant's local brain, the whole chain was local first. Speech to text on the graphics card, the model on the same card, text to speech on the processor. All of it in a local setup.
The first real test came back at between 15 and 25 seconds before Freya started talking. That was not the result I had expected.
Getting it to listen was the easy part
Local speech to text was the piece that worked immediately.
faster-whisper large-v3 on the RTX 5060 Ti transcribed 7.3 seconds of English speech in 0.40 seconds, roughly eighteen times real time.
Swedish landed at around fifteen times.
The model takes about 8 seconds to load and then holds around 3 GB of VRAM.
One detail that mattered later: when Swedish transcriptions occasionally misspelled the children's names, the fault was not Whisper's. It was the source audio, specifically the TTS voice that had read the test sentence, which pronounced the name wrong.
So listening was never the problem, and there was nothing wrong with the voice either. Piper on the CPU runs about twenty-four times faster than real time.
The diagnosis: it was not the model
The first thing you want to do when an LLM app feels slow is to switch to a smaller, faster model. After reading the logs, that turned out not to be the right way to go.
Every voice segment went into the voice agent. That meant a compiled system prompt of 52,719 characters, roughly 18,400 input tokens, nine exposed tools and thinking set to a high level. The client also waited for the whole answer and synthesised a single complete WAV file at the end. The model's own completion time was between 9 and 36 seconds.
The experiment that settled it was simple.
The same warm gemma4:12b, still with the production allocation of 65,536 tokens, answered in 0.85 seconds when it was given
- a 45 token prompt,
- thinking turned off,
- no tools,
- and a cap of 80 output tokens.
In this test, capacity was not the problem. The problem was what I was actually sending into it. A large context allocation is not the same thing as a large prompt, and the difference is worth keeping straight: changing the allocation forces a model reload that costs seconds by itself.
If you recognise the symptom, fast parts adding up to a sluggish whole, this test is worth more than the rest of this article. It takes a minute and it decides whether the model is a suspect at all.
curl -s http://127.0.0.1:11434/api/chat -d '{
"model": "gemma4:12b",
"think": false,
"stream": false,
"options": { "num_ctx": 65536, "num_predict": 80 },
"messages": [
{ "role": "system", "content": "You are a voice assistant. Answer in one short sentence." },
{ "role": "user", "content": "What is the capital of France?" }
]
}' | jq '{answer: .message.content, prompt_eval_count, eval_count}'Run it against the same warm model your app uses, with the same num_ctx.
If you get an answer in about a second the model is fine and the problem is in what you send or in how you wait for the reply.
I ran exactly this again while writing the article, three weeks after the original measurement: 42 prompt tokens, 12 generated, a little over a second including process start.
prompt_eval_count is the number that matters here. Put it next to what your app actually sends.
For me the difference was 42 against roughly 18,400, and that is where things started working the way I wanted.
The fast path
The voice portal got its own route that never goes through the agent bootstrap:
- straight to Ollama's
/api/chat, with no automatic cloud fallback, - a compact identity prompt instead of the whole personality,
think: false,- the last four rounds in memory and nothing more,
- a 96 token cap and an instruction to stay under about 45 spoken words,
- the same
num_ctx=65536as Telegram, specifically to avoid a reload when I switch platform.
So what made the biggest difference to the experience? Text is now streamed from the model and split at natural clause and sentence boundaries of at least twelve characters. Each clause is synthesised on its own and sent as NDJSON events. The browser queues them and plays the first clause while the rest is still being generated.
Once the fast path worked, the result was 0.40 seconds for speech to text, 0.60 seconds to Gemma's first token, 1.64 seconds to the first playable audio and 2.41 seconds to a finished two-clause answer. The Swedish stack that runs today sits at 1.77 seconds for an ordinary round and 527 milliseconds for a deterministic help question that never goes near the model.
Let us be honest about what those numbers are. They are synthetic end-to-end measurements with recorded audio, taken after the service has warmed Whisper and speech synthesis at startup, and the 1.64-second figure is also the first round after a restart. They prove the server's path, but nothing about how it sounds in a kitchen with the dishwasher running. The difference between those two things is what the section on daily use, further down, is about.
Within what they do measure the result is unambiguous: the same graphics card and the same weights, roughly an order of magnitude difference in experience, and all of that difference came from orchestration and streaming.
What a lean prompt costs
The next day I tried asking what day it was. Freya answered "Monday 20 May" and repeated it with the same confidence even when I questioned it.
The fast path had removed the agent's runtime context without replacing the part of it that was actually true. With only four rounds of history, the model treated its own hallucination as evidence.
The fix has two layers. Every ordinary voice input now gets a small authoritative system line with the correct time in Europe/Stockholm. Direct questions about the day, the date or the time never reach the model at all. Their answers come from the host clock and then enter the same streaming TTS path.
The lesson is not that lean prompts are dangerous. A lean prompt still has to contain the small pieces of authoritative runtime knowledge the application promises. Limiting what a model gets to read is therefore not enough, it also has to be fed deliberately with what it cannot possibly know. For exact facts, deterministic code is both faster and more reliable than inference.
Three routes, no LLM router
At first the fast portal could only talk. It could not turn on a lamp, start a timer or read my notes. It is tempting to solve that with a router model that classifies every sentence. That would have reintroduced exactly the latency I had just worked to get rid of.
Instead a cheap phrase match runs before the model on every round, and it grew into three routes:
- Commands are verified actions. Lights and memory writes never go through the model. They run the same fail-closed dispatchers as the Telegram tools, through a CLI entry point on the same plugin. One security layer, two surfaces. A neutral acknowledgement is spoken first, and success is only spoken once a read-back has verified that it actually happened.
- Questions are grounded retrieval. If I say "check my notes about X" a deterministic search runs and feeds the result in as a clearly labelled reference in the same lean streaming round. The model never picks a tool. It only summarises what code has already fetched. An empty result is spoken deterministically, without the model being asked at all.
- Everything else is chat, and then the unchanged fast path is used.
The third one is the most important. Never asking the model to answer when there is nothing to answer is the same lesson as the clock, generalised.
It was not just me
At around the same time a post was going around on Reddit that retold a case study where an agent went from about 90 seconds to about 4 per answer without changing model, with the argument that the model is the last place you should look. The four culprits were: retrieval doing heavy work inline, context that had grown too large, tool calls stacking up, and nothing in cache.
All four map almost one to one onto what I had arrived at independently. The morning summary is generated in advance by a timer instead of during the conversation. The session resets daily, max tokens is locked to the same 65,536 that Ollama actually runs, and the six tool policy exists precisely because every enabled tool's schema otherwise rides along in every call. The deterministic routes preempt inference entirely. And the model stays resident in VRAM, which is why a warm follow-up in Telegram can come back in 946 milliseconds.
Even their diagnostic aside, checking the processor column in ollama ps for silent GPU and CPU splits, is in my own restart checklist word for word, ever since a boot race in July made the model start entirely on the processor.
It looked healthy by every metric except the latency I actually experienced.
Someone else arriving at the same conclusions is worth more than my own numbers. It suggests the pattern is general and not a quirk of my setup.
In two places this story goes further than that case study.
The first is about capacity. The advice "do not change model to go faster" is true but, I find, incomplete. Gemma answers in under a second when it is warm, and it still could not handle a multi-step task that required reading several files and synthesising them: nine generic tool calls, zero actual reads, and a confused non-answer. No amount of latency work would have helped there. Routing therefore needs a capacity floor per task, which is a different axis from speed, and it is why the hard synthesis is now pinned to a cloud model while the conversation is not.
The second is that the latency work also turned into security work. Fail-closed dispatchers, verified writes with read-back and deduplication of retries were all introduced to cut round trips, but they are also what makes it defensible to let a local 12B touch real hardware in this setup. It is unusual for performance and safety to pull in the same direction.
One objection to the original post: "it is all in your logs" assumes that all tool calls are logged. Mine are deliberately redacted, so it is the session data that can actually be followed.
Two times language got in the way
The first time was the microphone test of the first command.
snus ut came back from the English Whisper model as "snusui" and matched nothing.
A two-word spoken command was enough to reveal that function words have to live in the same language as the speech recognition.
The temporary fix was to recommend "snus break", two English words that English speech to text hears every time.
The second case was more challenging. The entire warm TTS ecosystem is English. The choice was between Swedish and manageable quality (Piper), English and genuinely warm quality (Kokoro), or Swedish and warm high quality in the cloud for a fee (ElevenLabs). I first chose English locally, and then scrapped the cloud option entirely.
A real family test then made the challenge concrete in a way no demo does: a voice can sound as lovely as you like and still fail if it says a child's name wrong. The whole fast voice surface was moved back to Swedish with local Piper Lisa.
The fix was neither retraining nor a pronunciation lexicon. The name has an alternative spelling that is just as common, and that spelling happens to come out pronounced correctly. It now lives only in the text that is sent to speech synthesis. The page, the transcript, the memory and the logs still show the real spelling, so the speaker is the only thing that hears the difference.
The difference was one letter. Before it won I tested five variants blind, including raw phonemes in both Swedish and English styles, with the same voice and the same model. The simplest one sounded best, which I would not have guessed.
The name is not in this article. The children did not ask to be examples in a technical text, and further down I argue for the whole build on the grounds that their voices never leave the house.
That language switch was not a prompt change. It was a pipeline migration. Dates, command grammar, lights, Sonos, memory retrieval, deterministic error messages, the help text, the tests and the interface all contained English. A multilingual model does not make the deterministic system around it multilingual.
The interface became honest by accident
The portal has an orb on a Nordic night sky which is also the button that activates the speech function. It reacts to real audio and not to a timer. Playback goes through WebAudio with an analyser node, so the orb pulses with the actual amplitude of Freya's voice, and a second analyser node drives it from the microphone level while the button is held down.
That honesty fell out of the latency architecture for free. Because the server streams every clause as text and audio together, the transcript can mark exactly the clause being spoken, in time. A batch pipeline returning a finished WAV could only have faked that with an animation loop.
So the streaming made the assistant faster and let the interface show what the assistant is actually doing right now.
Why not just OpenAI Realtime?
The reasonable objection to this whole build is that the cloud has already solved the problem. A realtime API with speech in and speech out would probably feel better than my solution for basic conversation: it could be interrupted mid-sentence, with smoother emphasis and no seams between words. There is no point pretending my solution is better.
The reasons to do it locally anyway are not primarily technical challenges.
For me it is more about the children's voices never leaving the home. There is also no per-conversation cost, and such an API is billed separately from the subscription I already pay for. The assistant works if the internet is down. And no vendor's quota or uptime stands between the family and being able to control the lights in the house.
On top of that: the functions that carry the actual value, that is lights, timers and memory, have to touch things in the house. They would have been local regardless of which model carried the conversation. The cloud would only have replaced the part that is already fast.
The comparison remains a deliberately deferred decision, not a blind faith in local solutions. The right moment to make it is when the local baseline has been approved by the family, not before.
What daily use found
The caveat from the fast path section still applies: the numbers describe the server, not the room. They say nothing about how clause seams sound across a longer answer, how autoplay behaves on a phone, how the microphone copes with a kitchen where somebody is emptying the dishwasher, or whether four rounds of continuity is enough for a real conversation.
Since then the room has answered part of that. I start and stop the radio by voice from my phone every day, and it was that habit, not a test case, that found the one real flaw: Freya could say she had put the radio on without the radio having started to play.
The fault was neither in the speech recognition nor in the model, because the command path does not go through either of them. It was in what I had counted as success. Home Assistant answers OK to the command being received, and that is a different claim from the music playing. That exact difference is what the commands route promises to keep track of, and for Sonos it did not.
The correction is that the acknowledgement now reads back the observed state instead of the response to the call. After a command the speaker's state is polled until it actually says playing, for up to five seconds, and if it never does then Freya says the command was received but that the new state cannot be confirmed. That is a duller message, but it is one you can rely on. The behaviour is now covered by a test, so the same failure cannot pass unnoticed again.
The more interesting half of the same fault ran in the other direction.
A stop must not require the state to become stopped, because Home Assistant's Sonos integration reports PAUSED after media_stop on the kitchen player.
A naive verification would have started denying something it had just done correctly.
False success and false failure have the same root: a verification is worth nothing until you know what the true state actually looks like.
So what is left to do? The remaining work is mid-answer interruption, an explicit handover for hard tasks that is allowed to answer asynchronously, a wake word, integration with the kitchen tablet and a full acceptance test of the Swedish surface, not just of the routes that happen to be part of daily life.
Latency, capacity and safety
"All local" and "fast" are two different claims. Time to first audio matters more than time to the last token, and input tokens and hidden reasoning can dominate completely over the generation itself on the GPU. The most capable agent is therefore usually the wrong endpoint for a conversation, and what worked here is not two models on every round but a fast path with deliberate handovers to the stronger agent when it is actually needed.
Latency is still only one axis, and the one that has had the most attention. The other is that a model can be as fast as you like and still not be up to the task, and no orchestration repairs that. Knowing where that floor sits for each kind of work has been harder, and more useful, than knowing how many milliseconds the pipeline costs.
What I had not counted on is that nearly everything I did to bring the time down also made the system safer: fewer round trips, less context, deterministic code where it was possible, and verification before assertion. It is rare for performance and safety to pull in the same direction, so I will take it when it happens.
