I bought a GeForce RTX 5060 Ti with 16 GB mainly to run AI locally. The question was simple: how much of my family assistant Freya could I move from GPT-5.5 to a local LLM using Ollama?
It would be easy to frame the test as local AI versus the cloud, but that would also be misleading. A 12B model on a consumer card will not match a frontier model running in a data centre. The interesting part is finding where the boundary sits. Which tasks can run with local AI, what does it take to make them reliable and when do I still need to pass work to the cloud?
After running Gemma 4 12B through benchmarks, Swedish tool calls, OpenClaw and a family-facing test, the answer is a little clearer. 16 GB is enough for more than a chatbot, but the model is only one part of the system.
The starting point: GPT-5.5 with a small local fallback
At the moment, two agents share the performance available from this machine. It is not a modern machine: an Intel i7, 32 GB of RAM and an RTX 5060 Ti with 16 GB of VRAM. Freya is the family Telegram bot and runs in OpenClaw. Muninn is an experiment built with Hermes Agent for future tasks.
When the test started, both used GPT-5.5 as their primary model. The local fallback model was qwen3:4b, which uses about 2.5 GB of VRAM and generates roughly 98 tokens per second when warm. It supports native tool calling, but it is too small to carry Freya's full personality, long instructions and daily tasks with much headroom.
The new card made it possible for me to test Gemma 4 12B. Gemma 4 also requires a newer runtime, Ollama 0.30.5 or later, so the first step was upgrading Ollama. I chose Ollama's Q4_K_M build, which is 7.6 GB on disk and supports 256K context according to Ollama's model catalogue. Google positions the 12B version for desktops and small servers, with reasoning, function calling and multimodal input in its Gemma 4 overview.
For me, one requirement mattered more than the general quality of daily chat: the model also had to select the right tools in Swedish. If it cannot turn on a Trådfri light, read the bus timetable or choose the right Sonos system, it does not matter how pleasant its answer sounds.
What actually fits in 16 GB
The first result was positive. Gemma 4 12B Q4_K_M ran entirely on the graphics card and used roughly 8.1 to 8.5 GB of VRAM, depending on how Ollama reported the run. That left space for the desktop, the embedding model and other local AI without CPU offload.
Gemma is more than twice as slow as qwen3:4b in raw generation, but 42 to 43 tokens per second still feels fast in a normal chat. The important question was whether the model fit with a useful amount of context.
This also became the first major challenge. The old fallback ran with 4K context and truncated prompts of roughly 28,000 tokens without showing it in the chat. Later I found the same kind of problem in OpenClaw, where Gemma effectively received only about 2,000 tokens even though the agent prompt was much larger. The model appeared to forget instructions, tools and results, but the system had actually cut them away.
The fix was to define the operational limit explicitly on both sides: contextTokens=65536 in OpenClaw and num_ctx=65536 in Ollama.
At 65,536 tokens, the model still sat around 8.1 GB of VRAM in this setup.
The large context capacity was not the problem, but letting the runtime guess this value was.
The first Swedish tool tests looked almost too good
In the first test, Gemma received four Swedish requests with Freya-like tools. It selected the right tool and arguments for three actions, then answered a factual question without making an unnecessary tool call. The result was 4 out of 4.
A larger isolated test with Freya's real instructions and nine tools scored 7 out of 7. The model turned on the right light, selected the right radio station, requested departures from the correct stop and abstained when a question needed no tool.
That was enough to prove that Gemma 4 12B can perform native tool calling in Swedish. It did not, however, prove that it could run a real agent for daily tasks.
Muninn made that distinction clear. With 16 mixed MCP tools, the model lost track of filesystem tools that were present in the prompt and gave up. When I narrowed the surface to ten focused filesystem tools, it completed the same task with two correct tool calls.
My first conclusion was that small models work better with fewer tools. That is only partly true, and Freya showed that the problem is more specific. A smaller number of distinct tools works better than many overlapping tools with similar schemas. Tool ambiguity and the size of the context surrounding the decision matter more than the raw number of buttons the model can press.
Muninn later exposed a second boundary, and this one has nothing to do with tool ambiguity. I gave Gemma a weekly digest job: read a set of known files and summarise the week, an open-ended multi-step task instead of a single action. The model looped through nine generic planning calls without reading a single file and delivered a confused answer that asked what we had been working on. The tools were clear and available, but the task required holding a plan across many steps. That job now runs as a plain script that asks a cloud model instead, so the digest stays grounded in the actual files.
A correct tool call can still produce a dangerous answer
The first time Gemma ran against Freya's full production surface, it made the requested shell call, read the successful result and then continued with unrelated web searches. I switched Freya back to GPT-5.5 immediately to give myself time to think.
The silent truncation to roughly 2,000 tokens turned out to be behind that result. After setting 65,536 explicitly, the model completed the same continuation correctly. The next failure was definitely more interesting.
A bus timetable script was missing its environment variables and returned a clear error. Gemma still answered with plausible departure times that had not come from the tool. More prompting was not a sufficient safety model here.
I replaced general shell access with freya_ops, a narrow tool for a fixed set of household actions.
It only accepts validated lights, Sonos players, stops and habit logs.
The result is structured as success or failure, and a failure ends the run before the model gets another chance to write a plausible continuation.
A live test with an invalid device name confirmed the behaviour: one failed tool call, no continuation from the model.
The same principle now applies to reminders. Gemma understands a natural Swedish request and chooses the text to send, but deterministic code owns time zones, storage, deduplication and delivery. In one test, the timer was created and fired at the right time, but Telegram never received the message. The system had verified the job and the model response, but not that the person had actually received the result.
That changed my definition of a successful action: it is not enough for the agent to say that something is complete. It is not even enough for the job to exist in the database. The result needs to be verified where it reaches the user.
Less authority does not have to mean a worse assistant
I initially gave Gemma a broader tool surface so Freya would feel more like its GPT-backed counterpart. That gave the model more ways to turn a reasonable interpretation into a real operational mistake.
The current split is narrower but more useful:
| Responsibility | Owner |
|---|---|
| Conversation, intent and Swedish phrasing | Gemma 4 12B |
| Lights, Sonos, bus timetables and habit actions | Verified freya_ops calls |
| Reminders, time zones and delivery | Verified freya_reminder |
| Private knowledge and memory | Scoped read and write contracts |
| Maintenance and harder agent work | Explicit GPT-5.6 run |
Gemma has no exec, unrestricted file writes or general cron access.
It can read approved private knowledge, save a scoped memory entry, handle household actions and create reminders through contracts that can be verified.
This is an important distinction from a normal fallback. OpenClaw changes models when a provider errors or times out, not when the local model meets a task that is too difficult or lacks a tool. Difficult tasks therefore need to be delegated explicitly to the cloud model, today GPT-5.6 after an upgrade from GPT-5.5 partway through the test. I cannot just put the cloud first in a fallback list and assume it will automatically take over when Gemma becomes uncertain.
How does it feel in daily use?
For normal Swedish chat, the quality is good.
The tone is warm and natural, with occasional inflection errors such as using köken instead of kök.
It does not feel like machine-translated Swedish, even if the tone can be a little stiff at times.
The delay varies considerably depending on how much work I ask the model to do.
A first run with the full prompt often lands around 10 to 15 seconds.
A warm follow-up in the same conversation that had to remember the word blåbär took 946 milliseconds.
When I configured the model for high thinking with a complex reminder schema, a correct action could take several minutes.
That is why thinking is not only a quality setting.
In the first benchmark, the model spent its entire short output budget on hidden reasoning and left an empty visible response.
With thinking disabled it responded quickly, but it did not automatically become better at difficult tool calls.
More thinking helped sometimes, failed sometimes and added a lot of latency.
For now the family test runs with thinking set to high, where I accept the extra latency in exchange for better odds on the harder tool calls.
Freya now runs Gemma as the primary model in a family-facing test, with GPT-5.6 as the first fallback for provider failures and as an explicit choice for harder work. That is not the same as Gemma replacing the cloud. It means routine and private work can start through a local model.
The same card can also handle voice and games
The RTX 5060 Ti is not used by the language model alone. On the same machine, I run Whisper large-v3 for local speech-to-text and Kokoro for English text-to-speech. The first full voice agent took 15 to 25 seconds before it could start answering. The problem was not Gemma's maximum context, but a general agent prompt of roughly 18,000 tokens, nine tools, thinking and a pipeline that waited for the complete reply before speech synthesis could run.
A separate narrow voice path with a short prompt, thinking disabled and streamed text reached its first output after 1.64 seconds in my tests. That work will become a separate article, but the result reinforces the same lesson: the local system's hardware matters as much as the model size.
The card also sits in a machine used for gaming.
With OLLAMA_KEEP_ALIVE=-1, roughly 8 GB of the GPU's memory remains reserved for as long as the model stays loaded.
That was useful during testing because follow-ups stayed fast.
For normal use, a timeout such as ten minutes makes more sense, allowing VRAM to be released after the model has been idle.
I also confirmed how important it is to check the GPU after a reboot. Ollama looked healthy and answered API calls, but had started entirely on the CPU after a race condition in Distrobox. Speed dropped from roughly 38 to 3 tokens per second while the graphics memory sat empty. A health check that only asks whether the API responds does not prove that the AI system is using the right system resources.
What 16 GB actually bought me
The RTX 5060 Ti did not become a small data centre. Gemma 4 12B does not beat GPT-5.5 on difficult reasoning, broad tool calls or daily maintenance. Giving the model more permissions does not change the final result either.
The card does run a 12B model entirely on the GPU with 65,536 tokens of context, Swedish tool calling and room for the rest of my local stack. It gives Freya a base that is private, always available and independent of provider quotas and outages. It also turns qwen3:4b into the small backup model instead of the ceiling for what the machine can run locally.
My final verdict for now is therefore not local LLM or cloud. Gemma has proved to be a capable primary LLM for routine and private family work during continued testing, while GPT-5.6 remains for the difficult part and as a provider fallback. What makes the setup useful is not only 16 GB of VRAM. It is the combination of enough context, distinct tools, deterministic boundaries, verified results and a deliberate path to a larger model.
The next tests are the newer QAT builds and Gemma 4 26B A4B. The latter is available as a 16 GB QAT file in Ollama, but all weights still need to load and the runtime plus context also require memory. I therefore expect CPU offload rather than a comfortable replacement for the current 12B model on the same card.
That may be a less romantic result than moving the entire cloud into the basement. It is still a system I can actually use in daily life.
