Project note · 2025
Roo Code — Local LLM Evaluation
In early 2025, Roo Code offered an unusually open way to work with AI-assisted coding: the editor integration was open source, the model backend was selectable, and local inference through Ollama was a first-class option. The interesting question was not whether local LLMs could produce code at all. It was whether that promise held up on realistic consumer hardware, with limited VRAM, quantized models and the long contexts that autonomous coding tasks actually create.
The hypothesis: freedom without giving up usefulness
Roo Code was attractive because it reduced two kinds of lock-in at once. It did not insist on one hosted model provider, and it could run against a locally hosted model through Ollama. That meant the possibility of keeping code and prompts on your own machine while also avoiding a per-token meter for every experiment.
The evaluation took that idea into a less flattering but more useful environment: heavily quantized models, scarce VRAM and hardware that an enthusiast could realistically assemble. Cloud-hosted models were already showing how capable coding systems could be. The project asked how close a local setup could get, where it would fail, and which limitations were fundamental rather than simply configuration mistakes.
A deliberately ordinary high-end machine
The test machine was built largely from pre-owned parts collected over time: a previous-generation AMD Ryzen 7, 128 GB of DDR4 memory, 2 TB of NVMe storage, an RTX 3090 Ti and an additional RTX 3060. Together the GPUs provided 36 GB of VRAM, but split across two devices and unequal PCIe links.
That was important to the experiment. This was not a benchmark on a pristine workstation with several current-generation datacenter GPUs. It was closer to yesterday’s top consumer setup: still powerful, but compromised in exactly the ways real machines are compromised. The project was about applicability, not leaderboard performance.
Why auto-coding stresses local models
Roo Code does more than autocomplete a function. For a substantial refactoring task it has to build an understanding of an existing codebase, request files, call tools, reason about changes, apply them, inspect results and continue iterating. Each round adds more history to the next model call.
That makes context management the central problem. A task may begin small, then grow into hundreds of kilobytes of prompts and tool results. Early misunderstandings become part of the history. Larger context consumes more memory. When the model, context window, system prompt and available hardware do not fit one another, failures can amplify instead of correcting themselves.
Quantization: making the model fit without making it useless
Local inference forces a direct trade-off between model size, context size and numerical precision. Quantization makes larger models practical by storing their weights at lower precision, but pushing it too far can reduce the model’s reliability exactly where an auto-coder needs precision.
The report found a workable region around Q8 quantization on 14-billion-parameter Qwen-derived models with the available 36 GB of VRAM. Lower precision could save memory, but the cost showed up as weaker reasoning, failed tool interactions and repetitive correction loops. The old-couple illustration above came from that part of the report: one side cannot quite hear, the other cannot quite see, and the conversation deteriorates.
Context was as important as model size
One of the more useful findings was that simply maximizing the configured context window was not a solution. A model may advertise a very large context, yet behave worse when pushed beyond the range where it was trained or where the available VRAM can support it efficiently.
The practical work therefore became a balancing exercise: choose a model trained for coding, keep quantization as high as the hardware allows, leave enough VRAM for context, check that Ollama is actually using the GPUs efficiently, and reduce unnecessary prompt overhead. Roo Code’s configurable system prompt and modes were useful precisely because they allowed the tool itself to be adapted to the available model instead of assuming that one setup fits everything.
A snapshot from before today’s agent stack settled
The repository dates from February 2025, with the first commits appearing at the end of that month. At the time, the report still described products such as Roo Code as auto-coders and treated MCP as an emerging interoperability protocol rather than the default plumbing it would later become in many AI workflows.
Roo Code was already doing several things that would later be described more broadly as agentic work: maintaining task context, requesting tool calls, executing commands through VS Code, inspecting results and iterating toward completion. But the ecosystem around those ideas was much less settled. That makes the project useful now as a snapshot of the transition from “LLM in an editor” toward tool-using development systems.
The result: yes, but with boundaries
The conclusion was deliberately mixed. Local Roo Code worked. With careful model choice, tuning and appropriately scoped tasks, it could produce genuinely useful results without sending the codebase to a hosted model provider. But it did not match the speed, reliability or broad capability of the strongest cloud models of the time.
The useful lesson was not “local beats cloud”. It was that the freedom Roo Code promised was real, but freedom moved responsibility back to the operator. Model selection, quantization, context sizing, hardware utilization and task decomposition all became part of the engineering problem.
That was also the satisfying part. The setup was assembled from ordinary enthusiast hardware, and much of the work consisted of understanding why something failed rather than merely switching providers. The project turned a product claim — privacy and backend freedom — into something measurable on a machine that someone could plausibly own.
Full report and configurations
The complete evaluation, including the detailed discussion of Roo Code’s context flow, model and quantization experiments, Ollama configuration, hardware observations and example model files, remains available on GitHub.