Local LLM coding agents
Last updated:
assess
LLM coding agents are powerful tools that can execute coding tasks semi-autonomously. As we discuss in our Claude Code Tech Radar entry, they can help you navigate unfamiliar codebases, draft implementation, and perform checks. However, they come at a cost—both explicit (subscription fee) and hidden (privacy, security, externalities, access issues). Some of these costs and concerns can be addressed by using local LLMs instead of cloud-based SaaS like Claude Code. We specifically evaluate a combination of the Pi agent harness and Qwen open-weight models, served on a local llama.cpp server.
Why so many parts to replace one tool?
A Large Language Model (LLM) is a structured collection of numbers, or weights, which act as parameters in a big maths equation called the model architecture. The model is called open-weight if its weights and architecture are publicly available and free to use. We have tried a number of models, but settled on testing Qwen 2.5, 3, and 3.6 families1.
To produce any usable output with any model, we need software that actually processes the input (turns user prompts into numbers), runs the computation, and structures the output (produces text or other data out of numbers). This is the role of an LLM server platform like llama.cpp or LM Studio. While it seems like the most trivial part of the setup, in our experience, it's the most important one, as the server is responsible for all the hardware-specific optimizations, making the difference between a usable setup and a hot mess.
To then use the model as a coding agent, we need what's called an agent harness. This is the UI where we developers actually type in the input. The harness communicates with the LLM server, provides context, loops the model output back into its input, and interprets the output to run operations like reading and writing files, using CLI tools, and accessing the Web. The Claude Code CLI can be used with local models as well, but it adds a lot of features which might overwhelm your compute resources. Therefore, we decided to try the Pi agent for local LLMs, which comes with a minimal set of functionalities (the inference loop and tool calling) and a library of extensions to install as needed.
Claude Code and other SaaS come with everything packaged into one product. Using local coding agents requires us to set it all up ourselves, but also allows us to mix and match these components according to our needs.
The benefits
Privacy is the most cited reason some choose to run LLMs locally. Cloud-based models require all data necessary to compute the output to be sent via the network to the model provider. Depending on the specific license, this data may be stored and used to further train the models. Moreover, some providers hire human reviewers that read real user chats to improve their models. In situations where sensitive data is being processed, local models ensure that no data ever leaves your computer (except if the model inference is hijacked—more on that later).
Control over the whole process. The choice of available open-weight models is vast, and while none of them are as powerful as the closed ones like the Claude Opus, there are many specialized models that work decently well for certain tasks. Once you've downloaded the weights, you've got no dependence on any provider, dictating the price, terms of use, or access restrictions (or simply having uptime issues).
Lower cost. This depends on your particular context. If you're using LLMs lightly and sporadically, and you've already got a decent hardware to run local models, you may not need an additional subscription to a cloud-based model at all. If, on the other hand, you're an AI power user, using local LLMs might still be beneficial when you don't need the full power of a commercial model and you'd like to save up on tokens.
A note on security
Notice that we've not mentioned security as one of the benefits. Coding agents provide user-level access to non-deterministic computations, no matter whether it's run locally or on someone else's computer. There is still no industry-wide realization of that fact, and the guardrails that models or harnesses have got out of the box feel more like an afterthought. There are already examples of agents deleting an entire production database and all backups against explicit instructions as well as a proof-of-concept hijacking attack in which an external connection to Jira may result in a coding agent sending locally-stored credentials to an attacker. Locally run models are just as susceptible to these attacks as the cloud ones. Therefore, proper sandboxing is advised no matter which model and harness you're using. This can be achieved using Docker containerization or a tool like Nono.
Summary
There is no illusion, open-weight LLMs are simply not as capable as the frontier models. Some local models are powerful enough to compete with the cloud ones, but cannot be actually run on the hardware most devs already have2. Regardless of the model, the local setup requires more tinkering and a deeper understanding of the underlying technology. The questions one should actually ask are "Can I run a good enough model at a reasonable speed on the hardware I can get access to? Do the benefits outweigh the costs of hardware and figuring out the initial setup?"
There are many factors here, as what's "good enough" depends entirely on your specific needs. Some devs may not use coding agents at all, while others are maxing out existing top models and eagerly wait for the next big release. What is certain, local agents are definitely good enough for some developers already. At Kiwee, we're evaluating their applicability in our work, thus putting the local LLM coding agents in the Assess ring.