Your Model Isn’t The Problem
A guy on r/LocalLLaMA spent six months trying to make agentic coding work with local 30B to 120B models and came away with nothing but technical debt. His agent drifted off the methodology he specified (ask for functional code, get OOP soup instead), ignored instructions past roughly 50,000 tokens of context, wrote tests that checked nothing, and piled new code on top of old messes instead of ever refactoring. He blamed the models.
He was blaming the wrong thing. The people in that thread actually shipping code, a principal developer and a game engineer who maintains a 300,000-plus line C++ engine with local Qwen models, said the same thing from two different angles: cap your working context around 32K no matter what the spec sheet says, and structure your workflow so raw execution noise never touches your main conversation in the first place. Qwen3.6 and Gemma 4 ship with 256K to 262K token windows. The practitioners actually shipping production code use about an eighth of that on purpose. That gap between the marketing number and the working number is the entire article.
Full example: Clone the working files (capped Modelfile, llama.cpp launcher, a token-budget script, an AGENTS.md that uses positive framing, subagent sketches) at github.com/KingPin/sumguy-examples/llm/local-coding-agents-context-discipline
The Headline Spec Is A Trap
Model cards love a big context number because it benchmarks well and looks great in a table. Qwen3.6-27B advertises 262,144 tokens. Gemma 4 12B advertises 256K. Nobody advertises what actually happens to instruction-following at token 180,000, because “still coherent” doesn’t fit on a spec sheet next to “262K.”
Every token in the window is something the model has to weigh against every other token when it decides what to do next. Feed it your git log, three failed test runs, a half-finished refactor, and a Stack Overflow post it grepped up on its own, and by the time you ask for the actual feature, your instructions are one voice in a room full of noise. The model can count that high, sure. But “can technically hold 256K tokens” and “produces good code at 256K tokens” are two completely different claims, and only one of them is printed on the box.
The fix nobody wants to hear: stop treating context size like a fuel tank you’re supposed to fill up. Treat it like a workbench. A bigger workbench doesn’t make you a better carpenter if you cover it in sawdust and yesterday’s coffee cups.
Context Hygiene: The Boring Stuff That Actually Works
Cap it low, on purpose
Set a hard ceiling well under the model’s max, even though it feels like leaving capability on the table. The practitioners in that thread ranged from “100 to 150K as an outer limit” to “I run 32K and that’s it.” The person maintaining the biggest, oldest, most complex codebase in the conversation was the one running 32K. That’s not a coincidence. Less headroom means less temptation to let junk accumulate, because you feel the ceiling coming and you clean up before you hit it.
This is llama.cpp:
llama-server \ -m qwen3.6-27b-instruct-q5_k_m.gguf \ -c 32768 \ -ngl 99 \ -ctk f16 \ -ctv f16-c (or --ctx-size) sets the total context budget for the server. -ctk and -ctv set the KV cache data type for keys and values respectively. Both default to f16, and you leave them there. The next section covers why.
Same idea in an Ollama Modelfile:
FROM qwen3.6:27b
PARAMETER num_ctx 32768num_ctx in a Modelfile bakes the context size into the model so you’re not passing it as a runtime flag every time. Ollama’s own default scales with your VRAM: 32k of context on a 24 to 48 GiB card, and 256k once you’re at 48 GiB or more. So depending on the box, that one line either raises a small ceiling or drags a huge one back down. Pin it explicitly and you get identical behavior on every machine instead of a number that silently changes when you swap GPUs.
Yes, this contradicts Ollama’s own advice
Worth knowing you’re disagreeing with the vendor here. Ollama’s context-length docs say tasks like agents and coding tools “should be set to at least 64000 tokens,” and suggest using a model’s maximum context for best performance. That advice is about capacity: making sure a long file or a fat tool response physically fits. The 32K ceiling is about adherence: making sure your instructions still win against everything else in the window. Both can be true at once. If your tasks genuinely need 64k of code resident at the same time, raise the ceiling, then work twice as hard at the hygiene below, because every failure mode in this article gets worse with more room to fill.
Don’t quantize the KV cache for agentic work
This one trips people up because quantizing the KV cache is a legitimate, well-supported way to squeeze a bigger context into less VRAM. q8_0 roughly halves KV cache memory versus f16 with a small accuracy hit. For a single chat turn, that hit is nearly invisible.
Agentic coding isn’t a single turn. It’s dozens of turns chained together, each one building on the last. Small numerical errors from a quantized cache compound across that chain the way rounding errors compound in a spreadsheet nobody double-checks. One imprecise turn feeds a slightly-off next turn, which feeds the next, and eventually you’re debugging phantom behavior that has nothing to do with your code and everything to do with cache math. Nobody has benchmarked where that starts to bite, so treat it as practitioner caution rather than a measured threshold: leave -ctk f16 -ctv f16 alone for coding agents. Save the KV quantization trick for chat use cases where each turn stands alone.
If you’re on Ollama, OLLAMA_KV_CACHE_TYPE=q8_0 only takes effect when flash attention is enabled (set OLLAMA_FLASH_ATTENTION=1 if your build doesn’t already have it on), because that’s the only code path where Ollama quantizes the cache at all. For agentic coding, the simplest safe move is to leave OLLAMA_KV_CACHE_TYPE unset and take the f16 default.
Hand-pick your files, don’t let the agent go spelunking
If you let a coding agent grep and glob through your repo hunting for context, you get context, all right, and most of it is dead weight. Every file it reads to “understand the codebase” stays resident in the window whether it turned out relevant or not. Feed it the two or three files that actually matter for the task instead of turning it loose in the repo like a raccoon in a pantry. It’ll find plenty. Not all of it is food.
Roll back, don’t argue
This is the single best tip in that whole thread, and almost nobody does it. When the agent hallucinates a function that doesn’t exist, invents an API, or produces something you know is wrong, the instinct is to reply “no, that’s wrong, do it again.” Don’t. That bad output is still sitting in context, and a bad turn in the window raises the odds that the next turn is also bad. The model is now conditioning on its own mistake plus your correction plus its next attempt, which is three chances for the wrong pattern to reassert itself instead of one.
Wipe that turn out of the context entirely. Regenerate from a clean point, or edit the conversation history to remove the bad exchange before continuing. It feels like extra friction in the moment. It’s cheaper than the twenty minutes you’ll spend later untangling code the model wrote while half-anchored to its own hallucination.
Separate planning from execution
Arguing about architecture and writing code are different jobs that want different context. Draft a plan, hash out the design with the model, argue about the approach, then wipe the chat back to a clean slate and inject only the final agreed plan before you tell it to execute. This keeps the meandering “what if we used a factory pattern here, actually no” back-and-forth out of the context that’s about to do the actual work. The execution model gets a clean brief instead of a transcript of you two changing your minds four times.
Re-inject the system prompt
Instruction adherence decays over a long session the same way your own attention does after the fourth hour of debugging. Every few turns, resend the core rules (style, methodology, constraints) instead of trusting the model to keep them loaded from turn one. It’s a few hundred tokens of insurance against the exact failure mode from that Reddit post: instructions getting ignored past 50K tokens of drift.
Writing An AGENTS.md That Doesn’t Suck
The classic mistake is a wall of “don’t” rules with zero examples of what “do” looks like. A list of prohibitions with nothing to aim at is worse than no list at all, because the model burns tokens processing constraints without ever seeing the shape of a correct answer.
Here’s the useless version:
- Don't use classes unless absolutely necessary- Don't write god functions- Don't ignore error handling- Don't use global state- Don't skip type hintsNone of that tells the model what “good” looks like. Here’s the version that actually works, with a concrete example baked in:
## Code Style
We write functional-first Python. Prefer small pure functions over classes.Example of the pattern we want:
```python title="orders.py"def apply_discount(order: Order, pct: float) -> Order: """Return a new Order with pct% knocked off the total. Pure, no mutation.""" return replace(order, total=order.total * (1 - pct))```
Not this:
```pythonclass OrderProcessor: def __init__(self, order): self.order = order def apply_discount(self, pct): self.order.total *= (1 - pct) # mutates in place, no```
Match the first pattern for new business logic. Classes are fine for statefulthings like DB connections, not for a discount calculation.That’s the same rule, but now the model has a template to pattern-match against instead of a prohibition to avoid. Positive framing with a real example beats a negative rule every time, and it costs you maybe thirty extra seconds of writing.
Even better than a long markdown file: point the model at a real file in your actual repo as the style target. “Match the pattern in orders.py” carries more signal than three paragraphs of abstract prose about functional purity, because the model doesn’t have to translate your description into code. It’s already looking at code.
The Orchestrator Pattern: Keep The Main Thread Clean
The second half of this is architectural. Instead of one long conversation doing exploration, coding, and testing all in the same window, split the work into a main orchestrator that talks to you and holds the high-level goal, plus disposable subagents that do the messy work and get thrown away.
The orchestrator never touches raw execution noise. It doesn’t see every failed test run or every file the Explorer subagent opened while looking for the right module. It sees summaries. When you need to find where auth logic lives, you spin up an Explorer subagent with exactly one tool (read-only filesystem search) and one job. It reads a dozen files, its context balloons doing that, then it reports back “auth lives in src/auth/, entry point is verify_token()” and gets wiped. That balloon never touches your main thread.
A rough sketch of what a subagent role definition looks like in practice:
name: explorerpurpose: > Locate relevant files and report structure. Never writes code.tools: - read_file - list_directory - grepsystem_prompt: > You are a codebase explorer. Given a task description, find the files relevant to it and summarize what they do in under 200 words. Do not propose implementations. Do not modify anything. Report file paths and a one-line purpose for each.context_lifetime: single_taskname: coderpurpose: > Implement one specific, already-scoped change.tools: - read_file - write_file - run_testssystem_prompt: > You implement exactly the change described in the task. Follow the patterns in AGENTS.md. Do not refactor unrelated code. Do not explore the repo beyond the files you were handed.context_lifetime: single_taskExplorer, API-Researcher, Coder, Tester, Documenter: each one gets a minimal toolset for one explicit job, and each one’s context dies the moment the job is done. Giving a subagent five tools it doesn’t need for its one job is like handing the guy changing your oil a full mechanic’s toolbox when he needs a wrench and a drain pan. He’s not going to use the rest, and now there’s more stuff on the table to trip over. A small, sharp toolset beats a bloated system prompt listing every tool definition under the sun, both because it’s fewer tokens the model has to hold and because it structurally prevents scope creep. A subagent that can’t glob the whole repo can’t accidentally load the whole repo into context.
Pair this with a library of small, focused skill files instead of one giant instructions dump: a docstring convention file, a “how we create a new service class” template, a testing-pattern file. Load only the skill relevant to the current subtask instead of shipping every convention you’ve ever written to every agent on every call.
Spend 80% Of Your Time Not Writing Code
This is the part that stings if you’ve been sold the “AI writes your app while you sleep” pitch: the people getting good results from local models spend roughly 20% of their time generating new code and 80% evaluating, testing, and refactoring what’s already there.
Bad patterns self-propagate. Once junk lands in your codebase, the model reads that junk as precedent and copies it on the next feature, and the feature after that. It’s not being lazy, it’s doing exactly what you asked: match the existing style. If the existing style is a mess, you’ve just taught it to make more mess, faster, with a straight face. When you notice quality sliding, the fix isn’t a stricter prompt. Stop shipping features entirely for a session and go clean up architectural boundaries, kill lint errors, and pay down the debt. A clean codebase is worth more than a clever prompt, because the codebase itself is most of what the model reads before it writes a single new line.
This is also why local models perform better on an established codebase with a solid AGENTS.md and real test coverage than on a greenfield project. Counterintuitive on its face (shouldn’t a blank slate be easier?), but the existing code is a huge chunk of the effective prompt whether you write it out or not. Greenfield means the model has nothing to imitate and has to invent conventions from scratch, which is exactly the situation where it drifts into whatever its training data leans toward instead of what you actually want.
Review Everything, Trust Nothing The Agent Says About Itself
Two habits, both boring, both non-negotiable.
First: never trust the model’s own opinion of a test it just wrote. Ask “is this test good?” in the same session that generated it and you’ll get a confident yes every time, because it’s grading its own homework against its own blind spots. Review tests adversarially, in a separate session with a fresh context, with a prompt like this:
Here is a function and a test that claims to cover it. Review this adversarially.Does the test actually verify the function's stated behavior, or does it justassert that the code does whatever it currently does? Would this test stillpass if I introduced a bug in the core logic? List at least one input thistest doesn't cover.
Function:<paste function>
Test:<paste test>A fresh context with no investment in the test looking good will actually go looking for the gaps.
Second: review every line the model hands you before it goes anywhere near a commit. Ask it to explain any chunk you don’t immediately follow, or to perform a specific named refactor rather than a vague “clean this up.” If you wouldn’t have written the line yourself and you can’t explain why it’s there, it doesn’t go in. Call that paranoid if you like. It’s the same bar you’d hold a junior dev to on day one, and your local 27B model has read less of your codebase’s history than a junior who’s been there a week.
None of this is exotic. It’s mostly discipline: smaller windows, cleaner handoffs, adversarial reviews, and the patience to spend four sessions on tech debt instead of shipping feature five on top of a shaky feature four. The 256K context window on the box is real. So is the fact that the people actually shipping code with these models are using an eighth of it, on purpose, every single time.
