LLM Coding Harness Optimization — Boost 15 Models by Changing the Harness

Overview

“Which LLM is best at coding?”

Every time this question comes up in engineering teams, we overlook one critical variable: the harness. A harness refers to the entire interface layer through which an LLM reads files, receives prompts, and applies edits.

In February 2026, Can Bölük published “I Improved 15 LLMs at Coding in One Afternoon. Only the Harness Changed,” tackling this harness problem head-on. By changing just the edit format, coding performance improved by 5 to 14 points across 15 LLMs, and output tokens decreased by roughly 20%.

graph TD
    subgraph Conventional Thinking
        A["Model A vs Model B<br/>Which one is better?"]
    end
    subgraph Actual Performance Drivers
        B["Model"] ~~~ C["Harness"]
        C --> D["Edit Format"]
        C --> E["Context Injection"]
        C --> F["Middleware"]
    end
    A -.->|"Shift in Perspective"| C

This article covers the concept of harness engineering, benchmark data, and practical strategies from an Engineering Manager/CTO perspective.

What Is a Harness?

A harness encompasses all the infrastructure between an LLM and actual code.

Component	Description	Examples
Edit Format	How the model modifies code	diff, string_replace, hashline
System Prompt	Instructions given to the model	Self-verification loops, problem-solving strategies
Tool Interface	Tool definitions available to the model	read_file, edit_file, run_test
Context Injection	Pre-loaded environment information	Directory structure, evaluation criteria
Middleware	Execution flow control	Loop detection, pre-completion verification

Here is the key insight: the same model can perform dramatically differently depending on the harness.

In the Aider benchmark, simply changing the edit format caused GPT-4 Turbo’s accuracy to jump from 26% to 59%, proving this point.

Comparing Three Edit Formats

Today’s major AI coding tools use different edit formats.

1. apply_patch (OpenAI Codex Approach)

This is the diff-based patch format used by OpenAI in Codex. The model outputs modifications in unified diff format, and the harness parses and applies them to files.

Pros: Models familiar with diffs perform reliably. Cons: High failure rates for models with insufficient diff format training. Grok 4 recorded a 50.7% failure rate.

2. string_replace (Claude Code, Gemini Approach)

This approach specifies the exact string to find and the exact string to replace it with. Claude Code’s str_replace tool is a prime example.

Pros: Intuitive and simple to implement. Cons: A single mismatched space or indent triggers a “String to replace not found” error. Perfect string reproduction is required.

3. hashline (A New Approach)

Proposed by Can Bölük, this method assigns a 2-3 character content hash to each line of a file.

11:a3|function hello() {
22:f1|  return "world";
33:0e|}

Instead of reproducing the entire source code, the model references hash tags to specify edit locations. For example, “replace line 2:f1” or “insert after 3:0e.”

Pros:

• No need for perfect string reproduction, reducing errors
• Hash mismatches automatically detect file state changes, preventing conflicts
• Approximately 20% reduction in output tokens

Cons: Does not guarantee the same results across all models (GPT-3.5 struggles with hash reproduction itself).

What the Benchmarks Tell Us

Can Bölük’s benchmark ran 180 tasks across 16 models and 3 edit formats, with 3 runs each.

Model	Previous Format	hashline Format	Improvement
Grok Code Fast 1	6.7%	68.3%	+61.6pp
Gemini 3 Flash	—	78.3%	—
Grok 4	Low	Improved	61% reduction in output tokens
MiniMax	—	2x improvement	—

The Grok Code Fast 1 case is particularly striking. The model itself was identical, yet simply changing the edit format produced a 10x improvement from 6.7% to 68.3%. This is the potential of harness engineering.

Cursor’s Acknowledgment

The case that best illustrates the severity of this problem is Cursor. Cursor deployed a separate 70B-parameter neural network to fix edit failures. They acknowledged the edit format problem and committed an entire large-scale model to compensate for it.

Harness Engineering in Practice: LangChain’s Terminal Bench

Another case demonstrating the real-world impact of harness optimization comes from LangChain. Their team achieved a 13.7-point improvement from 52.8% to 66.5% on Terminal Bench 2.0 by optimizing only the harness without changing the model. They jumped from Top 30 to Top 5 on the leaderboard.

They employed three harness optimization techniques:

graph TD
    subgraph Three Layers of Harness Optimization
        A["System Prompt<br/>Emphasize self-verification loops"]
        B["Context Injection<br/>Pre-load environment info"]
        C["Middleware<br/>Loop detection, pre-completion verification"]
    end
    A --> D["52.8% → 66.5%"]
    B --> D
    C --> D

1. Self-Verification Loops

Agents tend to terminate at the first plausible solution. LangChain enforced a “build-verify-fix” loop across all three layers: system prompt, context injection, and middleware.

2. Reasoning Compute Allocation Strategy (“Reasoning Sandwich”)

Rather than allocating uniformly high reasoning to every step, they distributed it strategically:

• Planning phase: Maximum level (xhigh)
• Implementation phase: High (high)
• Verification phase: Maximum level (xhigh)

This “sandwich” strategy produced better results than uniform xhigh reasoning. It wisely allocated reasoning resources within timeout constraints.

3. Environment Onboarding

They pre-loaded environment information for the agent, much like onboarding a new engineer:

• Available tool inventory
• Directory structure
• Evaluation criteria
• Time constraints

This prevents the agent from wasting time exploring the environment.

Three Takeaways for EMs and CTOs

1. Harness Optimization May Offer Higher ROI Than Model Switching

Instead of switching vendors every time a new model launches, optimizing the harness for your current model can be more cost-effective. Model switching requires readjusting API keys, prompt formats, token limits, and more, while harness optimization allows incremental improvement on top of existing infrastructure.

2. Open-Source Harnesses May Beat Vendor Lock-In

One of Can Bölük’s key arguments: open-source harnesses benefit from a diverse community of model users, each fixing the failures they encounter, often outperforming vendor-specific harnesses in general-purpose scenarios.

In contrast, Anthropic blocking OpenCode and Google deactivating the author’s Gemini account illustrate the risks of vendor lock-in.

3. The Gap Between “Cool Demo” and “Reliable Tool”

“The gap between ‘cool demo’ and ‘reliable tool’ isn’t model magic. It’s careful, rather boring, empirical engineering at the tool boundary.” — Can Bölük

When evaluating AI coding tools as a CTO, you should measure actual edit success rates, retry ratios, and token efficiency rather than flashy code generation in demos.

Practical Implementation Guide

What Teams Can Do

1. Measure edit success rates: Track the ratio of successful edits to total edit attempts by your AI coding tools. Frequent “String not found” errors indicate a harness problem.

2. Introduce middleware: Add middleware for loop detection, pre-completion verification, and automatic context injection.

3. Differentiate reasoning strategies: Assign different reasoning levels to the planning, implementation, and verification phases.

4. Trace-based debugging: Use tools like LangSmith to track all agent actions, latency, and token consumption, then systematically optimize.

Practical Tools Shared by the HN Community

Tool	Purpose	Approach
Serena	Code intelligence	AST-based structural analysis
RepoMapper	Codebase mapping	Directory structure visualization
Tilth	Editing tool	Line hash + semantic sections (17-25% cost reduction)
Tree-sitter integration	AST-aware editing	Significant reduction in round-trips

Conclusion

In the 2026 AI coding tool competition, the deciding factor is not just “which model you use.” What harness you build on top of that model creates the real performance difference.

• A single edit format change: 6.7% to 68.3% (10x improvement)
• Harness optimization alone: Top 30 to Top 5 (13.7 points)
• Output token reduction: 20-61%

As an Engineering Manager looking to boost your team’s AI coding productivity, before waiting for the next model release, start by measuring your current harness’s edit success rate. That number may reveal more than you expect.

References

• I Improved 15 LLMs at Coding in One Afternoon. Only the Harness Changed — Can Bölük
• Harness Engineering for Agentic Coding Systems — ZenML
• Hacker News Discussion
• Addy Osmani’s LLM Coding Workflow 2026