The Harness Tax: Why the Same AI Model Can Deliver Very Different Results

What we learned building and operating Atomicwork's AI Workforce for IT teams

In 2024, I wrote about a new CAP theorem for B2B GenAI apps: Cost, Accuracy and Performance. At the time, most of the trade-offs we were dealing with looked like model choices. Which model was more accurate? Which one was faster? Which one was cheaper?

Two years later, our view is more nuanced. Atomicwork has evolved from an AI-native ITSM and ESM platform to one where AI Coworkers run ITSM and ESM. We ask models to do more than just answer questions or execute code scripted workflows. Atomicwork AI Coworkers investigate incidents, find the right tools, work across enterprise systems, take actions and verify that the work is complete. Once you operate AI this way in production, the model is only one part of the system.

The layer around the model matters just as much: the instructions, context, tools, state, retries, permissions and verification logic that turn a model into an agent. That layer is the agent harness. Change the harness and you can change the accuracy, latency and cost of the same model.

I think of the avoidable overhead in that layer as the Harness Tax. The practical lesson is more important than the term itself: for production agents, cost per token is not the metric to optimize. Cost per verified outcome is.

Same model, different system

HarnessTax is a useful new study because it isolates this effect. The researchers ran seven models through three coding-agent harnesses, Claude Code, Codex CLI and the minimal open-source Pi harness, on SWE-bench Lite and Terminal-Bench 2.0. The point was not to find a universal winner. It was to see how much the execution layer changes the behavior and economics of the same model.

The result below is the clearest way to see it. On SWE-bench Lite, Claude Fable 5 reached 97.8% success in Claude Code at $1.33 per rollout. Pi reached 96.7% at $0.67. The extra 1.1 percentage points came with almost twice the rollout cost.

The native harness was not always the best accuracy-cost pairing either. Claude Opus 4.8 reached 88.9% in Codex at $0.69 per rollout, compared with 86.7% in Claude Code at $0.98. Pi was cheaper still at $0.47, with a lower 82.2% success rate. There is no single "best harness" in those numbers. You are choosing a point on a cost-quality curve, and the right point changes with the job and the cost of being wrong.

The study also helps explain where the tax can come from. For Fable 5, Pi and Claude Code averaged almost the same number of turns, 15.4 and 15.3, yet Claude Code cost about twice as much. The researchers point to the amount of context carried into the model as one contributor: Claude Code's mean initial context was more than 10 times Pi's, with longer instructions and larger tool schemas.

More scaffolding is not automatically bad. Enterprise agents need identity, permissions, policy, observability and safety controls. Harder jobs may need more structure. But every instruction, tool description and orchestration loop should earn its place. Complexity that does not improve the outcome is just tax.

We saw the same effect in enterprise IT

Coding benchmarks are useful, but enterprise IT is a different environment. That is why we built ITSMBench with New Measure. We wanted to test whether an agent could actually close an IT ticket, not just produce a plausible answer.

ITSMBench puts agents inside a simulated enterprise with 42 mocked applications, roughly 1,800 database tables and more than 2,000 REST endpoints. It includes 89 tasks across Device Ops, Network Ops, Access, SecOps and Compliance. Each task is graded with 10 to 30 deterministic checks against the final state of the environment.

That last point matters. An AI Coworker investigating a compromised endpoint does not succeed because its explanation sounds right. It has to make the right judgment, complete the work across the systems involved, and avoid unsafe or excessive changes. In ITSMBench, we can verify all three.

When we changed the harness while testing the same model, the movement was larger than we expected. In our runs, switching harnesses moved Opus by seven percentage points, which was larger than the gap between some of the frontier models we were comparing. Opus also scored higher through Pi than Claude Code at every reasoning level we tested, while costing less. GPT through Pi versus Codex was much closer, usually within a point or two depending on reasoning level.

This changed a basic assumption for us. The unit of evaluation is not the model. It is the execution system around the model.

Cost per Verified Outcome = f(Model, Reasoning Effort, Harness, Context, Tools, Workload).

That is closer to what we are actually choosing in production. Saying "Claude scored X" or "GPT scored Y" hides too much. A model provider's native harness can be excellent for the job it was designed for and still be the wrong trade-off for a different workload. Claude Code is designed for coding. An Access Ops Manager, Network Ops Engineer or Security Ops Analyst has a different job, different risk and a different definition of done.

The production metric: cost per verified outcome

This is the part I think matters most for enterprise AI. Once agents start doing work, cost per token becomes a weak business metric. A long-running investigation may discover tools, query several systems, inspect dependencies, reason about policy, request approval, execute a change and then verify the result. Every unnecessary turn or chunk of context compounds. But a cheap run that stops halfway is not cheap either. Someone still has to finish the work.

So the metric we increasingly care about is simple:

Cost per verified outcome = all-in execution cost / verified successful outcomes

"All-in" should include the model, harness overhead, tool calls, retries and, when the workflow needs it, human review or remediation. "Verified" is equally important. For an access request, success may mean the correct entitlement was granted, the approval policy was followed, no extra privilege was added and the audit trail is complete. For an incident, it may mean the root cause was fixed, every affected service is healthy and unrelated systems were left untouched.

This is a better way to compare two agent systems. A cheaper model that takes more steps, repeatedly reloads context or leaves more work for a human can cost more per finished job. A more expensive model in a lean harness can be cheaper overall. The reverse can also be true. You cannot know from token price alone.

This is also why deterministic evaluation matters. If you cannot define what "done correctly" means for a workflow, you cannot measure the economics of autonomy with much confidence.

What this changes in how we build AI Coworkers

ITSMBench surfaced another useful pattern. Different models are good at different parts of the job. In our evaluation, Grok was particularly good at discovering the right APIs and tools. Opus was stronger at finishing once the right tools were found. GPT generally sat somewhere between those behaviors. Across the 89 tasks, 73 were solved at least once by some model in some trial, but no single model covered the entire solved set.

That is a strong argument against designing an AI Workforce around one model and one execution strategy. The best model for finding a path through an unfamiliar environment may not be the best one for executing it. A high-risk security investigation should not have the same reasoning budget as a routine software-access request. A five-step access workflow should not be forced through the same harness as a 50-step incident investigation.

What this pushes us toward is a more adaptive runtime: pick the model, harness, context, tools and reasoning budget for the job in front of you, then judge that combination by the outcome it produces.

Five principles we are carrying into production

1. Benchmark the agent system, not just the model. Public model benchmarks are useful signals. Production decisions should be based on the model-harness-tool-context combination you will actually run.
2. Optimize for cost per verified outcome. Instrument the full path, including inference, context, tool calls, retries, approvals and remediation. Then measure what it costs to finish work that passes your safety and completion checks.
3. Treat harnesses as workload-specific. There is no reason to assume the same harness should be optimal for coding, access management, security operations and incident response.
4. Make complexity earn its place. Extra instructions, tool schemas and orchestration steps are justified only when they improve reliability, safety or economics.
5. Decouple model choice from harness choice. Models will leapfrog one another. Harnesses will evolve too. You want the freedom to change either without rebuilding the entire system.

Harness engineering is becoming part of AI engineering

The first wave of enterprise AI focused on model engineering. Then came prompt engineering and context engineering. Long-running agents add another discipline: harness engineering.

The questions are practical. What should the agent know upfront and what should it discover? Which tools should it see? How much context should be loaded? When should it reason longer? When should it ask a human? How should it recover from a bad tool call? What evidence is required before it acts? How does it know the job is finished?

Those questions barely matter when AI is drafting an email. They become central when AI is changing production systems. The quality of the harness determines whether model intelligence turns into reliable work or expensive activity.

Revisiting the CAP theorem

When I wrote the CAP theorem post in 2024, my argument was that production GenAI systems force trade-offs between Cost, Accuracy and Performance. I also argued that teams should stop getting excited by leaderboard numbers and build evaluation systems around their own workloads.

I still believe that. The update I would make now is that the harness changes the shape of all three curves. It can change accuracy without changing the model. It can change cost without changing the model. And it can change latency and execution behavior without changing the model.

So the optimization problem has moved up a level. We are not choosing a model and then wrapping software around it. We are continuously tuning an execution system. Cost per verified outcome gives us a much cleaner way to judge whether that system is getting better.

The model leaderboard will keep moving. Access to a frontier model is not much of a moat because everyone can rent the same intelligence. The harder engineering work is everything around it: context, tools, identity, permissions, verification, recovery, observability and the ability to switch model-harness combinations as the economics change.

That is one of the most important lessons from building and operating Atomicwork's AI Workforce for IT teams: do not optimize the model in isolation. Optimize the verified outcome, and account for the Harness Tax.