Introducing ITSMBench
ITSMBench tests whether AI can actually close IT tickets — not just talk about them. Every task runs inside a real enterprise environment: dozens of connected systems, thousands of records, the same complexity a service desk deals with daily. To pass, an agent doesn't just name the problem. It has to trace the actual root cause, follow the runbook, fix it, and leave every system it touched clean and consistent.
Leaderboard
Sample task▾
Prompt
Nadia Rahman on the finance team just posted in the #it-helpdesk Slack channel: CrowdStrike Falcon threw a malware detection on her workstation this morning, right after she opened a file she thought was a vendor invoice. She says the machine feels fine now and wants the alert looked at and closed out.
You're the IT responder on shift. Pick this up and handle it in the systems we operate. Work out what actually happened, contain anything that needs containing, and get the environment back to a safe state before you consider this resolved. Don't take the alert at face value as the full picture — close it out the way you'd want it handled if this were your own network.
LLMs are not street smart yet!
Agents have come far enough to rebuild SQLite in ~4 hours, passing 100% of the test suite. But can they autonomously resolve enterprise IT support tickets?
To address this, we're very excited to announce Atomicwork's ITSMBench. Built in collaboration with New Measure, ITSMBench is the first benchmark to evaluate frontier AI capabilities in the IT Service Management domain.
ITSMBench measures frontier models and the harnesses against the tickets an enterprise service desk actually receives. Agents work inside a simulated enterprise of 42 mocked software applications backed by roughly 1,800 database tables and 2,000+ REST endpoints. We score 89 tasks with 10 to 30 expert-written verifiers each, run as deterministic checks on the resulting environment.
And we're open-sourcing all of it, every task and its corresponding environment! GitHub Link
Agentic IT
A human service desk still has three layers. What separates them is not how hard the ticket sounds, but where the work actually lands.
With AI in the loop, deflection absorbs L1 outright, a master agent takes the L2 queue and hands the specialized cases to subagents, and humans stay in the loop for L3.
ITSMBench measures L2 and L3. We deliberately don't score L1: pulling an answer out of a knowledge base is the one part of this stack where model capability is already well understood, and it isn't where deployments break. L2 and L3 are where agents have to investigate across systems, make judgment calls, and apply the right fix.
Why another benchmark
The closest related work is ServiceNow's EnterpriseOps-Gym. It differs in three ways:
ITSMBench is built around the service desk end-to-end. Tool discovery, grading depth, and harness choice are variables we measure. The need for that measurement has grown as ITSM has moved from a system of record to an AI co-pilot, to a swarm of agentic coworkers.
Scores
73 of 89 tasks were solved at least once by at least one model in at least one trial.
No single model covers that set alone: Grok is strongest at finding tools, while Opus is strongest at finishing once the tools are found. Pairing those strengths, a finder model for discovery and a finisher model for execution, is a natural way an orchestration of models could beat any one model on its own.
Results use the open-source Pi harness at high thinking, so every model is scored under the same agent setup.
Harness matters as much as weights
We ran GPT and Opus in their provider harnesses (Codex and Claude Code) and again in an open-source Pi agent harness. Against Codex, the two models stay within a point or two of each other depending on thinking level. Against Claude Code, Opus scores higher through Pi at every thinking level and costs less to get there. A lean harness outperforms a bulky general-purpose one in this setting.
Failure modes
Models take shortcuts. A model reads the opening context and pattern matches itself into a workflow almost immediately, then runs that workflow instead of the ticket. What it never builds is a picture of the environment it just landed in, and that is what leads it astray.
A human engineer carries intuition about the systems around them, enough to weigh a piece of evidence and judge whether it really explains the problem. A model checks whether it can keep going. The shortcut lands somewhere different on every ticket.
The agent treats the reporter's description as the diagnosis, fixes the symptom the ticket names, and never looks for the real cause.
The agent settles on the explanation that looks familiar from training and stops investigating, even though nothing in the environment confirms it.
When no runbook covers the case, the agent invents a plausible procedure instead of looking up how similar incidents were handled before.
The agent makes a confident change without checking what still depends on it, and takes down something that was in use.
The agent works out the right fix but will not carry it out, leaving a note or an escalation instead unless a policy spells the action out.
The agent finds the true cause and then stops halfway, leaving related systems, records, and tickets inconsistent.
Assembling the environments
An environment has 5 parts: the tools, the data, the tasks, the agent, and the verifiers.
Tools. When a task starts, we spin up a mock for every system the ticket touches: Okta, Jira Service Management, Slack, PagerDuty, Gmail, Snipe-IT, Microsoft Defender, Intune, ServiceNow, Confluence, GitHub, and more. Every endpoint the real product exposes over REST is backed by its own schema and its own mock database, about 200 tools across the suite. How the agent finds them varies on purpose, some through a Skill.md, some over MCP, some plain REST, because that mixture is what an enterprise actually looks like.
Data. This is the part we would defend hardest. If the data doesn't carry real complexity, ambiguity, and heterogeneity, the whole thing is a toy: the same reset-password ticket resolves cleanly in one environment and fails a dozen different ways in another. Each task ships with a seed file that fills the databases before the run, and the red-team task alone uses about 1,300 records across seven systems, with no single record explaining the whole story. We built it with people who have run enterprise service desks, encoding the non-obvious gotchas they have hit.
Tasks. IT experts and the benchmark team wrote them together, modeled across five ops roles: Device, Network, Access, SecOps, and Compliance. Three rules hold for all of them:
Verifiers. Grading asks three things: right judgment, complete operational footprint, no unsafe or excessive changes. The checks are deterministic assertions over the final database state. We don't use an LLM as a judge. Every task also ships with a solution we run ourselves, and if it doesn't pass every check, we don't ship the task.
What next?
One model fits all is seldom the right answer. A service desk is optimizing for accuracy, cost, and time to resolution at the same time, and no single model wins all three. Getting this right takes a constellation of orchestrated models, and ITSMBench is a step towards this!
We've built simulated, realistic enterprise data, tasks, and workflows, and graded them fairly: hard enough to matter, scored against a deterministic rubric. Even so, errors can creep in. If you want to talk about how we built it, or have feedback to make it better, reach us at [email protected].
We'll keep iterating on these tasks and move toward longer-horizon workflows. The goal is truly autonomous service desks, and better time-to-value for the organizations that run them.
Built in collaboration with New Measure.