5 things we learned building an agentic security tool that verifies its own findings

Every security scanner has the same failure mode. You run it, wait, get 200 findings, spend an afternoon triaging, find 180 are false positives, file the 20 that survive, and watch half of those get deprioritized because the repro steps were too thin to act on. Then you do it again next week. None of it ranks the output. None of it verifies the output. None of it remembers anything between runs.

The scanning was never the bottleneck. The triage was.

So we built a security tool to attack that bottleneck: one that verifies its own output, remembers what it learned, and compounds over time. We call it SecHound, and we run it against our own platform continuously, not as a periodic audit. Here's how it runs, and the design decisions that mattered most.

How we run it

It scans every release, not once a quarter. Our product ships continuously, and every release is new attack surface, and a fresh chance to reintroduce a bug we already fixed. SecHound runs against each release rather than as a one-off, so coverage tracks the product as it changes instead of going stale between audits.

It's a team of agents, not a single prompt. A coordinator breaks the work into stages and hands each to a specialized agent: one maps the attack surface, others hunt specific classes of bug, another tries to reproduce and confirm what they find, and another tries to chain confirmed issues into something with real impact. They run autonomously and keep going until a pass stops surfacing anything new, then they stop. Nothing gets padded to hit a number, so what lands in the queue is signal rather than volume.

A human still owns the verdict. This is the part that matters if you've ever lost an afternoon to triage: what reaches a security engineer isn't 200 raw hits, it's a short list of findings that already have a working reproduction attached. Your team spends its time on impact, priority, and the judgment calls that actually need a human, not on separating the 20 real bugs from the 180 that never were.

Building and running that forced a set of decisions that aren't obvious upfront. Here's what actually mattered.

1. A smarter model won't fix your false positive problem

The first version of SecHound was confidently wrong a lot. The instinct is to upgrade the model. The fix turned out to be more mechanical: a checklist of the exact patterns that had already produced false positives, built up from real retractions.

The list includes things like:

An existing control that neutralizes the bug: the vulnerability exists in theory, but something upstream already blocks it.

An ingress filter that kills the attack path: the route is vulnerable, but the path to reach it isn't.

Preconditions no real attacker has: the exploit requires a foothold or permission level that isn't achievable in practice.

Cited code locations that don't resolve: the model referenced a line that doesn't exist or was misread.

Every confirmed finding runs against this list automatically before it's filed, and again before any retraction is accepted. A side effect worth noting: a well-built agentic tool retracts a lot of its own output. That's the point. A false retraction is as costly as a false positive, so the checklist is designed to cut both ways. This kind of discipline is what responsible AI practices look like at the tooling layer.

2. The confirmation gate must be deterministic, not delegated to the model

This is the most important architectural decision in the project. Promotion from "candidate" to "confirmed" does not go through the LLM. It goes through a confirm gate: plain code that checks two things.

No live evidence, no confirmation, regardless of how convincing the model's reasoning is. For cross-tenant or IDOR-class bugs, the bar is higher still: a single-identity repro doesn't count. You need a two-identity diff that proves identity A can reach identity B's resource, not just that the endpoint exists.

A scanner that only pattern-matches can't tell you when a control actually holds. Deterministic gates can, and that second result is what builds trust in the first. It's the same principle behind deterministic guardrails in AI systems: execution policies that enforce limits in code, not just in prompts.

3. Invariant-breaking finds what vuln-class matching misses

Catalog-based hunting only finds bugs that already have a name and a pattern. The more productive framing is to ask what the system claims is always true, and then try to falsify each claim:

"A user can only ever access their own organization's data."

"This action requires explicit authorization before it can be invoked."

"Internal services are never reachable from user-controlled input."

Write down those invariants. Rank them by how often similar claims have already been broken in systems like yours. Hunt in that order. This is where the off-taxonomy bugs come from: the ones no scanner has a rule for because nobody has formally described them yet. It's a pattern that maps closely to how agentic AI systems are designed: goal-oriented, context-aware, and not limited to a fixed catalog of known patterns.

4. State on disk compounds faster than starting fresh each run

When state lives on the filesystem rather than in model context, it keeps two persistent knowledge layers that survive between runs:

Per-target layer: how each service authenticates, where its sinks are, which bug classes have already been disproved, what surface hasn't been touched.

Global layer: framework-level patterns, reusable techniques, dead ends not worth retrying across targets.

Coverage stops resetting to zero. Because SecHound runs against every release, next week's run starts with everything this week's run learned. The compound stage folds each new confirmed finding back into both layers, so the system gets sharper over time rather than repeating the same surface area. This is what universal context looks like applied to security: building a persistent, compounding picture of your environment rather than re-deriving it from scratch every time.

One practical note: hand-drawn service topology drifts and lies. A code graph generated directly from source doesn't.

5. Model-agnosticism isn't a nice-to-have: it's how you survive model churn

Design every stage to talk to an LLM through exactly one function. Swapping the backend should be one environment variable and touch zero stage logic.

The reason this matters in practice: model quality moves fast, model pricing moves faster, and the best option for code-reading tasks today isn't necessarily the best option in six months. If your security tooling is tightly coupled to one provider, you absorb every pricing change and every capability regression. The new CAP theorem for GenAI apps (cost, accuracy, performance) applies here too: the right backend trade-off depends on the task, and you can only make that choice if the seam exists.

The caveat worth being honest about: stages that read code and run grep are only as good as the backend's tool-use capability. Agentic backends that can call tools give sharp verdicts. Completion-only APIs that reason from text alone come out weaker. The seam lets you match the model to the task rather than committing to one for everything.

What this looks like in practice

To ground this, here's how a conventional scanner workflow stacks up against the agentic loop we run:

The shift is about consistency: the verification runs the same way every time, regardless of how much patience is left for triage that afternoon. That's the part that scales.

If you're fighting the same triage bottleneck on your own surface, or want to see how these principles carry into the way we approach enterprise security at Atomicwork, we'd be glad to compare notes. Reach out.