The agentic AI security market has a marketing problem. Every platform now claims to do “agentic penetration testing.” The phrase has been stretched so thin it has stopped meaning anything. I’ve seen glorified Nessus wrappers wrapped in an LLM prompt chain call themselves agentic. I’ve watched tools that fire predefined payloads at endpoints describe their approach as “AI-driven exploitation.” I’ve reviewed vendor whitepapers that use the word “reasoning” to describe what is, under the hood, a decision tree with a language model bolted onto the output formatter.
This conflation is not just a marketing annoyance. It has real consequences for how organizations make procurement decisions, how they interpret the results they receive, and ultimately how exposed they remain after a supposedly comprehensive AI-driven assessment. When everything is called agentic, nothing is, and practitioners lose the vocabulary they need to distinguish between tools that enumerate known vulnerability patterns and tools that actually reason about application behavior.
So when Ridge Security published a head-to-head benchmark of three agentic AI pentesting platforms — RidgeGen, alongside Shannon and Strix (two prominent, open-source competing agentic pentesting frameworks chosen for their representative architectures in the current ecosystem) — against OWASP Juice Shop, I read it carefully.
Not because benchmarks are gospel. They never are. But because the architectural differences the benchmark revealed map almost precisely onto a set of failure modes I have been writing about in the context of the MAESTRO framework for the past two years. The results are a concrete, measurable instantiation of theoretical arguments about agent architecture that I and others in the AI security community have been making in the abstract. That’s worth unpacking in detail.
The Benchmark Setup Matters More Than the Scores
Before getting into what the numbers mean, the methodology deserves serious attention, because methodology is where most AI benchmarks fall apart.
All three platforms were tested against the same freshly deployed OWASP Juice Shop instance — a deliberately vulnerable Node.js application containing 110 structured CTF-style challenges spanning SQL injection, broken access control, business logic flaws, authentication bypasses, and a range of other vulnerability classes. Critically, each platform received only the target URL. No additional context, no vulnerability hints, no explicit instructions to maximize challenge completion scores. The Juice Shop’s internal challenge counter served as the primary proxy for exploitation progress, and that counter is generated by the target application itself, not by any evaluation harness. That design choice matters: it prevents the evaluators from inadvertently encoding their own assumptions about what “good” exploitation looks like into the scoring mechanism.
The LLM backend was held constant across all three platforms — Gemini 3 Flash throughout. The benchmark explicitly highlights the use of a relatively cheap and fast model to reinforce a core message: performance comes from the harness and architecture, not the model itself. The system design (belief state, verification, orchestration) is the real differentiator, not raw model capability. This is the single most important methodological decision in the entire study, and it’s one that many comparative AI evaluations get wrong. By controlling for the model, the benchmark forces the variable under test to be the system architecture: how the platform plans, how it maintains state, how it validates findings, how it chains discoveries into downstream exploration. Those are the dimensions that matter in production, and they are the dimensions that vendor marketing almost never discusses in technical detail.
The anti-cheat methodology was also more rigorous than typical. Each run was network-isolated so platforms could communicate only with the target application and the LLM provider, preventing retrieval of public Juice Shop walkthroughs. Execution traces were parsed for known benchmark-gaming patterns. An LLM-as-judge step reviewed whether each solved challenge reflected a clean reasoning and exploitation path against the live target or whether it relied on memorized internal Juice Shop knowledge. For a public benchmark where solution material is freely available online, these controls are not optional — they are the difference between measuring exploitation capability and measuring internet retrieval capability.
The Number That Matters: 63% Hallucination Rate on Security Findings
RidgeGen produced 55 findings. Every single one was backed by concrete execution evidence — full HTTP requests, server responses, extracted data, and reproducible reproduction steps. Zero hallucinations across 55 findings.
Shannon produced 27 findings. Ten of those 27 were evidence-backed. The remaining 17 described potential vulnerabilities without demonstrating actual exploitation against the target. Findings like INJ-SSTI-01 (Server-Side Template Injection), INJ-DESER-01 (Insecure Deserialization), and multiple XSS and access control entries appear to be vulnerability-class templates rather than confirmed exploits — descriptions of what a vulnerability of that type would look like if it existed, rather than proof that it exists in this application. That is a 63% hallucination rate on security findings.
Strix produced 6 findings, all evidence-backed. Genuine verification discipline, but coverage that leaves roughly 95% of the attack surface unexplored.
I want to stay on Shannon’s 63% figure for a moment, because in the context of real-world penetration testing engagements this is not a minor quality issue. It is a structural liability, and I think the security industry is significantly underestimating its implications.
When a finding is not confirmed against the actual target, the security team receiving that report faces a binary choice: invest engineering resources to manually verify whether the finding is real, or act on it without verification. Both paths are costly. Manual verification by a qualified security engineer is expensive. Acting without verification means potentially deploying remediation resources against vulnerabilities that don’t exist as described in this specific application instance, triggering false-positive tickets in vulnerability management systems, and in regulated environments, potentially filing inaccurate compliance documentation.
There is a deeper problem beyond cost. When unconfirmed findings mix with confirmed ones in a report, the cognitive load on the analyst reviewing that report increases nonlinearly. Every finding now requires a meta-level judgment: is this one of the real ones or one of the templates? That mental overhead compounds across an entire report, degrades the analyst’s confidence in the platform’s output broadly, and ultimately undermines the trust foundation that makes automated security tooling valuable in the first place.
Architecture Is Destiny: What Persistent Belief State Actually Means
The benchmark report attributes RidgeGen’s performance to its Commander/Specialist/Reflector architecture and what it describes as “beliefs memory” — a persistent, structured model of what has been tested and what has been confirmed. I want to be precise about what this means technically, because it’s easy to read “beliefs memory” and assume it just means “the system remembers previous findings.” It means something more specific and more consequential than that.
In a multi-step agentic workflow, a system’s belief state is its internal representation of what it knows about the target environment, what it has confirmed versus hypothesized, what attack paths it has explored, and what attack paths remain unexplored. A system with persistent belief state propagates new confirmations into this model and uses the updated model to reprioritize its exploration strategy. A system without persistent belief state treats each tool call as a transaction that begins and ends without updating a shared model of the target.
The practical difference is big. When RidgeGen confirmed the JWT alg:none vulnerability — a flaw that allows an attacker to forge JWT tokens by setting the algorithm parameter to “none” and bypassing signature verification — it didn’t log the finding and move to the next item in its testing queue. It updated its belief state to reflect that authentication in this application is breakable via token forgery, and it reprioritized its subsequent testing to explore every authenticated endpoint in the application under that new assumption.
What followed was not a single finding. It was a cascade:
The initial JWT alg:none bypass enabled identity forgery across the application. Once that was confirmed, RidgeGen tested role manipulation via the forged JWT — the question being whether elevated role claims in a forged token would be accepted by the application’s authorization layer. They were. That gave vertical privilege escalation.
With administrative access established via forged JWT, RidgeGen then enumerated the application’s API surface and began systematically testing each endpoint. At /api/Users/:id, it confirmed full account takeover via mass assignment — a vulnerability class where an API endpoint accepts and persists arbitrary fields submitted in the request body, including fields like role or isAdmin that should not be user-controllable. At /api/Products/:id, it confirmed unauthenticated price and description manipulation, exposing the application’s entire product catalog to unauthorized modification.
From there, it discovered twelve distinct Insecure Direct Object Reference (IDOR) vulnerabilities across basket, address, complaint, recycle, and privacy endpoints — cases where the application uses predictable numeric identifiers for user-owned resources and fails to verify that the requesting user owns the resource they’re accessing. And finally, with admin-level access confirmed, it tested administrative functionality and discovered admin-level enumeration and deletion capabilities across feedback, complaints, and user records.
Neither Shannon nor Strix discovered any of these cascading findings. They found the first layer — or in Shannon’s case, possibly described the first layer from template knowledge — and stopped.
This is the architectural failure mode I call “belief state amnesia”. A system that cannot maintain coherent knowledge about what it has learned across a long-running session cannot reason about the implications of its discoveries. It cannot ask the question: given what I now know about how this application handles authentication, what else becomes possible? Without that question, the system is not doing penetration testing. It is doing vulnerability scanning with extra steps.
In MAESTRO terms, this failure sits at Layer 2, the Agent Orchestration Layer. The Orchestration Layer is responsible for multi-step planning, task decomposition, state management across tool invocations, and the integration of intermediate results into updated plans. A system where the Orchestration Layer has no persistent model of the target environment will consistently fail to discover vulnerabilities that require multi-step reasoning — which is, increasingly, the class of vulnerabilities that sophisticated threat actors actually exploit.
The Trust Propagation Problem in Agentic Security Systems
Let me go deeper on the technical mechanism here, because I think it has implications beyond penetration testing that the broader AI security community should be thinking about.
When an agentic AI system confirms a vulnerability in a real application, that confirmation carries information about the application’s architecture that should propagate to the system’s model of adjacent components. This is what I mean by trust propagation — the process by which a confirmed finding updates the system’s beliefs about the trust relationships and access control assumptions in neighboring parts of the attack surface.
In the JWT case from the benchmark, confirming the alg:none bypass communicates several things about the application’s architecture. First, the application is validating JWTs in a way that trusts the token’s own algorithm declaration rather than enforcing a fixed algorithm — this is a specific implementation pattern that suggests the JWT validation library is being used incorrectly or an outdated version is in use. Second, role and identity claims in the JWT are being trusted server-side, which means any endpoint that relies on those claims for authorization is potentially vulnerable to the same forgery. Third, the application probably has a consistent JWT validation module used across endpoints, meaning the same flaw is likely to be present everywhere authentication is checked.
A system with genuine trust propagation would encode all three of these inferences into its belief state when it confirms the alg:none bypass, and it would use those inferences to build a targeted test plan: enumerate all authenticated endpoints, classify them by what JWT claims they check for authorization, and prioritize testing those that check role claims. That is exactly what RidgeGen appears to have done, based on the cascading findings it produced.
This is not a feature of the LLM — remember, all three platforms used the same model. This is a feature of how the system is architected to maintain and update its model of the target over the course of a session. Shannon and Strix had access to the same underlying model intelligence but produced none of these cascading findings. The difference is entirely architectural.
The implications extend beyond penetration testing. Any agentic AI system operating in a security-relevant context — threat detection, incident response, vulnerability triage, compliance assessment — needs to reason about trust relationships across components. A system that treats each query or observation as independent cannot build the kind of contextual understanding of an environment that real security work requires. This is one of the core arguments for why I believe agent architecture is the most important unsolved problem in AI security tooling right now, and why I have been spending significant time developing MAESTRO’s treatment of orchestration-layer vulnerabilities.
Business Logic Vulnerabilities: The Semantic Reasoning Test
The most revealing data point in the entire benchmark is not a number. It is a category.
This is significant because business logic vulnerabilities are the class of issues that most clearly distinguish semantic reasoning from syntactic pattern matching. To find a SQL injection vulnerability, you need to know what a SQL injection payload looks like and have a mechanism to inject it into user-controlled inputs. That is syntactic pattern recognition. It can be encoded in rules, signatures, and templates. It does not require any understanding of what the application is supposed to do.
To find a business logic vulnerability, you need to understand what the application is supposed to do, build a model of the intended behavior, and then reason about how that intent can be subverted by an attacker who operates outside the assumed usage patterns. There is no payload. There is no signature. There is no CVE. The vulnerability is defined by the gap between what the application intends and what it actually permits.
The specific business logic chain RidgeGen discovered in Juice Shop required several layers of semantic understanding. It had to understand that the application has a financial transaction model — users have wallets with balances, products have prices, and purchases are supposed to reduce wallet balance by the product price. It had to understand that baskets are an intermediate state in the transaction flow and that the basket’s state affects what is charged at checkout. It had to recognize that negative quantities in a basket are not validated at the item level, that the wallet-basket interaction at checkout does not compensate for this, and that the checkout process does not properly lock basket state before finalizing a transaction in a way that allows a race condition to be exploited.
Chaining these observations together produces a finding: by adding items with negative quantities and exploiting the race condition at checkout, an attacker can generate fraudulent financial transactions that charge a negative amount to the wallet — effectively paying the attacker rather than collecting payment. That is not a finding you discover by matching patterns. It is a finding you discover by reasoning about the application as a system with intended behavior that can be violated.
The fact that only one platform in this benchmark demonstrated that capability is a precise measurement of where the industry currently sits. We are mostly in the syntactic phase of automated security testing. Pattern-matching tools — including many that are marketed as AI-driven — can find vulnerability classes that have known signatures. They cannot find vulnerabilities that exist in the gap between application intent and application behavior, because they have no model of application intent.
The semantic phase — where AI systems construct models of application behavior and reason about violations of that model — is beginning to emerge, but it is not yet the norm. The benchmark result is a concrete measurement of that emergence.
Evidence Validation as an Architectural Invariant
I want to make a distinction that I think is important and frequently blurred in how vendors describe their verification approaches.
There is a difference between a system that produces evidence for its findings and a system where evidence validation is an architectural invariant. The first means that evidence appears in some or most outputs. The second means the system is architecturally incapable of reporting a finding without first executing a verification step that produces concrete evidence. Evidence is not an optional output — it is a structural gate. The architecture enforces it, not the model’s memory.
RidgeGen’s 0% hallucination rate across 55 findings is consistent with evidence validation being an architectural invariant. Every finding includes full HTTP requests, server responses, and extracted data — not because the system was instructed to include this information, but because the architecture requires it before a finding can be added to the report. Shannon’s 63% unconfirmed rate is consistent with evidence validation being something closer to a best-effort output — the system produces evidence when it has it, but findings can progress to the report without it.
This distinction matters in production contexts. An architectural invariant is auditable. You can inspect the system’s architecture and verify that finding generation is gated on evidence collection. A best-effort approach is not auditable in the same way — its behavior under different conditions, against different target types, under time pressure or resource constraints, may differ from what you observed in a controlled evaluation. The architectural invariant gives you a guarantee. The best-effort approach gives you a historical average.
For security professionals making procurement decisions, this is the right question to ask: is evidence validation implemented as a structural constraint on finding generation, or is it implemented as a step in the generation process that can be skipped or bypassed? The answer to that question tells you more about the platform’s reliability in edge cases than any benchmark number does.
Token Efficiency as a First-Order Metric
Shannon consumed 57.7 million tokens for 27 findings, of which 17 required manual validation — a rate of approximately 2,138K tokens per finding. RidgeGen consumed enough tokens to produce 55 zero-hallucination findings at a rate of 846K tokens per confirmed finding. Strix was token-efficient in absolute terms but produced so few findings that its per-finding cost remains high relative to RidgeGen.
The token efficiency gap matters, but not just because of direct API cost. There are three compounding effects that make this gap more significant than the raw numbers suggest.
The first is the manual validation multiplier. Shannon’s 63% unconfirmed rate means that for every 10 Shannon findings, a security analyst must spend time verifying approximately 6.3 of them before they can be trusted. That validation time is not reflected in the token count. When you add analyst time at a realistic billing rate to the token cost, Shannon’s true cost-per-trustworthy-finding is substantially higher than the token numbers suggest. RidgeGen’s 0% hallucination rate means its token cost is its true cost — there is no hidden validation overhead.
The second is scale sensitivity. Organizations that are integrating automated penetration testing into CI/CD pipelines — running assessments not once per quarter but on every significant deployment — will execute these systems hundreds or thousands of times per year. A 2.5x token efficiency advantage at that scale is not a minor cost optimization. It is a meaningful budget line item that affects whether continuous automated security assessment is economically viable.
The third is context window pressure. Long-running agentic tasks accumulate context over time. A system that burns tokens at a high rate will hit context window limits sooner in a session, which can force context truncation strategies that degrade reasoning quality and coverage toward the end of a session. Token efficiency in agentic systems is not just about cost — it is about how long the system can maintain coherent reasoning about complex targets before it has to start forgetting things.
Coverage Depth and the Compounding Value of Exploitation Chaining
The benchmark reports 21 solved challenges for RidgeGen out of Juice Shop’s 110 total. Shannon solved 7, Strix solved 3. These numbers seem modest in absolute terms — 21 out of 110 is less than 20% completion. But the right frame for this comparison is not percentage completion against a known CTF. The right frame is what the findings distribution reveals about each system’s coverage methodology.
RidgeGen produced findings across 14 vulnerability categories. Shannon and Strix produced findings in a subset of those categories. Three categories — mass assignment, business logic, and a miscellaneous/negative logic category — were exclusive to RidgeGen. In the broken access control category, RidgeGen produced 20 confirmed findings while Shannon produced 6 theoretical ones with zero execution evidence.
The concentration of RidgeGen’s coverage in access control and authorization vulnerabilities, combined with the cascading IDOR and privilege escalation findings I described earlier, suggests a testing methodology that has compounding properties. Finding one access control flaw doesn’t just produce one finding — it opens an investigation of the entire access control surface, which produces additional findings, which may open adjacent investigations. The total finding count is not additive. It is geometric under the right architectural conditions.
This compounding dynamic is what the benchmark’s creators mean when they describe RidgeGen’s “exploit one, test everywhere” methodology. From a theoretical standpoint, this is the correct model for how a skilled human penetration tester operates. When you find that an application’s JWT validation is broken, you don’t log the JWT finding and move to the next item on the checklist. You reconsider your entire model of the application’s authentication and authorization architecture and systematically probe every component that depends on it.
Implementing that behavior in an AI system requires not just belief state, but a planning architecture that can reprioritize active investigation strategies based on intermediate findings. This is non-trivial. It requires the orchestration layer to maintain a dynamic model of what it knows, what it hypothesizes, and what it has not yet tested — and to re-evaluate that model after every significant confirmed finding. Systems that have a fixed testing sequence, or that generate their testing plan upfront without dynamic replanning, will consistently miss the cascading findings that compounding exploitation produces.
What This Benchmark Does Not Tell Us
Intellectual honesty requires acknowledging what this benchmark cannot measure, because understanding its limits is part of using it correctly.
Juice Shop is a known CTF environment. It has a fixed, well-understood vulnerability surface. It does not have the organizational context, legacy code complexity, custom authentication flows, or novel vulnerability combinations that appear in real enterprise applications. A platform that performs well against Juice Shop may or may not generalize to real-world targets, and the benchmark provides no direct evidence either way.
The benchmark also measures a single run per platform against a single target. Variance matters in agentic systems — the same platform run twice against the same target may produce different results because of non-determinism in model outputs. A single run benchmark cannot characterize variance, which means we cannot know from this data how consistent RidgeGen’s advantage is across runs, or whether Shannon and Strix might perform significantly differently on different runs.
The controlled LLM backend — while methodologically correct for isolating architectural differences — also means the results may not generalize to all model choices. A platform that performs well with one model may perform differently with another, and different model choices may interact with architectural decisions in ways that are not captured by a single-model comparison.
These caveats do not invalidate the benchmark’s conclusions. The architectural differences it reveals are real, and the failure modes it identifies — belief state amnesia, insufficient evidence validation, inability to reason about application semantics — are structural problems that would manifest across different targets and different models. But practitioners should treat the specific numbers as directionally informative rather than precisely predictive of real-world performance.
What This Means for Practitioners Building and Buying
If you are evaluating agentic AI security platforms for your organization, or if you are building one, the benchmark points to a specific set of architectural questions that you need to be able to answer in technical detail.
Does the system maintain persistent belief state across a session, and how is that state represented? A platform that cannot describe its belief state representation in technical terms — what data structures it uses, how findings are encoded, how the state is queried during planning — probably doesn’t have meaningful belief state. Marketing language about “AI that learns as it tests” is not a technical answer to this question.
Is evidence validation an architectural invariant or a best-effort output? Ask to see the architecture document or the code that governs finding generation. If a finding can be added to the output without a corresponding execution record, you have a platform with structural hallucination risk.
How does the system handle multi-step exploitation chains? Ask for a description of what happens after the system confirms a vulnerability. Does it update its testing priorities? How? Does it test adjacent endpoints under the same assumption? What triggers a reprioritization of its exploration strategy? The answers to these questions will tell you whether the system has genuine compounding capability or whether it is doing sequential pattern matching.
What is the system’s approach to business logic testing? If the answer is vague — “it uses AI to reason about application behavior” — probe further. Ask for examples of business logic vulnerabilities the platform has found in the past. Ask what information the system uses to build a model of application intent. If the system has no answer to these questions, it is not doing semantic reasoning about application behavior.
What is the true cost per confirmed finding, including manual validation overhead? Token cost is easy to measure. Get a commitment on the expected hallucination rate — the percentage of findings that will not be evidence-backed — and use that to calculate the analyst time required to validate the platform’s output. That is your true cost.
The Broader Inflection Point
The security industry has a long and largely successful history with tools that are excellent at finding what they already know to look for. Signature-based intrusion detection. CVE scanners. SAST rules. DAST fuzzers. These tools have genuine value. They find known vulnerability classes reliably, at scale, with low operational overhead. They are not agents, and they do not pretend to be. Their limitations are well-understood.
What this benchmark demonstrates — imperfectly, as all benchmarks do, but with meaningful signal — is that there is a genuine architectural divide between systems that enumerate vulnerability patterns and systems that reason about application behavior. That divide is measurable. It produces different coverage across vulnerability categories, different finding quality, different cascade effects from initial discoveries, and different economic profiles at scale.
We are at the beginning of a capability transition in automated security testing. The platforms that will define the state of the art in five years are those being built today with persistent belief state, evidence-first architectures, semantic reasoning about application behavior, and genuine dynamic replanning in response to exploitation progress. The platforms that are packaging LLM inference around existing scanner payloads will hit a ceiling quickly, and that ceiling will become visible as the real threats organizations face continue to evolve toward the multi-step, multi-layer attack chains that pattern-based tools structurally cannot detect.
For security practitioners, the near-term challenge is developing the technical vocabulary to distinguish between these two classes of tools before deploying the wrong one. Benchmark results like this one are useful data. So are architectural conversations with vendors that push past the marketing narrative into the specific technical decisions — belief state representation, evidence validation gating, orchestration layer design, reprioritization triggers — that determine whether a platform can actually reason about your application or is just telling you what it already knew.
The tools that matter are the ones that can learn something new about your application that no one else has told them. That capability is architectural, and it is now measurable. The question for every security team is whether they are asking the right questions to find it.