How Does the Security Harness Work?
An agentic penetration testing platform has to let its agents access real secrets: test credentials, session cookies, and tokens pulled from a vulnerable API.
So the question customers keep asking is a fair one:
If a pentest agent discovers a password, session token, or customer record during a test, does that sensitive data have to be sent to the LLM?
Our answer is no.
Getting to “no” is an architectural problem rather than simply a filtering problem. This post walks through that architecture, which we call the Security Harness.
Let’s start with a simple example.
A Simple Example: API Security Testing
The distinction is easiest to see in a concrete test.
Imagine an authorized API penetration test:
- The customer provides a test username and password.
- The agent signs in and receives a session cookie.
- The agent opens an account API.
- The API unexpectedly returns another user’s email address and an access token.
- The agent wants to determine whether that token can access an administrative API.
In a typical agent architecture, the workflow looks something like this:
Tool → raw result → LLM → next action
The API response is returned to the agent, placed into the conversation context, and sent to the LLM with a question such as, “What should I do next?”
That means the LLM may receive the actual email address and access token.
But the LLM does not need those values. It only needs to understand what happened.
Instead of sending:
Authentication succeeded with eyJhbGciOi…
the Harness can present the LLM with a reference value:
Authentication succeeded with <CREDENTIAL_01>.
And instead of exposing the raw API response:
Email: alice@example.com
Access token: eyJhbGciOi…
the LLM receives:
The response contained:
- an email address: <EMAIL_02>
- an access token: <ACCESS_TOKEN_03>
The LLM now knows that a token was discovered and that it may be relevant to the administrative API. It can reason about what to do next without ever seeing the actual secret.
So How Does the Agent Use the Credential?
This raises the obvious question: if the model only ever sees <ACCESS_TOKEN_03>, how does the real token ever reach the target?
This is where the Harness becomes important.
Suppose the LLM decides that the discovered token should be tested against an administrative API. It can request:
{
"name": "send_http_request",
"arguments": {
"method": "GET",
"url": "https://target.example/api/admin/profile",
"credential": "<ACCESS_TOKEN_03>"
}
}The Harness receives this request, resolves <ACCESS_TOKEN_03> to the underlying credential inside the trusted environment and passes the real credential to the authorized tool.
The target therefore receives the real token.
The LLM does not.
In simplified form:
Target → Tool → Harness → LLM
The sensitive value stays inside the trusted side of the boundary, while the LLM works with a safe reference.
That is the key idea:
The agent can use a secret without the LLM having to see the secret.
What Exactly Does the Harness Do?
Resolving a reference is only the visible part of the job.
Think of the Harness as a security boundary around the agent. Its purpose is not simply to “remove secrets,” it controls the movement and use of sensitive information between the testing environment, the agent, its tools, and the LLM.
In practice, this breaks down into five functions.
1. Detect Sensitive Information
Pentesting tools return sensitive information in many forms:
- passwords and API keys
- session cookies and access tokens
- authentication headers
- email addresses and personal records
- database connection strings
- secrets embedded in JSON or XML
- terminal output
- error messages
- screenshots
- scanner results
The first challenge is simply recognizing what should be protected.
2. Replace the Value with a Safe Reference
Once a value is recognized, it should never enter the LLM context in raw form.
The Harness replaces it with a stable reference:
<ACCESS_TOKEN_03>
The original value is retained securely inside the trusted environment. The reference allows the agent to reason about the object without exposing the object itself.
3. Preserve the Relationship Between the Reference and the Secret
This is an important difference from ordinary redaction.
If we simply replace a token with [REDACTED], the agent may know that something was removed, but it cannot necessarily use that value later.
A reference such as <ACCESS_TOKEN_03> is different because the Harness knows what sits behind it: the actual token discovered during this authorized test.
The reference can therefore persist throughout the agent run.
4. Authorize Its Later Use
Persistence is not the same as open access.
The Harness should not automatically reveal every secret whenever an agent asks for it. It should determine whether the requesting agent and tool are authorized to use that value for the requested operation.
The workflow is:
Agent discovers credential → credential stored by Harness → LLM requests credential reference → Harness checks policy → authorized tool receives real credential
Credential access therefore becomes a controlled operation rather than an exposure of the secret to every component in the system.
5. Keep the Secret Away from Other Data Paths
Finally, there is a subtler problem.
Even if the secret is removed from the LLM prompt, it can still surface elsewhere:
- debug logs
- traces
- retries
- caches
- saved artifacts
- error reports
- conversation history
- later reporting requests
Protecting the LLM prompt alone is therefore not sufficient.
The Harness has to consider the entire data path.
Why Not Simply Ask the LLM to Find the Sensitive Data?
Reading that list, the first shortcut most people reach for is the model itself.
Detection is step one, and LLMs are often very good at recognizing passwords, tokens, email addresses, and personal information.
But there is a fundamental problem.
If the rule is “do not send sensitive information to the LLM,” you cannot send the information to the LLM first and ask it whether the information is sensitive.
The filtering decision therefore has to happen inside a trusted environment.
That filter can use deterministic rules, regular expressions, local privacy models, locally hosted LLMs, specialized secret-detection tools, or a combination of these approaches.
The important question is not whether AI is used for detection.
It is whether the raw sensitive value leaves the approved environment before the decision is made.
Why a Simple Filter Is Not Enough
Ruling out the remote model leaves the deterministic filter as the obvious fallback:
“Fine, just build a good redaction filter.”
Unfortunately, it is not that simple.
Pentesting data is messy.
A real security finding can contain IP addresses, UUIDs, hashes, file paths, commands, CVE numbers, unusual strings, and other information that may look sensitive but is actually essential to the test.
If the filter removes everything suspicious, the agent becomes less useful.
If it is too relaxed, it may miss the very secrets it was designed to protect.
More importantly, redaction alone does not solve authorized reuse.
A pentest agent may legitimately need to use a credential several steps after discovering it.
That is why the Harness needs to do more than hide information. It needs to provide a controlled mechanism for the full chain:
detect → protect → reference → authorize → use
What Can We Use Today?
None of this means starting from zero.
There are already useful open-source and commercial building blocks for parts of the problem:
- Presidio can identify and anonymize many types of personal information.
- OpenAI Privacy Filter can identify sensitive information locally before text leaves the environment.
- LangChain PIIMiddleware provides blocking, redaction, masking, and hashing capabilities for agents.
- Private AI provides tools for detecting, replacing, and recovering sensitive information.
- Skyflow provides vault-based tokenization and controlled reveal.
- Gitleaks and TruffleHog focus on detecting credentials and secrets.
- Cloud services such as Amazon Bedrock Guardrails, Google Sensitive Data Protection, Cloudflare AI Gateway DLP, and LangSmith Gateway data protection provide related capabilities.
Each of these covers part of the chain.
That points to the distinction that matters:
Finding and hiding a secret is not the same thing as securely managing a secret throughout an agent’s lifecycle.
A complete agent Security Harness has to connect all of it:
1. Find sensitive information.
2. Replace it with a safe reference.
3. Store the original securely.
4. Control which agent or tool can use it.
5. Retrieve the original value only when an authorized operation requires it.
6. Prevent the value from appearing through other data paths.
What Do Existing Benchmarks Tell Us?
If the building blocks are partial, the natural next question is how we would measure whether a Harness actually works.
There is growing research around redaction, secret detection, privacy, and agent security.
Benchmarks such as RedactionBench, SecretBench, CredData, the Text Anonymization Benchmark, AgentDAM, CI-Work, AgentDojo, InjecAgent, ToolPrivacyBench, PrivacyPeek, AgentLeak, and PIIBench each address pieces of the problem.
They help answer questions such as:
- Can a system identify sensitive information?
- Can it detect credentials?
- Does an agent collect information it does not need?
- Can prompt injection cause an agent to leak private information?
- Does an enterprise agent share information appropriately?
These are valuable tests, but they stop short of the complete workflow we care about:
A tool returns a sensitive value → the Harness keeps the raw value away from the LLM → the LLM receives enough information to reason correctly → the agent requests an approved action → the Harness authorizes and retrieves the value → the trusted tool uses the original credential.
That complete workflow is the real security boundary we need to evaluate.
The Bigger Idea: Permission Is Not Exposure
The workflow above is specific to penetration testing, but the principle behind it is not.
An AI agent may legitimately have access to sensitive information because it needs that information to complete a task.
That does not mean every component involved in the task should receive the raw value.
The human analogy is close enough to be useful.
A penetration tester may be authorized to use a customer’s credential. That does not mean the credential should be pasted into every document, chat window, logging system, or third-party service the tester uses.
The same principle should apply to AI agents:
Permission to use a secret is not permission to show that secret to every component in the system.
Enforcing that distinction is the purpose of the Harness.
The agent can discover a credential.
It can preserve the credential.
It can reason about what the credential might enable.
It can use the credential when an authorized test requires it.
But the LLM does not need to know the credential itself.
That is the security boundary we believe agentic security platforms need to build:
Let the agent have the capability it needs, without unnecessarily exposing the underlying secrets to the model.