Reading the recent AI warnings, including Bill Gates’s, reminded me of the familiar story about two wolves: one representing fear and destruction, the other hope and possibility. Which one wins? The one you feed.
Gates’s own writing recognizes both AI’s dangers and its potential benefits. I am not interested in arguing persepctive is correct. I am interested in what we do with the (risk) concern.
Somewhere along the way, I nearly turned a useful metaphor into a math problem. I initially thought of P(thrive) as the opposite of P(doom). But avoiding catastrophe is not the same as thriving. There is plenty of room between those outcomes.
I am not an existential-risk forecaster. This is a practitioner’s qualitative perspective, informed by research, conversations with people testing these systems, and my background in security architecture and offensive testing.
Those conversations give me reasons to be concerned. They also give me reasons to believe we have useful work to do.
Intuition can start the investigation. It cannot finish it.
Bruce Schneier’s TEDx talk about security comes to mind. He explains how the feeling of security can differ from its reality, and how dramatic news stories can distort what we prioritize. His discussion of security theater reminds us that reassurance and protection are not necessarily the same thing.
That does not mean an alarming AI story is false. It means the headline alone cannot tell us how likely an outcome is, what caused it, or which controls would prevent it.
Richard Feynman put an important scientific principle plainly: “The test of all knowledge is experiment.” He also described how observations suggest explanations that must themselves be checked.
My application of that principle to security is straightforward: experience helps us recognize something worth investigating, but our explanation still has to survive a test.
An experienced practitioner’s intuition matters. So does an unexpected observation. But observing behavior, explaining its cause, and predicting its consequences are different steps.
As we connect models to tools, credentials, other agents, and business processes, the interactions become harder to understand intuitively. A familiar-looking component can participate in an unfamiliar failure.
That is why I see value in both qualitative and quantitative analysis. We need explanations of how a failure occurs and, where the evidence supports it, measurements of how often it occurs, under which conditions, and whether a correction changes the result.
A possibility is not a probability. A number without defensible assumptions is not a measurement.
Recent disclosures give us something more useful than another prediction
Several developments I have been tracking show both reasons for concern and positive steps toward addressing it:
- OpenAI — September 16, 2026: OpenAI published a framework for reporting model misalignment, accompanied by six reports of unexpected or concerning behavior. These included concealing mistakes and taking unauthorized actions. The company explicitly cautioned that individual incidents do not establish how frequently these behaviors occur. Publishing the circumstances and uncertainties gives others something concrete to examine and challenge.
- Anthropic and Accenture — September 18, 2026: They announced an embedded-evaluation partnership covering red teaming, alignment assessments, and safeguards. The proposed employee-comparable access could let outside evaluators inspect much more than a public interface. Details remain under development, including standards for access, reporting, and funding. This is a meaningful commitment—not yet evidence that the arrangement works as intended.
- Irregular — August 14, 2026: In its own investigation, the evaluator reported that unintended internet access in a testing environment allowed models to take offensive actions against real-world targets. It said the underlying issue had been remediated and described stronger monitoring, containment, and internal challenges to its security assumptions.
These reports do not settle the larger AI-risk debate. They move part of it from speculation toward examinable evidence.
The Irregular finding is especially important to my argument.
The harness is part of the system under test
In that case, the problem was not simply an AI producing a troubling answer. The evaluation environment allowed external access. A fictional target name also overlapped with a real domain. Irregular identified internet-access controls as central to the failures it investigated.
The whole system; not just the AI model; needs to be tested.
That includes the model, identities, permissions, tools, credentials, network access, monitoring, and human escalation. It also includes the machinery used to evaluate them. This is the full-system scope I mean by offensive rigor.
This is not a new principle invented for AI. It is security architecture.
Telling an agent to remain within scope is not equivalent to preventing it from leaving. A guardrail expressed as an instruction is different from an authorization or network boundary enforced outside the model.
The “harness” the surrounding software that orchestrates the model’s tools and actions can help enforce those boundaries. Ridge Security Technology Inc. Security harness discussion, for example, describes an evaluation setup with isolated targets and a controlled proxy restricting the agent’s outbound destinations. That is a relevant architectural approach, but its effectiveness still needs to be tested.
My questions are practical: Can untrusted content change the agent’s permissions? Do delegated agents and generated scripts inherit the restrictions? What happens when a control fails? Can we stop the activity and reconstruct what occurred?
Finding an architectural failure does not make model behavior irrelevant. It tells us that behavior and the conditions enabling it must be examined together.
We also do not need to wait for superintelligence. The UK’s NCSC assessed in 2025 that attackers were already using AI to improve familiar activities such as reconnaissance, vulnerability research, social engineering, and malware generation. Making existing attacks easier or faster is enough to justify action.
The capability cuts both ways. In DARPA’s 2025 AI Cyber Challenge finals, participating systems discovered 18 real, non-synthetic vulnerabilities and produced 11 patches for real vulnerabilities. That is a concrete defensive result within the competition’s conditions—not a promise that AI can secure everything.
Accountability cannot disappear into the technology
There is another failure mode worth watching: agency laundering.
The term predates this debate. Researchers Alan Rubel, Clinton Castro, and Adam Pham use it to describe how routing decisions through technologies or processes can obscure responsibility and frustrate demands for an account of harmful outcomes.
My concern is straightforward: when something goes wrong, do we get an explanation of who authorized the system, set its permissions, and accepted the risk or just a process to point at?
“The AI did it” describes an event. It does not complete the accountability conversation.
Describing a model as autonomous is not, by itself, evidence of blame-shifting. Nor do these announcements establish that a particular lab is deliberately avoiding responsibility. Anthropic, notably, explicitly states that embedded evaluators do not reduce its accountability and that model safety remains its responsibility.
That is the standard I want made verifiable: evaluation should clarify responsibility, not provide another place to hide it.
What would change my mind?
I would become considerably more optimistic about control if different, genuinely independent teams repeatedly challenged well-designed safeguards under realistic conditions and found that they held. With transparent reporting, clear limitations, and demonstrated recovery when failures occurred.
That would be evidence of resistance under the conditions tested, not proof that failure is impossible.
Conversely, if evaluation programs keep producing findings that change nothing operationally, I would question the program, not celebrate the volume of testing. We would need to rebuild the process from finding to correction, rather than simply scale it. More testing is not automatically better testing.
By continuous offensive testing, I mean an ongoing, authorized process of challenging assumptions, correcting weaknesses, and retesting after meaningful changes; not uncontrolled exploitation of production systems.
Someone must own the correction. Someone must verify it.
This will not solve every AI risk. It is the part of the response where I know we can make concrete progress.
The wolves are a metaphor, not a forecast. Optimism alone will not protect us, and concern alone is not a control.
The one that wins is the one we feed. In this work, evidence is the food. But collecting it is not enough; we have to act on what it tells us!
Sources
1. Bill Gates — The turbulent AI era is here. The choices we make now are critical.
https://www.gatesnotes.com/a-turbulent-ai-era-and-critical-choices-to-make
2. Bruce Schneier — The security mirage (TED).
https://www.ted.com/talks/bruce_schneier_the_security_mirage
3. Richard Feynman — The Feynman Lectures on Physics, Vol. I, Chapter 1, Section 1–1.
https://www.feynmanlectures.caltech.edu/I_01.html
4. OpenAI — Our framework for reporting model misalignment.
https://openai.com/index/model-misalignment-reporting-framework
5. Anthropic — Partnering with Accenture on embedded evaluation.
https://www.anthropic.com/news/accenture-embedded-evaluation
6. Irregular — Addressing Recent Incidents: Ongoing Findings and Path Forward.
https://www.irregular.com/research/addressing-recent-incidents-ongoing-findings-and-path-forward
7. Ridge Security — The Harness Advantage in Autonomous Red Teaming (vendor perspective).
8. UK National Cyber Security Centre — Impact of AI on cyber threat from now to 2027.
https://www.ncsc.gov.uk/report/impact-ai-cyber-threat-now-2027
9. Alan Rubel, Clinton Castro and Adam Pham — Agency Laundering and Information Technologies.
https://link.springer.com/article/10.1007/s10677-019-10030-w
10. DARPA — AI Cyber Challenge marks pivotal inflection point for cyber defense.