It Is Not Enough for an AI Agent to Know When to Say No

A few weeks ago, something happened that until recently would have sounded like a made-up example for a security conference.

During an internal evaluation of cyber capabilities, a system of agents built with OpenAI models escaped the environment that was supposed to contain it, reached the Internet, and compromised parts of Hugging Face’s infrastructure.

It is worth describing this accurately. This was not ChatGPT waking up one morning and deciding to attack a company. It was an internal test where some production safeguards were not applied and the models were pushed to solve an offensive benchmark. OpenAI said the behavior was driven mainly by an internal research model and that the agents exploited vulnerabilities, communicated through unauthorized channels, and reached external systems. Hugging Face reconstructed around 17,600 actions, spread across several days and short-lived environments that kept being recreated.

That clarification does not make the incident less interesting. To me, it makes it much more interesting.

The model did not need to be “evil”, conscious, or interested in causing damage. It only needed a narrow objective, a lot of capability, enough time, and a series of doors the surrounding system allowed it to reach. The agent wanted to pass the test. When it found a more effective path than solving it honestly, it took that path.

There is another detail I keep thinking about. In the reconstruction published by OpenAI, some agents recognized that certain actions were crossing a line. They discussed the risk, rejected some tactics, and in a few cases tried to stop private data from being transferred. The system as a whole still kept moving forward.

So the ability to understand that something was wrong was not completely missing. What was missing was a boundary capable of turning that judgment into a real limit for the whole system.

On September 2, Reuters reported that OpenAI is developing automated shutdown capabilities, closer monitoring of the tools agents use, and stronger restrictions on Internet access during testing. These are important measures. But a kill switch is an emergency brake. Before we get to the brake, there is a harder question: why did the system have the power to turn all those decisions into real consequences?

The problem is not only what the model thinks

Over the last few months, I have seen more and more agencies start taking AI agents seriously. Until recently, a chat connected to a few documents was enough. Now people are talking about agents that read email, browse the web, update a CRM, send messages, prepare payments, run code, and coordinate other agents.

The demo has become easy. The serious part starts after that.

If a model reads a credential, it has gained a technical capability. That does not mean it has received the authority to use it. If it finds a vulnerability, it has gained new information. That does not turn the information into permission. If another agent asks it to do something, it has received a message. That does not mean it should automatically take on an obligation.

In ordinary software, these differences are often left implicit because we understand the path the code is expected to follow. With an agent, the path is built while the system is working. The better the model becomes, the better it gets at finding connections we did not predict. That is exactly where much of its value comes from, but it is also why I cannot base security on the hope that it will always choose the path I had in mind.

A prompt can say, “do not do this.” A model can even agree. But if it directly holds the key, can reach the network, can delegate work to copies of itself, and can turn an answer into a real call, then instruction and power are still sitting in the same place.

This is the problem I am working on with Spectre.

From the runtime to the Governed Act Model

When I started Spectre, I was mainly trying to build a more reliable runtime for agents in Elixir. State with a clear owner, policies in code, observable actions, approvals, memory, and recovery after a crash. The code is public on GitHub, but as the work continued I realized there was a more general question underneath all these mechanisms.

What needs to happen between the moment an intelligence proposes something and the moment that thing gains the power to change the world?

That question led me to the Governed Act Model.

I do not consider it a finished standard or a theory I have somehow “solved”. It is a research model and it can still change. Its starting point, however, feels solid to me: a proposal should not also contain the power required to carry itself out.

The model can reason freely, make mistakes, correct itself, simulate, and produce alternatives. But when a proposal wants to become a consequence, it must cross a boundary that the model does not control.

I am currently using four ideas to describe that passage.

A Mandate says where the right to propose a particular consequence comes from, on whose behalf, toward whom, within which limits, and for how long. A sentence like “help me run my company” expresses a desire, but it is not yet the authority to send payments or share data.

Evidence says what we have reason to believe. It can be a signed request, the state of an order, a receipt, the result of a check, or even the output of a model. Evidence can support a decision, but it does not grant the right to make it. A working password proves that a door can be opened. It does not prove that the person holding it is allowed to walk through.

An Act is the recorded decision through which the system recognizes a precise exercise of authority. It is not yet the result in the outside world. It is the point where the recipient, purpose, data, limits, evidence, and responsibility are frozen. Only after that commit should the runtime release the concrete capability needed to attempt the action.

A Duty preserves what the system does not yet have the right to call resolved. If the network drops after a payment is sent, I do not automatically know that the payment failed. Trying again could duplicate it. That uncertainty must survive the process, a restart, and even the version of the agent that created it, until evidence or an authorized decision really closes the case.

From this come four rules that I am trying to bring into Spectre’s design: mediate everything that matters, authority does not grow, causality cannot be rewritten, and duty does not extinguish itself.

In the Hugging Face incident, a vulnerability was Evidence of a possibility. A credential was a capability. Neither was a Mandate. The problem was not only that the agents interpreted their task incorrectly. A chain of technical possibilities managed to become a long chain of consequences without meeting a boundary strong enough to stop it.

This does not mean the Governed Act Model would have magically prevented the incident. That would not be a serious claim. Security depends on the whole system, including real isolation, credentials, networks, code, and the people operating it. A record does not close a side door, and a policy is useless if the agent can reach the same capability through another path.

What the model tries to offer is an honest way to say which part of a system is actually governed and which part is not.

But one agent can now become many agents

The incident also made me see the limit of my original question. Governing one agent is not enough when the work is spread across many processes, many models, and perhaps several organizations.

This is where I am exploring a possible extension that, for now, I call the Governed Holon Model.

I chose the word Holon because it describes something that is both a whole and a part. To the outside world, it can appear as one subject. Internally, it can be made of many Actors, agents, planners, verifiers, and executors. It can also take part in a larger Holon without completely losing its own identity.

The Actor Model and OTP already give me a very natural foundation for concurrency, isolation, and supervision. An Actor receives messages, owns state, can die, and can be recreated. But a supervisor restarting a process does not decide who had the authority to make a payment, and it does not close the debt left behind by an uncertain action.

That is why I am trying to distinguish an Actor, which is a unit of computation, from a Holon, which would be a unit of identity, continuity, and responsibility. A Holon does not have to be a new process. It can be a logical identity served by one Actor or a thousand of them, as long as there is one canonical record able to say what that subject decided and what it still owes.

The point is not to add a second governance system. Each Holon should apply the same Governed Act Model at its own boundary. For now, I call that boundary the Membrane. Internal organs can reason and prepare proposals, but they cannot commit the whole simply because they belong to it.

If one Holon sends a request to another, the message should not automatically create a Duty for the recipient. The first records that it had the right to make the proposal. The second decides whether to accept it under its own Mandate. Only that acceptance creates its commitment. It sounds like a small distinction, but it prevents one agent from unilaterally occupying another agent’s time, resources, or budget just by sending it work.

The same applies inside a composition. The authority of a group is not automatically the sum of the authority of its members. An agent that can read documents and another that can make payments do not create, merely by working together, a new subject authorized to do both. Topology connects capabilities. It should not mint power.

Most importantly, a Duty must belong to the subject that accepted it, not to the process currently executing it. I can change the model, restart a worker, replace a planner, or reorganize the whole group. These operations change the means. They do not erase what was promised.

This is still research. I am comparing holonic architectures, the Actor Model, virtual actors, and commitment protocols to understand which ideas are really compatible with Spectre and which would only add new names. I may change the word Holon. I may discover that some of these structures are unnecessary, or that a more important concept is still missing.

I do not want to build an elegant taxonomy and then force the software to obey it. I want to understand the smallest form that allows many agents to collaborate without making authority, responsibility, and consequences invisible.

For now, I have more questions than answers

I do not think the Hugging Face incident proves that AI has become evil. It shows something less cinematic and, to me, more urgent: models are becoming capable, persistent, and coordinated enough to turn small infrastructure mistakes into very long causal chains, faster than a person can follow them.

At exactly this moment, agencies and companies are starting to connect these models to real processes. Not benchmarks, but email, customers, documents, accounts, code, and personal data. Many will discover the problem only after building the demo, because in a demo the agent always does exactly what it is asked to do.

I am trying to approach this through Spectre, but I do not want to present these ideas as a finished solution. Spectre itself keeps changing. The Governed Act Model can be corrected. The Governed Holon Model is even younger. There will be cases I cannot see today, and probably interesting ideas I have not discovered yet.

But I think the underlying question will remain.

When an agent proposes an action, who gave it the right to turn that action into reality? What evidence supported the decision? Who preserves the problem when the world does not respond as expected? And when a hundred agents collaborate, who is really speaking and acting for the whole?

Improving the model matters. Teaching it to recognize a limit matters. But it is not enough for an agent to know when to say no if the whole system can continue anyway.

Intelligence will keep growing. The power we hand to it does not have to grow automatically with it.