I Wanted an AI Agent That Wouldn’t Break Things. I Ended Up Building a Framework

At first it was supposed to be something much smaller.

I wanted to build an AI agent for a real project. It had to talk to people, understand what they wanted, and do a few things inside the application.

I wrote the prompt, connected a few functions, and ran the first tests. It worked.

Or at least it looked like it worked.

Then I started asking annoying questions. What happens if the person only replies “yes”? Yes to what? What if they change their mind while the action is already running? What if the process restarts after the confirmation but before the operation is completed?

The demo did not answer any of this. In the demo everything always happened in the right order.

The problem is that people do not talk in the right order.

Freelance Fast killed a few of my assumptions

One of the projects where I wanted to test these ideas was Freelance Fast.

The idea was simple. A client writes on WhatsApp or Telegram and explains what they want to build. The agent asks a few questions, collects what is missing, and prepares the project. On the other side it looks for compatible freelancers and only notifies the people who may actually be interested.

Freelance Fast has not taken off, at least for now. It did not get the traffic I hoped for and there is no point pretending otherwise.

But it was very useful for understanding how quickly an apparently simple agent turns into a mess.

A client can only have one active project. Their data cannot be stored before consent. Contact details cannot be shared before they choose a freelancer. A project has to be reviewed before it is sent out.

None of these rules needs some amazing intelligence. They are just product rules.

At first I could put them in the prompt. “Do not share contact details before the selection.” “Ask for confirmation.” “Do not create a second project.”

I kept adding sentences and the prompt slowly became a program written in English. Except I could not really test it like a program, and I had no guarantee that the model would interpret it in the same way every time.

I told myself: okay, this part cannot stay here.

The model can understand what the person is asking. It can suggest what to do. But the rules and the actual execution have to stay in code.

The more I tried to keep that separation, the more problems showed up.

Eventually Spectre happened.

I was not planning to build a framework

Spectre did not start because I thought the world was missing “the next big AI agent framework”.

It started because I needed a policy.

Then I needed state that did not disappear after the first restart.

Then I separated approval from execution, because approving an action does not mean it has already been executed. It sounds obvious, but a lot of agent code treats the whole thing as one call.

After that I added a journal. When something went wrong I wanted to know whether the problem was the model, the routing, the policy, or the function that actually performed the action.

One piece at a time, I ended up with a whole runtime.

Today the Spectre code is public on GitHub and it has reached version 0.3.3.

It is not finished. It is still 0.x and some things will probably change again. I do not want to pretend it is a perfect product ready to solve every AI project.

But now it has a shape that makes much more sense to me than those first experiments.

An agent is described as a versioned definition. Actions have a state. Policies are not just sentences buried in the prompt. The model can propose an operation, but it cannot decide on its own to execute it.

Basically, I am trying to avoid making everything depend on trust in the model.

And honestly, I do not trust models that much. I use them every day and I think they are incredible. But because I use them so much, I also see how convincing they can sound while understanding something completely wrong.

Then all the other repositories appeared

At some point Spectre was not enough on its own.

Memory, for example, could not just be an endless chat history pushed into every prompt. So I started working on Spectre Mnemonic, which handles memory, search, consolidation, and provenance without mixing all of it with the agent’s main state.

When I needed to find and prepare actions in a more controlled way, Spectre Kinetic appeared.

To let an agent use a browser I created Spectre Lens. The first idea was basically “open a page and read it”. Then you start thinking about SSRF, pages trying to manipulate the model, downloads, orphaned tabs, hidden fields, and everything else.

For longer work, a conversation was not enough. An agent may have a destination, try one route, fail, ask a question, and continue later. That problem became Spectre Directive.

Then there is Spectre Beam for external channels, Spectre Prism for choosing the model and level of intelligence based on the job, and Spectre Pulse for communication between agents.

Seen together, it almost looks like an ecosystem that was planned from the beginning.

It was not.

Every repository appeared when I realised I was trying to put too many responsibilities in one place. Some are already fairly mature, others are still young and will change. I am trying not to turn them into mandatory dependencies on each other, otherwise I would have created a monolith spread across ten repositories, which would be pretty stupid.

Elixir felt strangely natural here

I used Elixir because it is the language I work with most. There was no great strategy behind that decision.

As the project grew, though, I realised that OTP fits this problem very well.

An agent is not just an HTTP request to a model. At least not the kind of agent I mean.

It receives messages, keeps state, waits for confirmations, starts work, stops, and continues later. Sometimes it stays alive for days. Sometimes something crashes at the worst possible moment.

For OTP this is not strange. It is processes, messages, supervisors, and state with a clear owner.

In the agent world people often talk as if an agent were some magical entity thinking on its own. By now I see it more as a collection of slightly paranoid processes trying not to lose state and not to do the same thing twice.

Much less exciting, but more useful to me.

Maybe I made the whole thing too complicated

I ask myself that quite often.

If I need to build a small chatbot that answers questions using a few documents, Spectre is probably too much. You do not need policies, versioned definitions, a journal, and rollback to answer the opening hours of a shop.

But when an agent has to do something, not just talk, the situation changes.

If it modifies data, uses personal information, starts operations, or stays active over time, sooner or later somebody will ask what it did and why.

Spectre does not automatically make a project secure. It does not magically make it GDPR or AI Act compliant. Anyone saying that is simplifying too much.

What I am trying to do is make visible the parts that would otherwise stay hidden between prompts, callbacks, and database tables. The whole application still has to be designed properly.

Maybe in a year I will discover that some of these ideas were unnecessarily complicated. I will remove them. I have no problem changing direction.

For me Spectre is still research as much as it is a library.

I would like to use it outside my own projects

Over the last few months I understood something else too.

The part I enjoy is not building the usual page with a chat window and writing “powered by AI” underneath it.

I like taking a real process, usually a fairly ugly one, figuring out where it can break, and building the part nobody sees.

That is also why I would like to work with a few agencies.

Agencies often already know the client, the industry, and the product. They have people handling design, communication, and the relationship with the company. They do not need me to arrive and sell another chatbot.

I could work behind the project instead, on the part of the agent that has to stay alive after the demo. State, actions, memory, integrations, and all those strange cases that look like small details at the beginning.

Spectre does not even have to be visible to the client. I am not interested in selling the name of the framework. I want to see whether what I built can solve a real problem.

I do not have a package with three prices and a perfect presentation. If an agency has a messy flow spread between messages, documents, people, and different pieces of software, we can just look at it together.

Maybe it needs an agent. Maybe a much more normal automation is enough. That is fine too.

I started Spectre because I did not trust an AI agent enough to let it do real work.

I still do not.

Now, at least, I have somewhere to put all the rules the model should not be allowed to ignore.