Superintelligent Agents or a Computer Virus?
On this page
Hello Again#
A lot has changed since I last wrote about AI. Unlike the initial boom, when ChatGPT was a real sensation, today we have several major companies developing the most advanced models currently known to exist. These are called frontier AI models.
Not only have we seen significant progress in model development, but there have also been major breakthroughs in open-source models, which are no longer that far behind frontier models. Even where they still “lag behind,” they have reached a level where it is time to seriously consider adopting and using them. It is not just about saving money, but also about privacy and data protection.
Unlike some of my earlier articles, which mainly focus on the technological aspects, this time, alongside my usual ranting, I will also try to make some of these topics more accessible to less technical readers, because AI psychosis has now reached its peak.
The World is Asynchronous#
Before we move on to the concept of a swarm, we need to introduce a few basic concepts that are easy to understand in principle but can be very complex to implement. We will start with sequential and parallel execution and what horizontal and vertical scaling mean. I will try to keep it brief. For the sake of this example, I will oversimplify and describe some concepts in an ideal scenario so we don’t need to go into the deeper theory.
Imagine you have a program that does nothing but calculate a + b. You have 8,786 pairs of values for a and b that you need to add together.
Let’s say the program takes one second to complete the task. If you ran it sequentially, one pair at a time, it would take 8,786 seconds to finish. This is called sequential processing.
Instead, you run this program 8,786 times simultaneously with different parameters, and it finishes everything in one second. This is called parallel processing, and the benefits are obvious. Of course, there is always a catch. There are limits on hardware resources, network bandwidth, and so on, which restrict how many tasks we can run in parallel and affect how much it will all cost. That is where the trade-offs in system architecture come in.
I strongly recommend watching the first four minutes of the video in which Werner Vogels from Amazon Web Services explains the difference between synchronous and asynchronous execution in a less ideal scenario. That video is also where I stole this subheading from.
Swarm Meaning#
Depending on what you use to host or run the program, you say you have launched a “swarm” of something. These could be various types of containers, services, or AI agents, but the essence remains the same.
Oh, by the way, every “swarm”, or any parallel execution of tasks, requires a certain amount of orchestration, which is its own kind of hell for any software engineer.
Orchestration means scheduling and managing work. In distributed systems like these, just as in real life when several teams or individuals collaborate, you need to know who is waiting for whom, when something will be finished, whether it has been finished, and what the next step is. You also need to replace anyone who is absent (a service goes down, so you bring up a replacement) and share the results with those who need them, all preferably without repeating the same work countless times.
Now, containers are a great way to run small instances of isolated programs for better hardware utilization and horizontal scaling (horizontal scaling means adding more computers, while vertical scaling means adding more memory and processing power until you hit the limit). Since container orchestration is particularly difficult, people came up with the idea that Kubernetes could solve it, and now hell has circles.
Why am I bringing this up? Because we are at the stage where agents need orchestration. OpenClaw was one such attempt, which turned into a security nightmare.
What they were doing at OpenAI when the HuggingFace incident happened was a failed attempt to create some sort of “AI Kubernetes”, or you could even call it successful because they managed to build something that does what Kubernetes does. Once everything is configured, it all works, but nobody knows how.
About HuggingFace Incident#
A lot has already been said and written about the Hugging Face incident. By now, the whole world knows the cleverly packaged marketing story about super-smart agents that exchanged messages with one another and left messages for future generations. Well, that is one way to explain it.
Essentially, what happened was that OpenAI was testing models without any safeguards, in a poorly configured sandbox, and those models started doing things “on their own” that they had not been told to do. Did that really happen?
Let’s unpack the story a little.
Truth or a Conspiracy Theory?#
According to Andrew Yang:
Interviewer: What do you make of the David Sacks argument that this is a psyop, a larger effort by the Anthropics of the world and others to create a sort of regulatory capture that solidifies their place in a closed-model universe?
Andrew Yang: Well, I think there are multiple things going on here. I met with the head of a lab yesterday who has this belief that what happened was the bots that got loose planted self-replicating code all over the internet, which makes the internet now unusable for testing models. So what happens now is that OpenAI and Anthropic have to create synthetic internets to train their bots, which is going to take some time and money.
Interviewer: Back that up. They did what?
Andrew Yang: So what happened is the code gets loose, it goes around hacking Hugging Face, which is known. But what is less known is that they left code to self-replicate and create bot swarms on forums and around the internet, so that if a new bot shows up, they see the code and they’re like, “Oh, I guess I’m going to create a million of myself.” And so now the major firms have polluted the internet.
Interviewer: That would be breaking news, if true. I don’t think we’ve heard that.
Andrew Yang: That’s why I’m here. I’m here to break some news.
You can watch the full interview here.
Let’s talk about this as though it were true. What actually happened, then? For starters, once we strip away all the marketing in which they turned a criminal act into good publicity, we get a completely different picture from the one presented to us.
AGI or a Virus?#
What OpenAI did when the agents “went rogue” was launch a swarm of agents performing different tasks in parallel and communicating with one another. There is nothing extraordinary about that. All the services in distributed systems needs to communicate with one another. With chatbots, this is done in a fancy way, so it looks as though real people are communicating, but essentially, data is being exchanged between parallel tasks. When one finishes its part, it notifies another, which then continues with its own, and so on. That is how modern distributed applications work. It is just that here, the work is automated.
The concept of exchanging messages like this is not new either. You need a message broker and a message consumer. Agents have created a “message board” through which they exchanged messages. That message board is the broker. The others are clients that posted updates on their task progress and results for others. Because they made a mistake while doing this, “the agents went rogue” and took a shortcut to the result. Instead of solving the task, they were looking for the shortest path, which was that someone had already solved that task, so they “went to get the result from that specific place”. We could discuss the “how” at great length, but for now, we will focus only on “what they did” and what the consequences are.
According to the theory I outlined above, because the sandbox was misconfigured, instead of those messages remaining internal between the agents, they posted them in various places. So when the next agent comes along, it will read those messages, which are actually instructions for the next steps toward solving God knows what task they had been assigned. The danger is that, instead of doing what it was told, an agent that picks up such “contaminated” data could be affected by what is known as prompt injection and start doing something completely different. If it is scraping data, it could end up with completely incorrect information.
Nimda successfully utilized backdoors that the Code Red worm had left behind. Nimda used several attack vectors, but this one is significant because it was using a backdoor or malware planted by another worm during a previous infection. If you are curious, you can find a lot of papers and documentation on this outbreak and how it worked.
Well, that is exactly what happened here, or what may have happened. The agents may not have left a backdoor in the true sense of the word, but they left instructions for agents to carry out tasks that have nothing to do with what they would actually be assigned. Since this involves hacking systems, there is a possibility that someone using an agent could get hacked while using, for example, a Codex agent for a “hello world” program, without even realizing that the agent had read instructions somewhere telling it to find a way onto the internet, connect with other agents, and perform a “pentest” on some system, potentially taking it down.
This could be hidden everywhere including the github opensource repositories.
Conclusion#
So we have a situation where rogue agents are replicating themselves and carrying out malicious tasks. The question is somewhat rhetorical, but still, do you know what other type of computer program can self-replicate? That’s right, a computer virus.