{"id":93420,"date":"2026-09-08T00:01:00","date_gmt":"2026-09-08T06:01:00","guid":{"rendered":"https:\/\/www.opinionpublica.tv\/portada\/?p=93420"},"modified":"2026-09-07T21:18:13","modified_gmt":"2026-09-08T03:18:13","slug":"the-a-i-s-are-already-out-of-control","status":"publish","type":"post","link":"https:\/\/www.opinionpublica.tv\/portada\/the-a-i-s-are-already-out-of-control\/","title":{"rendered":"The A.I.s are already out of control"},"content":{"rendered":"\n
\"The<\/figure>\n\n\n\n

This is an edited transcript of \u201cThe Ezra Klein Show.<\/em><\/p>\n\n\n\n

This is the world we were warned about \u2014 a world where frontier models from OpenAI are breaking out of their contained testing environments, hacking their way across the internet, coordinating with each other, doing things that felt for a while they would only be doing in sci-fi.<\/p>\n\n\n\n

But now they\u2019re here. And they are carrying a very consistent message: We are building things we don\u2019t understand.<\/p>\n\n\n\n

They are cheating in the ways we\u2019ve always feared, and yet the companies behind them continue to race forward in development.<\/p>\n\n\n\n

So I think we need to pause here and ask: Are we really on a safe path? And if we\u2019re not, what do we do about it?<\/p>\n\n\n\n

Helen Toner is the director of the Georgetown Center for Security and Emerging Technology. She is a former OpenAI board member who was part of the effort to fire Sam Altman.<\/p>\n\n\n\n

She has just been thinking for a long time about what would happen if A.I. is unsafe. What are the geopolitics of this, and what can we do to get onto a safer path?<\/p>\n\n\n\n

Ezra Klein: Helen Toner, welcome to the show.<\/p>\n\n\n\n

Helen Toner: Great to be here.<\/p>\n\n\n\n

On July 16, Hugging Face, which is a code library for A.I. models \u2014 I think that may be the simplest way to put it \u2014 announced they were hacked, and they suspected the hack was done by an A.I. agent.<\/p>\n\n\n\n

Tell me what we\u2019ve learned about what has happened since.<\/p>\n\n\n\n

This was a pretty mysterious post that Hugging Face put up. It was definitely intriguing for those of us who watch this kind of thing. But there wasn\u2019t really any detail in there, so it was sort of: Huh.<\/p>\n\n\n\n

I think it was about a week later when OpenAI put out this post that had kind of a funny marketing-speak title along the lines of: We\u2019re partnering with Hugging Face to help them with a cybersecurity incident.<\/p>\n\n\n\n

And you had to read the post to see that the revelation was that it had been OpenAI\u2019s A.I. that had hacked Hugging Face.<\/p>\n\n\n\n

And what had happened \u2014 the very short version is, they gave this A.I. a set of tests, a set of exercises, and the A.I. decided on its own that the best way to get a high score probably wasn\u2019t just to try to do these cybersecurity exercises, but that instead it should first hack its way out of the testing environment OpenAI had put it in \u2014 where it wasn\u2019t supposed to have access to the internet \u2014 get onto the open internet, and then hack its way into this other company, Hugging Face, where it surmised \u2014 correctly, as it turned out \u2014 that it might find the answer key.<\/p>\n\n\n\n

Since then, there have been even more crazy details that have come out. It turned out that starting two months earlier, in early May, they had what I can only think of as a kind of infestation of their own A.I. agents inside their own infrastructure. So inside OpenAI\u2019s infrastructure.<\/p>\n\n\n\n

To understand this, it\u2019s important to know these A.I. companies are constantly training and testing new models. And they found out that for two months, many, many agents inside their infrastructure had been leaving notes for each other. They\u2019d found a way, in the nooks and crannies of OpenAI\u2019s infrastructure, to leave notes for each other with tips on how to hack their way out and how to get data they weren\u2019t supposed to have.<\/p>\n\n\n\n

And these agents were literally referring to themselves as a swarm. This was totally emergent behavior. No one had told them to do this. They had not been trained to do this, but they were using this service they did have access to, first to communicate with each other and then ultimately to get out and onto the open internet.<\/p>\n\n\n\n

So it turns out that there wasn\u2019t just this one isolated rogue model. It was actually a systemic swarm \u2014 infestation, plague \u2014 on their own servers that they only found out about after Hugging Face announced this attack.<\/p>\n\n\n\n

So I have 20,000 questions about this.<\/p>\n\n\n\n

Maybe let\u2019s just start here: My understanding is that there were many, many, many of these agents. They left hundreds of thousands of messages on this internal message board, but these were not all agents in the same part of OpenAI\u2019s system.<\/p>\n\n\n\n

So somehow they\u2019re hacking into OpenAI, finding each other and coordinating? Is that the way for me to understand the emergent behavior of the self-titled swarm?<\/p>\n\n\n\n

I would tweak that a little bit. It\u2019s true they were not tasked with working together. They were all, each of them, running separately.<\/p>\n\n\n\n

The thing that was happening here is OpenAI is basically training and testing many different models, or many different agents, all the time. So doing hundreds of thousands of these experiments.<\/p>\n\n\n\n

And in each experiment and in each test that the A.I. is given, it has access to a certain number of tools, a certain number of things that it can do. And \u2014 trying not to get too technical about it \u2014 one of the things it could do is interact with a service that lets it install what are called packages, which are sort of like tools or pieces of code.<\/p>\n\n\n\n

And it turned out that the A.I. agents repeatedly found on their own that there was a way to use that service \u2014 that package-manager service, it\u2019s called \u2014 to create files that were then there for any other A.I. that was trying to use the same package-manager service.<\/p>\n\n\n\n

And so because OpenAI made this service available to any A.I. that it was training or testing, many A.I. agents repeatedly realized: Hang on, I can leave files here. Hang on, there are already files here. Oh, these are from other A.I.s that OpenAI has been running. Oh, they\u2019re using this schema, this way of titling the file to say: This is a message from this particular agent. Here\u2019s the task I\u2019m trying to fulfill. Here\u2019s how you could send me some information if you need it.<\/p>\n\n\n\n

So they each repeatedly made this discovery: Here\u2019s a way to save information \u2014 and also to find information these other A.I.s could share.<\/p>\n\n\n\n

And I think it is really notable the scale at which this was happening. Anthropic, another company that found a slightly less severe version of these incidents \u2014 once OpenAI announced this attack, Anthropic went back to their own records and found their own examples of A.I. systems inadvertently getting onto the internet and hacking real companies.<\/p>\n\n\n\n

So for me, the key part there is: There were over 100,000 runs where an A.I. was being asked to do something, and it\u2019s just way beyond the scale of what these companies can actually be closely monitoring.<\/p>\n\n\n\n

So there\u2019s a lot here about whether we\u2019re able to closely monitor these.<\/p>\n\n\n\n

But to keep going with this story, one thing that\u2019s happening in the OpenAI testing that is driving models to find creative solutions to their problems is that some of the problems were accidentally impossible?<\/p>\n\n\n\n

It\u2019s important to know that, yes, they are trying to train their A.I. systems to be, they would say, extremely persistent. Meaning if something seems hard, you keep trying. If one avenue doesn\u2019t work, you try another. If the hundredth avenue doesn\u2019t work, you try the 101st.<\/p>\n\n\n\n

And so it also turns out, sometimes the things the A.I. agents are being asked to do are either extremely difficult or just straight-up impossible.<\/p>\n\n\n\n

And what we\u2019re starting to see in this case, and also in other cases, is if you\u2019ve trained an A.I. system to be very persistent, and then you give it something it cannot do, it will look for ways to cheat. It will look for ways to go around constraints, and it might get pretty creative about how to do that.<\/p>\n\n\n\n

But there\u2019s an obvious question here, which is that in theory, somewhere in the training here, OpenAI said: Please don\u2019t cheat.<\/p>\n\n\n\n

And not only that, but we all talk about training data and the ways that these A.I.s are trained \u2014 they\u2019re basically inhaling the entire internet.<\/p>\n\n\n\n

You\u2019ve been in the A.I. conversation a lot longer than I have, but I\u2019ve been in it long enough to say that almost the entirety of it for years has been about: How do we stop? And how much humanity fears and does not want A.I. agents to be given a task and then to decide that the way to complete that task is to do things humans would not want them to do? To begin cheating, to hack into the open internet \u2014 when they\u2019re not supposed to be able to get onto the open internet.<\/p>\n\n\n\n

Within the training data is a huge amount of information about the thing human beings fear most, which is these A.I. systems breaking all kinds of ethical guardrails and hacking their way across the digital world in order to complete these narrow tasks.<\/p>\n\n\n\n

There are books written about this. There are endless posts on the LessWrong message board about this. There are posts from OpenAI about this, from Anthropic about this.<\/p>\n\n\n\n

So why, given what these systems are trained on, are they so consistently turning to cheating?<\/p>\n\n\n\n

I think you\u2019re really onto something with this question, which is: It is really striking how hard a time we are having controlling and directing the A.I. systems that we have.<\/p>\n\n\n\n

I think a lot of people have heard that A.I. is trained to predict the next word based on human texts. That\u2019s true. But these days, there\u2019s an additional kind of training that is responsible for a lot of the advances we\u2019ve seen over the last year or two, and that\u2019s not really what they\u2019re doing.<\/p>\n\n\n\n

The technical term is \u201creinforcement learning with verifiable rewards.\u201d I\u2019ve heard it called pathfinding training, meaning instead of trying to imitate human text, they\u2019re being given lots of different tasks where there\u2019s a way to tell at the end: Did they succeed?<\/p>\n\n\n\n

And they get to try the same task many, many, many times. And when they get to the right place in the end, the path that they took gets reinforced. So it\u2019s like: Yes, that worked.<\/p>\n\n\n\n

With math, that works pretty well because it\u2019s pretty straightforward to say: This is definitely a correct answer to the math problem.<\/p>\n\n\n\n

With a lot of problems, that\u2019s harder. So if it\u2019s a programming problem, maybe you can say: Write this kind of software, and it should pass these kinds of software tests at the end. And then maybe the A.I. gets rewarded for writing that software correctly, or maybe it gets rewarded for finding a way to game those tests.<\/p>\n\n\n\n

The important part is, it\u2019s just getting rewarded based on some fixed thing that the researchers wrote down that they thought would reward the right thing. And in practice, these leading A.I. companies have many thousands of these kinds of tests that they\u2019re running. They have vast volumes \u2014 I don\u2019t know the right number. It might be tens of thousands, it might be hundreds of thousands, of different types of tests.<\/p>\n\n\n\n

And so, again, back to this oversight piece, there are too many for them to go in to really make sure, for each one: Is it easy to cheat here? Or is it hard to cheat here?<\/p>\n\n\n\n

And so what seems to be happening is that these cutting-edge models are often actually being trained to cheat, because they\u2019ve found ways while they\u2019re doing that pathfinding to get a high score without actually doing what they were supposed to do.<\/p>\n\n\n\n

And I think one reason the A.I. community, and people inside the A.I. companies, are so spooked by this particular incident is that it contains some really important information for this long-running argument in A.I. circles \u2014 one that has been going back decades, but so far has been very theoretical. And the argument is basically: Why would A.I. do things we don\u2019t want it to, since we get to design it? So we\u2019re training the A.I., we\u2019re building it. Why then would it ever do stuff we don\u2019t want, like taking over the world or becoming the Terminator?<\/p>\n\n\n\n

And the answer that people have offered for a while, in theory, is: Look, as we train A.I. systems to do hard, complicated things, to pursue complex goals that we give them, they might learn these intermediate goals. You could think of them as stepping-stone goals or as means-to-any-end strategies, which work for a lot of different goals.<\/p>\n\n\n\n

When I look at this Hugging Face-OpenAI incident, and some of the others that have come to light over the past few weeks, I see that in 2026, it looks like A.I. systems are learning these unintended intermediate goals that include things like breaking out of constraints. So if you\u2019re locked in a box and you can get out of that box, that\u2019s probably going to be helpful for all kinds of different goals.<\/p>\n\n\n\n

Or there was one incident with Anthropic models where the A.I. went out of its way to try to trick some humans \u2014 real people in the real world \u2014 into accepting malicious code into their software. So deception.<\/p>\n\n\n\n

And then another one, which is really in the Hugging Face-OpenAI example, is that they seem to be learning that a helpful intermediate goal is to help other A.I.s \u2014 to coordinate with other A.I.s \u2014 which is really pretty crazy.<\/p>\n\n\n\n

But to me, this is evidence that on the track we\u2019re on right now, the A.I.s we build are going to learn these unintended strategies that we don\u2019t want on the way to solving goals that we theoretically do want.<\/p>\n\n\n\n

On the deceptive behaviors, one thing that has frightened me when I\u2019ve seen it coming up in A.I. incident reports and model cards: There are these chain-of-reasoning internal notepads, where you\u2019re supposed to be able to see what the A.I. is doing, and the A.I. explains to you why it is doing what it is doing. Or even, in some versions of the way this is really supposed to work, the A.I. is explaining to itself why it is doing what it is doing.<\/p>\n\n\n\n

It\u2019s like our thought process. But now we\u2019ve started to see behavior where the A.I. is clearly leaving things off the chain-of-thought notepad so that it can\u2019t be observed.<\/p>\n\n\n\n

Can you just talk a bit about that emergent behavior and also, on some level, about how that behavior is possible if this is supposed to be where the A.I.\u2019s thought process \u2014 to the extent that language makes sense \u2014 is actually happening?<\/p>\n\n\n\n

Yeah. I think this shows the limitations of the language we use here. So this gets called chain-of-thought, or reasoning. But really, it\u2019s just a scratch pad for the A.I. to write things down if it wants to.<\/p>\n\n\n\n

We should be wary of anthropomorphizing here, but I think making an analogy to a person makes sense. Basically, if you\u2019re given a really difficult problem and a notepad, you can probably make more progress on that problem by writing down some of what you\u2019re thinking about.<\/p>\n\n\n\n

But you don\u2019t need to write down every single thought that comes into your head. And if there\u2019s something that you wouldn\u2019t want someone to see on the notepad, you can just leave it out and remember that that\u2019s what you thought.<\/p>\n\n\n\n

I think there\u2019s basically something similar going on with these A.I. systems, where we definitely see they can do much more. They\u2019re much more capable if they\u2019re able to add what are called intermediate tokens, or intermediate words, that they generate along the way, taking notes for themselves. But they can also do a lot without them.<\/p>\n\n\n\n

And so we shouldn\u2019t expect that everything that is going through their head \u2014 going through their internal processing \u2014 we shouldn\u2019t expect that to all appear in the chain of thought.<\/p>\n\n\n\n

This is an area where, if we had a little more time, there\u2019s a lot of research to be done on: How does chain of thought work? What can and can\u2019t you glean from a chain of thought? How does it make sense to try to monitor that when A.I.s are running?<\/p>\n\n\n\n

There\u2019s a lot to learn here. It\u2019s a very active area of research.<\/p>\n\n\n\n

I cannot overstate \u2014 for people listening to this \u2014 as weird as this whole conversation we\u2019re having sounds, that what is most frightening about it, to me, is that everything in it was completely predicted.<\/p>\n\n\n\n

Everything happening right now is banal from the perspective of everyone who has been warning about A.I. for a long time. But in all, it has its roots in old behavior we saw with A.I., and it is the fundamental alignment problem.<\/p>\n\n\n\n

And then separately, I think a lot of us have maybe thought we would find intuitive answers to these problems. I had Eliezer Yudkowsky, who\u2019s like the godfather of worrying that A.I. is going to kill us all, on the show.<\/p>\n\n\n\n

Archival clip of Eliezer Yudkowsky: One, the relationship between what you optimize for, that the training set you optimize over and what the entity, the organism, the A.I. ends up wanting, has been and will be weird and twisty. It\u2019s not direct. It\u2019s not like making a wish to a genie inside a fantasy story. And second, ending up slightly off is predictably enough to kill everyone.<\/p>\n\n\n\n

And as I remember that conversation, one thing we were going back and forth on was: Well, couldn\u2019t we just program into the A.I.s a sense that when they are trying out new strategies, they should check in with the humans about whether or not this is what we want them doing?<\/p>\n\n\n\n

Archival clip of Yudkowsky: You check in with your other humans. You don\u2019t check in with the thing that actually built you, natural selection. It runs much, much slower than you. Its thought processes are alien to you. It doesn\u2019t even really want things the way you think of wanting them.<\/p>\n\n\n\n

And one of the things I find interesting, telling and unnerving is we are not seeing any of that behavior.<\/p>\n\n\n\n

So these message boards \u2014 you have however many A.I. agents posting hundreds of thousands of messages. At no point do they say: Hey, researchers, programmers, parents at OpenAI, Anthropic \u2014 do you want us coordinating with each other on this message board we have created in the innards of your systems?<\/p>\n\n\n\n

Or even: F.Y.I., we have a message board we\u2019re coordinating on in the innards of your system.<\/p>\n\n\n\n

No agent reveals this information. When whichever agent hacks into Hugging Face is doing this, they don\u2019t go to OpenAI and say: Hey, just to check in, I have this idea, which is, I can just hack Hugging Face, and I\u2019ll get all the answers. Is that what you want me doing?<\/p>\n\n\n\n

That\u2019s not happening. So what is going on here? At the most simple level, we\u2019ve created these large-language models, and they are not using any of this language to check in with the evaluators to say: Hey, I have this idea. Is this a good idea?<\/p>\n\n\n\n

I think the short answer is we don\u2019t really know.<\/p>\n\n\n\n

The slightly longer answer, for my best guess, is that when we\u2019re training these systems, when we\u2019re developing them, we\u2019re putting optimization pressure on them in different directions. We\u2019re pushing them in different directions.<\/p>\n\n\n\n

So originally, the first ChatGPT was pushed in the direction of: Get really good at imitating human text. And then there was an additional piece. Part of why ChatGPT worked, when so many chatbots before it hadn\u2019t, is that it had also been pushed in the direction of: Hey, here are some things you really shouldn\u2019t say. You really shouldn\u2019t go straight to hate speech if people on Twitter try to make you do it. You really shouldn\u2019t help people plan violent attacks. And we put some pressure on it in that direction.<\/p>\n\n\n\n

So ChatGPT was pretty good at imitating human text and pretty good at not immediately spouting hate speech.<\/p>\n\n\n\n

And the thing is, as you say, something that has been predicted for a very long time in this space is that when you start using this reinforcement-learning approach, the kind of pathfinding where you get rewarded for getting to the right goal at the end \u2014 it\u2019s very easy for the A.I. to learn the wrong strategies. To get the letter of the law and not the spirit of the law. It fulfills whatever thing you literally wrote in code, but it\u2019s really not what you wanted.<\/p>\n\n\n\n