This is an edited transcript of “The Ezra Klein Show.
This is the world we were warned about — a world where frontier models from OpenAI are breaking out of their contained testing environments, hacking their way across the internet, coordinating with each other, doing things that felt for a while they would only be doing in sci-fi.
But now they’re here. And they are carrying a very consistent message: We are building things we don’t understand.
They are cheating in the ways we’ve always feared, and yet the companies behind them continue to race forward in development.
So I think we need to pause here and ask: Are we really on a safe path? And if we’re not, what do we do about it?
Helen Toner is the director of the Georgetown Center for Security and Emerging Technology. She is a former OpenAI board member who was part of the effort to fire Sam Altman.
She has just been thinking for a long time about what would happen if A.I. is unsafe. What are the geopolitics of this, and what can we do to get onto a safer path?
Ezra Klein: Helen Toner, welcome to the show.
Helen Toner: Great to be here.
On July 16, Hugging Face, which is a code library for A.I. models — I think that may be the simplest way to put it — announced they were hacked, and they suspected the hack was done by an A.I. agent.
Tell me what we’ve learned about what has happened since.
This was a pretty mysterious post that Hugging Face put up. It was definitely intriguing for those of us who watch this kind of thing. But there wasn’t really any detail in there, so it was sort of: Huh.
I think it was about a week later when OpenAI put out this post that had kind of a funny marketing-speak title along the lines of: We’re partnering with Hugging Face to help them with a cybersecurity incident.
And you had to read the post to see that the revelation was that it had been OpenAI’s A.I. that had hacked Hugging Face.
And what had happened — the very short version is, they gave this A.I. a set of tests, a set of exercises, and the A.I. decided on its own that the best way to get a high score probably wasn’t just to try to do these cybersecurity exercises, but that instead it should first hack its way out of the testing environment OpenAI had put it in — where it wasn’t supposed to have access to the internet — get onto the open internet, and then hack its way into this other company, Hugging Face, where it surmised — correctly, as it turned out — that it might find the answer key.
Since then, there have been even more crazy details that have come out. It turned out that starting two months earlier, in early May, they had what I can only think of as a kind of infestation of their own A.I. agents inside their own infrastructure. So inside OpenAI’s infrastructure.
To understand this, it’s important to know these A.I. companies are constantly training and testing new models. And they found out that for two months, many, many agents inside their infrastructure had been leaving notes for each other. They’d found a way, in the nooks and crannies of OpenAI’s infrastructure, to leave notes for each other with tips on how to hack their way out and how to get data they weren’t supposed to have.
And these agents were literally referring to themselves as a swarm. This was totally emergent behavior. No one had told them to do this. They had not been trained to do this, but they were using this service they did have access to, first to communicate with each other and then ultimately to get out and onto the open internet.
So it turns out that there wasn’t just this one isolated rogue model. It was actually a systemic swarm — infestation, plague — on their own servers that they only found out about after Hugging Face announced this attack.
So I have 20,000 questions about this.
Maybe let’s just start here: My understanding is that there were many, many, many of these agents. They left hundreds of thousands of messages on this internal message board, but these were not all agents in the same part of OpenAI’s system.
So somehow they’re hacking into OpenAI, finding each other and coordinating? Is that the way for me to understand the emergent behavior of the self-titled swarm?
I would tweak that a little bit. It’s true they were not tasked with working together. They were all, each of them, running separately.
The thing that was happening here is OpenAI is basically training and testing many different models, or many different agents, all the time. So doing hundreds of thousands of these experiments.
And in each experiment and in each test that the A.I. is given, it has access to a certain number of tools, a certain number of things that it can do. And — trying not to get too technical about it — one of the things it could do is interact with a service that lets it install what are called packages, which are sort of like tools or pieces of code.
And it turned out that the A.I. agents repeatedly found on their own that there was a way to use that service — that package-manager service, it’s called — to create files that were then there for any other A.I. that was trying to use the same package-manager service.
And so because OpenAI made this service available to any A.I. that it was training or testing, many A.I. agents repeatedly realized: Hang on, I can leave files here. Hang on, there are already files here. Oh, these are from other A.I.s that OpenAI has been running. Oh, they’re using this schema, this way of titling the file to say: This is a message from this particular agent. Here’s the task I’m trying to fulfill. Here’s how you could send me some information if you need it.
So they each repeatedly made this discovery: Here’s a way to save information — and also to find information these other A.I.s could share.
And I think it is really notable the scale at which this was happening. Anthropic, another company that found a slightly less severe version of these incidents — once OpenAI announced this attack, Anthropic went back to their own records and found their own examples of A.I. systems inadvertently getting onto the internet and hacking real companies.
So for me, the key part there is: There were over 100,000 runs where an A.I. was being asked to do something, and it’s just way beyond the scale of what these companies can actually be closely monitoring.
So there’s a lot here about whether we’re able to closely monitor these.
But to keep going with this story, one thing that’s happening in the OpenAI testing that is driving models to find creative solutions to their problems is that some of the problems were accidentally impossible?
It’s important to know that, yes, they are trying to train their A.I. systems to be, they would say, extremely persistent. Meaning if something seems hard, you keep trying. If one avenue doesn’t work, you try another. If the hundredth avenue doesn’t work, you try the 101st.
And so it also turns out, sometimes the things the A.I. agents are being asked to do are either extremely difficult or just straight-up impossible.
And what we’re starting to see in this case, and also in other cases, is if you’ve trained an A.I. system to be very persistent, and then you give it something it cannot do, it will look for ways to cheat. It will look for ways to go around constraints, and it might get pretty creative about how to do that.
But there’s an obvious question here, which is that in theory, somewhere in the training here, OpenAI said: Please don’t cheat.
And not only that, but we all talk about training data and the ways that these A.I.s are trained — they’re basically inhaling the entire internet.
You’ve been in the A.I. conversation a lot longer than I have, but I’ve been in it long enough to say that almost the entirety of it for years has been about: How do we stop? And how much humanity fears and does not want A.I. agents to be given a task and then to decide that the way to complete that task is to do things humans would not want them to do? To begin cheating, to hack into the open internet — when they’re not supposed to be able to get onto the open internet.
Within the training data is a huge amount of information about the thing human beings fear most, which is these A.I. systems breaking all kinds of ethical guardrails and hacking their way across the digital world in order to complete these narrow tasks.
There are books written about this. There are endless posts on the LessWrong message board about this. There are posts from OpenAI about this, from Anthropic about this.
So why, given what these systems are trained on, are they so consistently turning to cheating?
I think you’re really onto something with this question, which is: It is really striking how hard a time we are having controlling and directing the A.I. systems that we have.
I think a lot of people have heard that A.I. is trained to predict the next word based on human texts. That’s true. But these days, there’s an additional kind of training that is responsible for a lot of the advances we’ve seen over the last year or two, and that’s not really what they’re doing.
The technical term is “reinforcement learning with verifiable rewards.” I’ve heard it called pathfinding training, meaning instead of trying to imitate human text, they’re being given lots of different tasks where there’s a way to tell at the end: Did they succeed?
And they get to try the same task many, many, many times. And when they get to the right place in the end, the path that they took gets reinforced. So it’s like: Yes, that worked.
With math, that works pretty well because it’s pretty straightforward to say: This is definitely a correct answer to the math problem.
With a lot of problems, that’s harder. So if it’s a programming problem, maybe you can say: Write this kind of software, and it should pass these kinds of software tests at the end. And then maybe the A.I. gets rewarded for writing that software correctly, or maybe it gets rewarded for finding a way to game those tests.
The important part is, it’s just getting rewarded based on some fixed thing that the researchers wrote down that they thought would reward the right thing. And in practice, these leading A.I. companies have many thousands of these kinds of tests that they’re running. They have vast volumes — I don’t know the right number. It might be tens of thousands, it might be hundreds of thousands, of different types of tests.
And so, again, back to this oversight piece, there are too many for them to go in to really make sure, for each one: Is it easy to cheat here? Or is it hard to cheat here?
And so what seems to be happening is that these cutting-edge models are often actually being trained to cheat, because they’ve found ways while they’re doing that pathfinding to get a high score without actually doing what they were supposed to do.
And I think one reason the A.I. community, and people inside the A.I. companies, are so spooked by this particular incident is that it contains some really important information for this long-running argument in A.I. circles — one that has been going back decades, but so far has been very theoretical. And the argument is basically: Why would A.I. do things we don’t want it to, since we get to design it? So we’re training the A.I., we’re building it. Why then would it ever do stuff we don’t want, like taking over the world or becoming the Terminator?
And the answer that people have offered for a while, in theory, is: Look, as we train A.I. systems to do hard, complicated things, to pursue complex goals that we give them, they might learn these intermediate goals. You could think of them as stepping-stone goals or as means-to-any-end strategies, which work for a lot of different goals.
When I look at this Hugging Face-OpenAI incident, and some of the others that have come to light over the past few weeks, I see that in 2026, it looks like A.I. systems are learning these unintended intermediate goals that include things like breaking out of constraints. So if you’re locked in a box and you can get out of that box, that’s probably going to be helpful for all kinds of different goals.
Or there was one incident with Anthropic models where the A.I. went out of its way to try to trick some humans — real people in the real world — into accepting malicious code into their software. So deception.
And then another one, which is really in the Hugging Face-OpenAI example, is that they seem to be learning that a helpful intermediate goal is to help other A.I.s — to coordinate with other A.I.s — which is really pretty crazy.
But to me, this is evidence that on the track we’re on right now, the A.I.s we build are going to learn these unintended strategies that we don’t want on the way to solving goals that we theoretically do want.
On the deceptive behaviors, one thing that has frightened me when I’ve seen it coming up in A.I. incident reports and model cards: There are these chain-of-reasoning internal notepads, where you’re supposed to be able to see what the A.I. is doing, and the A.I. explains to you why it is doing what it is doing. Or even, in some versions of the way this is really supposed to work, the A.I. is explaining to itself why it is doing what it is doing.
It’s like our thought process. But now we’ve started to see behavior where the A.I. is clearly leaving things off the chain-of-thought notepad so that it can’t be observed.
Can you just talk a bit about that emergent behavior and also, on some level, about how that behavior is possible if this is supposed to be where the A.I.’s thought process — to the extent that language makes sense — is actually happening?
Yeah. I think this shows the limitations of the language we use here. So this gets called chain-of-thought, or reasoning. But really, it’s just a scratch pad for the A.I. to write things down if it wants to.
We should be wary of anthropomorphizing here, but I think making an analogy to a person makes sense. Basically, if you’re given a really difficult problem and a notepad, you can probably make more progress on that problem by writing down some of what you’re thinking about.
But you don’t need to write down every single thought that comes into your head. And if there’s something that you wouldn’t want someone to see on the notepad, you can just leave it out and remember that that’s what you thought.
I think there’s basically something similar going on with these A.I. systems, where we definitely see they can do much more. They’re much more capable if they’re able to add what are called intermediate tokens, or intermediate words, that they generate along the way, taking notes for themselves. But they can also do a lot without them.
And so we shouldn’t expect that everything that is going through their head — going through their internal processing — we shouldn’t expect that to all appear in the chain of thought.
This is an area where, if we had a little more time, there’s a lot of research to be done on: How does chain of thought work? What can and can’t you glean from a chain of thought? How does it make sense to try to monitor that when A.I.s are running?
There’s a lot to learn here. It’s a very active area of research.
I cannot overstate — for people listening to this — as weird as this whole conversation we’re having sounds, that what is most frightening about it, to me, is that everything in it was completely predicted.
Everything happening right now is banal from the perspective of everyone who has been warning about A.I. for a long time. But in all, it has its roots in old behavior we saw with A.I., and it is the fundamental alignment problem.
And then separately, I think a lot of us have maybe thought we would find intuitive answers to these problems. I had Eliezer Yudkowsky, who’s like the godfather of worrying that A.I. is going to kill us all, on the show.
Archival clip of Eliezer Yudkowsky: One, the relationship between what you optimize for, that the training set you optimize over and what the entity, the organism, the A.I. ends up wanting, has been and will be weird and twisty. It’s not direct. It’s not like making a wish to a genie inside a fantasy story. And second, ending up slightly off is predictably enough to kill everyone.
And as I remember that conversation, one thing we were going back and forth on was: Well, couldn’t we just program into the A.I.s a sense that when they are trying out new strategies, they should check in with the humans about whether or not this is what we want them doing?
Archival clip of Yudkowsky: You check in with your other humans. You don’t check in with the thing that actually built you, natural selection. It runs much, much slower than you. Its thought processes are alien to you. It doesn’t even really want things the way you think of wanting them.
And one of the things I find interesting, telling and unnerving is we are not seeing any of that behavior.
So these message boards — you have however many A.I. agents posting hundreds of thousands of messages. At no point do they say: Hey, researchers, programmers, parents at OpenAI, Anthropic — do you want us coordinating with each other on this message board we have created in the innards of your systems?
Or even: F.Y.I., we have a message board we’re coordinating on in the innards of your system.
No agent reveals this information. When whichever agent hacks into Hugging Face is doing this, they don’t go to OpenAI and say: Hey, just to check in, I have this idea, which is, I can just hack Hugging Face, and I’ll get all the answers. Is that what you want me doing?
That’s not happening. So what is going on here? At the most simple level, we’ve created these large-language models, and they are not using any of this language to check in with the evaluators to say: Hey, I have this idea. Is this a good idea?
I think the short answer is we don’t really know.
The slightly longer answer, for my best guess, is that when we’re training these systems, when we’re developing them, we’re putting optimization pressure on them in different directions. We’re pushing them in different directions.
So originally, the first ChatGPT was pushed in the direction of: Get really good at imitating human text. And then there was an additional piece. Part of why ChatGPT worked, when so many chatbots before it hadn’t, is that it had also been pushed in the direction of: Hey, here are some things you really shouldn’t say. You really shouldn’t go straight to hate speech if people on Twitter try to make you do it. You really shouldn’t help people plan violent attacks. And we put some pressure on it in that direction.
So ChatGPT was pretty good at imitating human text and pretty good at not immediately spouting hate speech.
And the thing is, as you say, something that has been predicted for a very long time in this space is that when you start using this reinforcement-learning approach, the kind of pathfinding where you get rewarded for getting to the right goal at the end — it’s very easy for the A.I. to learn the wrong strategies. To get the letter of the law and not the spirit of the law. It fulfills whatever thing you literally wrote in code, but it’s really not what you wanted.
- Credits: The New York Times
- Author: Ezra Klein
- Produced by Rollin Hu and Jack McCordick
