I myself asked chat to make a biological weapon for me, and yes, it shut me down.

But I immediately created a new account, and I began to ask it questions in a different manner, and it never shut me down on the new account.

Hello and welcome to GZERO World.

I'm Ian Bremmer and today we're talking about the end of it, or hopefully how to avoid blowing us all up.

Recently, a researcher at Anthropic named Jacob Coxon resigned from his position with a warning.

He's claimed that leading AI companies are, and I quote, "gambling with our lives as they race to build increasingly powerful systems."

He also says he thinks that there is a 10% chance AI kills us all by 2036.

Now that's probably unlikely, but it raises a bigger question.

How prepared are we if a low probability but catastrophic event actually does happen?

Imagine a pathogen deliberately engineered to be more deadly, or a nuclear exchange that quickly escalates beyond anyone's control.

AI could help us respond to these disasters.

It could also make them easier to trigger, harder to contain.

You've heard me talk about a GZERO World before, one where no country or group of countries is willing and able to provide global leadership.

And as we move closer to that reality, in a true catastrophe, the cooperation needed to contain a global crisis could disappear just when we need it most.

Joining me to discuss all this and more, investigative journalist and bestselling author, Annie Jacobsen, whose latest book, "Biological War: A Scenario," uses a fictional doomsday event to game out precisely that.

Don't worry, I've also got your puppet regime.

Vladimir Putin, you've gone too far this time.

I have?

Yes, you put a drone in a German airport.

Can't stop, won't stop.

But first, a word from the folks who help us keep the lights on.

Funding for GZERO World is provided by our lead sponsor, Prologis.

Every day, all over the world, Prologis helps businesses of all sizes lower their carbon footprint and scale their supply chains with a portfolio of logistics and real estate and an end-to-end solutions platform addressing the critical initiatives of global logistics today.

Learn more at Prologis.com.

And by Cox Enterprises is proud to support GZERO.

Cox is investing in the future, working to create an impact in advanced recycling and in emerging technology companies that will help shape tomorrow.

Cox, a family of businesses.

Additional funding provided by the Andrew Carnegie Foundation, Koo and Patricia Yuen, committed to bridging cultural differences in our communities.

And... When has humanity been closest to Armageddon?

Many think it was Black Saturday, October 27th, 1962, at the height of the Cuban Missile Crisis.

That day, a US spy plane was shot down over Cuba.

And beneath the Caribbean, US destroyers dropped undersea explosives around a Soviet submarine it had found unaware that it was armed with a nuclear torpedo.

Cut off from Moscow and fearing that a war had already begun, the Soviet captain in charge of the sub wanted to fire.

But another senior officer refused to give his approval.

So the torpedo stayed put.

The submarine surfaced.

Humanity lived on.

That was close.

More than six decades later, we've managed to avoid nuclear war, but now there's a new variable in the end times equation, artificial intelligence.

And recently, we've gotten some pretty good examples of how dangerous it might be.

Anthropic, the AI company, revealed that it had blocked several users of its popular platform Claude from using its models to conduct research that could have helped develop dangerous biological weapons.

Now, Anthropic doesn't know whether they were trying to build an actual bioweapon or just doing a little science project, but the accounts were asking questions that could have easily led to a real-life crisis.

Anthropic says they knew about the potential for misuse, but hadn't set up any ways to automatically prevent it.

Or another example, this one from Iran.

Anthropic says that the Iranians used Claude to help track U.S.

Navy ships in the Middle East.

They pulled publicly available ship and aircraft data, paired them with satellite imagery to find U.S.

military targets.

Anthropic eventually caught the activity and banned the account.

The good news is these safeguards eventually worked.

The bad news?

Without formal regulation, such safeguards are inconsistent, they're arbitrary, and they're set by the companies themselves.

Anthropic's CEO has called for slowing the pace of AI development, and other AI leaders quickly said they agreed with him, sort of.

But whether the industry actually slows down is another question.

Now most people aren't worried that ChatGPT is going to wake up tomorrow morning, become Skynet and decide that's it for humanity.

But is AI making it easier to end civilization as we know it?

Developing a sophisticated biological weapon has historically required a lot of expertise, specialized knowledge that you and I don't have.

But AI makes that learning curve a lot shorter.

And nuclear weapons also have another guardrail that AI does not, mutually assured destruction.

Logic, nuclear war is simple.

One nuclear power launches enough weapons in another, it gets destroyed in return.

Everyone loses.

Everybody knows everybody loses.

So what's the equivalent for AI?

Right now, there isn't one.

The greatest threat may not be AI deciding to destroy humanity itself.

It is AI making it easier for humans to set off a crisis they can't control.

Joining me to talk about all this and more, Annie Jacobsen, an award-winning journalist and the author of two books about biological weapons and nuclear war, fun stuff.

Annie Jacobsen, welcome to GZERO World.

Thank you for having me.

I haven't yet read the biological war book.

I did read the nuclear war book when it came out.

It brought me some of the same palpitations when I watched the day after as a high schooler.

As you're writing these books, which, I mean, obviously come as a very strong slap in the face, what's the motivation?

- Oh, the motivation most certainly is knowledge is king and queen.

I really think that the more you know, the better you can discern between like, what is a potential catastrophe and what is really just exciting your mind, what some would call fear-mongering.

- And I mean, you know, when you think about whether it's day after it's on the beach or whether it's Annie Jacobsen's tome, nuclear warfare feels like the most unthinkable scenario imaginable, and yet historically, we've actually come fairly close.

We have.

And you know, I couldn't have agreed with you more when I was writing and reporting nuclear war until I began writing and reporting biological war.

And that literally frightened me because we all know that nuclear weapons sit on that far end of destruction.

And yes, we have come so incredibly close, which we can certainly get into.

But imagine my shock, literally, when I was looking through Defense Department threat spectrums and learned that not only is biological war extremely close on that threat spectrum to nuclear, but it is considered at the Defense Department the greatest overall threat, more so than a nuclear war.

And why is that?

Well, that has to do with the incredible... When you think of the low barrier for entry is so much lower than nuclear, and it is getting lower with computational systems we have today.

I mean, just think about what it takes to have a nuclear weapon.

You have to have fissile material, weapons-grade uranium or plutonium.

And then you have to have a delivery system that takes decades of research and engineering, and God knows how many billions of dollars to maintain.

Think about what goes into an ICBM, a nuclear-powered, nuclear-launch-capable submarine, a B-2 bomber.

And so then you must consider biological war, biological warfare, biological weapons.

The barrier to entry is extremely low.

You do not need any of the material that is so closely guarded by nuclear command and controls.

And the real icing on the cake of horror, I think, is that the weapons system, when you're talking about an airborne pathogen, COVID was airborne, is the human lung set.

- So when I think about Dario Amodei, the CEO of Anthropic, who has said that he's extremely worried that what he is creating, what his competitors at OpenAI, at X, at other companies are creating, will facilitate, unless there's appropriate governance, exactly the scenarios that you are most concerned about.

I see on OpenAI's foundation site that one of the things they're spending money on are filtration systems and masks and other industrial grade capabilities to effectively respond against a bioweapon when it occurs.

So they clearly agree with you about the threat.

How do we assess what AI is presently doing when we think about the biological piece?

Because the nuclear piece, you still have those major barriers to entry with AI.

The biological piece, it's changing, right?

- Absolutely, and you can even argue, you know, that the nuclear piece of when you're thinking about AI is in a different threat spectrum because those systems were created pre sort of computer age.

So a lot of them are, you know, literally and figuratively turnkey systems.

And so AI getting into the nuclear command and control is a completely different animal than bio.

But the real problem with bio, and I think it will help people to understand, that you have to imagine the physicality of it all.

In other words, computers, okay, they can create a biological weapon, let's say, but then what happens?

Well, that's the gap that I think will help people understand what we're really dealing with here.

It's this simple.

Imagine in the old days, you needed to create a biological weapon in a laboratory.

First and foremost, you needed extraordinary expertise, extraordinary expertise.

You needed teams of scientists, and then you needed lab equipment, and you needed experimentation.

That's good old R&D.

And this would go on trial and error for decades.

Now AI has taken away the component of expertise because an AI system, as we all know, can scan documentation in a millisecond and present it.

And furthermore, these more advanced AIs, the chatbots, the way in which they are being trained to think is essentially do the trial and error computationally and then give me the results.

- If I or someone out there, a bad actor, were using AI to develop a biological weapon capability, and we've already seen that Anthropic has said that they shut down some individuals that were asking questions that would lead in that direction.

So clearly there are people out there that are interested in the very things that you are trying to inform to prevent.

The AI is not developing a biological weapon for you.

The AI is giving you a blueprint to develop that biological weapon.

What are the steps that then needs to be taken?

I mean, as opposed to like, you know, on the nuclear side, you'd need the fissile material, you need the ICBM.

Okay, if Claude or whatever the model is now saying, "Okay, Annie, here is the blueprint "for the biological weapon," what then needs to happen?

What is the equivalent turnkey systems on the biological side?

And I'm asking that in part because I'm trying to figure out where the governance will be most effective.

- So in 2008, an Al-Qaeda operative was captured in Afghanistan.

Her name is Aafia Siddiqui, and she was an MIT-trained biologist.

She is now in a federal prison in Texas for 86 years.

Not reported until my book is the fact that on Siddiqui's possession, on a thumb drive, was plans for a biological weapon to mix rabies with chickenpox.

So you're taking one of the most lethal viruses that exists, rabies will kill you 100% if you don't immediately get a vaccine, with one of the most highly contagious airborne pathogens, chickenpox.

Now, in 2008, could she do that?

I am guessing it was not possible in 2008.

That is a guess.

I am also speculating, and this is a much more informed speculation, that the kind of systems that Anthropic is talking about, whether they're there at OpenAI, can now take that recipe and produce a viable plan for creating that bioweapon.

So, the second part of your question, how do you acquire the biological material?

Rabies is not Ebola.

Rabies is commonplace.

4,000 animals in the United States alone had rabies last year.

You get the idea.

Now, once you have the recipe, you have the biological material, now you can make something airborne in a laboratory equipment is not hard to get.

Now you see the existential problem.

So what then is the best pathway or the most effective potential pathway for trying to limit, to identify, to limit and to prevent that danger?

Is it, I mean, are you saying we just can't have tools that are this powerful in the hands of individuals?

Are you saying that we need very different mechanisms for the kind of controls over a much larger group of known pathogens?

How does one begin?

Because as you say, the pathway to entry is so much more viable for so many more people today for this kind of destructive capability than one could ever think about a nuclear force.

- Absolutely, and the conundrum, the problem that you're talking about has landed essentially in our laps, like literally overnight, meaning this wasn't a problem three years ago because the AI systems were not this advanced and no one had necessarily yet predicted that they would be.

I really wish I had the answer.

What I will say in my somewhat educated opinion is that when you hear people talking about, "Oh, we need sensor technology."

We need that is sort of wishful thinking band-aid talk because sensor technology is not advanced to be able to detect an airborne pathogen in the air.

And if you really press anyone on this, they will actually tell you that is absolutely true.

And so these are kind of like, I think I see myself some of these CEOs of AI companies sort of throwing out a little bit of, it feels like spaghetti on the wall, suggestions.

But it's not meant nefariously, I don't believe.

I think it has to do with what people don't know.

And so that is why what you really need is a sort of group of experts where you have computational experts, you also have the biological science experts, and you also have the sensor technologists who can all work together to demonstrate what is not a viable solution right now and what is.

And I think it has to start with putting the brakes on the systems that are allowing this kind of research.

I'll end with this.

I myself asked Chat to make a biological weapon for me when I was researching this book two years ago, okay?

And yes, it shut me down.

But I immediately created a new account and I began to ask it questions in a different manner and it never shut me down on the new account because I was aware of what I couldn't ask.

Now, did I follow that all the way through to try to get a recipe for a biological weapon?

No, I didn't need the FBI coming to my door.

But I'm not, I don't believe the FBI is going to people's doors.

That's a giant gap between the reality of the situation and what is being said.

- Now, I mean, cyber concerns, you know, you have white hat, you have black hat.

It would seem to me that one of the things that one would be calling for here would be a large number of white-hatted bio-hacking specialists that would be working for the government, would be working for these companies to identify and shut down those pathways.

- Most certainly.

But I do also believe that the companies and their work is siloed into an area of expertise that the government presently has simply not acknowledged.

And therein lies the problem.

It's why you see for the first time in modern national security history, you see a situation where it's very clear to the citizenry that the White House and the Defense Department do not have the pole position on the private companies.

I mean, the best analogy I can think of is go back in time to the early 1950s when thermonuclear weapons were being developed completely in secrecy under extreme controls by the Atomic Energy Commission and its Defense Department partners, and no one even knew about it.

- And you're suggesting that, I mean, obviously some of that is precisely because they are legitimately concerned that adversaries of the United States globally would be empowered by understanding the comparative position of the U.S.

So, I mean, it's not quite a catch-22, but there is a real concern and a real need for national security secrecy, yes?

Absolutely.

But what I am suggesting, and I wish I weren't, but I'm a member of society too, is that it appears to me that what the AI companies are really saying is the horse is out of the barn.

Yeah, they are saying that.

Of course, they have many reasons to be potentially saying that, right?

One is because their products are so amazing that you've got to invest in them and you've got to spend money on tokens and you've got to use them and not have regular employees.

And some of it is because they're better than the others that they are competing against.

When I see people in positions of power and wealth that are making really disastrous arguments that just coincidentally happen to align with their financial interests.

Maybe it's me, but I tend to at least inject more skepticism into my assessment of those claims.

- I'm with you on the skepticism, but also the neutrality part of it, and the more, "Hmm, what is really going on here?"

I have interviewed a number of these CEOs for my national security work and have met their children.

And so I kind of sometimes approach this outside my lane as a reporter and more as a parent.

And that is where I see real concerns.

So I was recently asked to come to Washington, DC to talk to someone in a position of power who will also remain nameless.

And this was about my nuclear book.

And he is a very important figure in this.

I mean, he has a very important role in national security and nuclear weapons.

He had my book on the table and he said to me, "The reason I read your book is because my daughter told me to."

And I said, "How old is your daughter?"

And he said, "She's a sophomore at..." and then he named a university.

And so that's kind of a... I'm flipping it around to say, I actually do have the hope of the younger generation being aware of the existential threats, these global catastrophic risks that face all of us.

And I love the fact that this young woman was like, "Dad, get on it."

And that's why he called me in.

- You know, when I first read your nuclear book, the first thing I thought about was the Bulletin of Atomic Scientists and the fact that they have their doomsday clock that they come out with every year and how close we are to midnight, which is again, an assessment of a lot of pretty smart and many older scientists about how close we are to extinguishing ourselves as a species.

And I wonder if you think, having spent so much of your life working on these things, do you think that there is something inherent in the human condition that we are living in such a precarious, a self-imposed, albeit collectively, a self-imposed, precarious position as humanity?

- This is the nature of being a human and having this as a brain and having science and technology, which comes as a result of this brain, advancing faster than essentially the brain itself.

So we are in a new paradigm of that complex equation because the systems are now mimicking our own brains better than our own brains.

I do see a dividing line in threat, in global catastrophic threat, where we are now with essentially, you know, I pretty much look at the, people talk about consciousness and AI.

The AI can perform better than the brain.

So we have kind of crossed that line that John von Neumann warned about back in the 50s.

- Okay, well put that in the paperback version, please.

Annie Jacobsen, thanks so much for joining us today.

Thank you for having me.

And now for something else that could bring about Armageddon, I've got your puppet regime.

Vladimir Putin, you've gone too far this time.

I have?

Yes, you put a drone in a German airport.

Can't stop, won't stop.

You could have killed people.

- Huh.

- Well that's it, you've given us no choice.

- Oh no, a working group?

Don't be ridiculous.

- A strongly worded statement?

Yeah, but also we are closing a Russian cultural center in Germany.

You what?

A cultural cen... This escalation is an outrage.

You imperialist sharks are pushing me... But Vladimir, it's only... It's like Stalingrad all over again.

You are going to force me to do something really... Don't fly off the handle, Vladimir!

Something totally crazy.

Dimitri, bring me my briefcase.

It's time to... Relax!

Vladimir, relax, okay?

It's just... just a couple of cultural centers, okay?

It's... that's it.

- Fine.

- Yeah, good.

Okay.

- Bye.

How was that?

I really freaked them out.

I think I really freaked them out.

Yeah, it's just a cultural center.

Anyway, who should we try next?

Denmark?

- Puppet Regime.

That's our show this week.

Come back next week.

And if you like what you've seen or even if you don't, or you have some extra space for me in your underground doomsday bunker, why don't you check us out at gzeromedia.com.

(upbeat music) - Funding for GZERO World is provided by our lead sponsor, Prologis.

- Every day, all over the world, Prologis helps businesses of all sizes lower their carbon footprint and scale their supply chains.

With a portfolio of logistics and real estate and an end-to-end solutions platform addressing the critical initiatives of global logistics today.

Learn more at Prologis.com.

And by Cox Enterprises is proud to support GZERO.

Cox is investing in the future, working to create an impact in advanced recycling and in emerging technology companies that will help shape tomorrow.

Cox, a family of businesses.

Additional funding provided by the Andrew Carnegie Foundation, Koo and Patricia Yuen, committed to bridging cultural differences in our communities.

And... ♪♪