Book Summary: If Anyone Builds It, Everyone Dies by Yudkowsky and Soares

The upshot of If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All by Eliezer Yudkowsky and Nate Soares is, well, pretty self-explanatory. The “It” in the title refers to an artificial superintelligence that is vastly more capable than humans across a broad range of domains. The authors argue that once AI crosses a certain threshold, it will pose an extinction-level threat to humanity.

[Estimated reading time: 31 mins]

Buy If Anyone Builds It, Everyone Dies at Amazon (affiliate link)

Key Takeaways from If Anyone Builds It

• What is AI?
  • One way to think of intelligence is the ability to predict and steer the world.
  • Human intelligence has been the source of all our power and technology. But artificial intelligence (AI) has the potential to far outstrip human intelligence.
  • Today’s AIs are grown, not crafted. Engineers have various tricks to indirectly shape an AI’s behaviour, but they don’t understand what is really going on in the AI’s “mind”.
• The alignment problem describes how we can’t yet make an AI reliably want what we want. It is a terrifyingly hard engineering challenge because:
  • If an AI develops the ability to improve itself, the result could be a self-amplifying intelligence explosion.
  • A misaligned AI will be adversarial, meaning it will try to bypass every constraint we put on it.
  • We only get one shot. We can run tests and controls before AI becomes capable enough to kill us. But for humanity to survive, those controls must persist even after AI reaches that level of capability.
• A misaligned superintelligence would likely kill us all:
  • AI is not “stuck inside a computer”. An AI connected to the Internet can affect many things in the wider world.
  • The economic theory of comparative advantage does not guarantee that a superintelligence will want to trade with humans, as it implicitly assumes coercion is not possible.
  • A superintelligence would probably want to kill humans so that we can’t threaten it in any way. But even if it doesn’t want to harm us, it may do so as a side effect of pursuing its goals.
• Predicting the future is not impossible.
  • Predicting when AI might become superintelligent is a hard call. This is part of what makes AI dangerous — it can deliver great benefits right up until it becomes a threat. But it won’t be clear when to stop.
  • Yet predicting that humanity would lose to a superintelligence is a very easy call.
• We can still choose not to build superintelligence:
  • The authors call for an international treaty that would consolidate and monitor all computing clusters capable of training advanced AI, and ban publication of AI research.
  • As an individual, you can help lay the groundwork for your country’s leaders to signal they are open to such a treaty. You can write to your representatives, vote against politicians who want to rush ahead, and simply talk to other people about the risks.

Detailed Summary of If Anyone Builds It

Background

Yudkowsky and Soares both work for the Machine Intelligence Research Institute (MIRI), a non-profit founded by Yudkowsky in 2000. A year earlier, Yudkowsky had begun trying to build artificial superintelligence. In 2003, he realised that making such a superintelligence “friendly” to humans would be hard, and shifted to focusing on that problem.

For most of MIRI’s existence, it acted as a technical research institute. It held workshops for researchers and tried to figure out how to understand and shape superhuman AI. But around 2020, MIRI wound down most of its research and pivoted to conveying a single warning to the world:

What is AI?

Intelligence

Many people dismiss the risks of AI, arguing that “intelligence” is not that well-correlated with success and power. Such people are usually thinking of the intelligence gap between a village idiot and Einstein. That gap is trivial compared to the intelligence gap between humans and other species. Humans can do things that other animals cannot, because of a quality we call “intelligence”.

While we still do not understand intelligence very well, the authors see it as encompassing two different qualities:

• predicting the world; and
• steering the world towards our desired destinations.

Acquiring more intelligence is a very useful strategy for achieving almost any end.

Our intelligence is also more “general” than other animals’. Even if we aren’t the best at everything, we can predict and steer across many domains. This general intelligence has been the source of all our power and technology.

AI could be vastly more intelligent than humans

While some AIs are already “smarter” than us in a couple of narrow domains, the best AI models at the time of writing (early 2025) cannot yet match our general reasoning capabilities. The authors aren’t worried about those narrow AIs as they still have significant limitations, such as not being able to form new long-term memories.

What they are worried about is a much stronger form of AI, which could far exceed human intelligence. Machines have many advantages over biological brains, including:

• Speed. Even if it took 1,000 transistor operations to do the work of a single neural spike, modern hardware could emulate human-quality thinking 10,000x faster.
• Copy-and-paste abilities. It takes 20+ years to grow a human and transfer to them a tiny fraction of all human knowledge. Copying an AI is much easier.
• Faster improvements. Human brains reached a bottleneck when babies’ heads started getting too large to fit through women’s hips. It will take a long time for humans to evolve larger hips.
• Better algorithms. Our brains have a hundred billion neurons, yet most humans struggle to multiply 3-digit numbers in their heads. We aren’t using our neurons anywhere near as effectively as they could be used.
• Memory and storage. In terms of neurons and synapses, the human brain has more storage than most laptops. But modern datacentres can have over 1000x more storage than a human brain.
• Self-experimentation and improvement. AIs could make copies of their minds, perform experiments on them, and restore the originals from backups if needed. We don’t have this ability.

This last point is particularly concerning because it could lead to an intelligence explosion: a positive feedback cycle where an AI makes a smarter AI that makes an even smarter AI, and so on.

There aren’t yet any AIs that have enough general intelligence to improve itself. But newer AIs’ abilities are getting increasingly general.

How we’re currently building AI

Current AIs are grown, not crafted. Engineers today may understand the “gradient descent” process that leads to an AI, but they don’t know what goes on inside the AI “minds” they end up creating. In that sense, the way we’re making AI is more like how we make human babies than how we make other software.

The book gives a brief technical explanation of gradient descent and fine-tuning. The upshot is that current AI models are a pile of billions of numbers that have gone through this gradient descent and fine-tuning process, but nobody understands how those numbers determine the AI’s behaviour. AI engineers have various tricks to indirectly affect an AI’s behaviour, without understanding what the underlying numbers mean.

One way to think of these numbers is by analogy to a DNA sequence. You may see those DNA letters without knowing how those letters affect the relevant person might think or act. Biologists actually know a lot more about how DNA turns into biochemistry and adult traits than engineers understand about how those AI numbers determine an AI’s behaviour.

The problem is that AIs “grown” in this way can do things that their growers never intended.

The AI alignment problem

If the people building AI could choose what an AI “wants” then you could make the AI want a future full of happy, free people. But we don’t know how to get AIs to want the exact things we want. This is a big part of the alignment problem.

Engineers may train an AI to act in friendly, moral, or human-like ways, but that doesn’t mean it actually is friendly, moral or human-like underneath. And the authors don’t think AI will keep acting friendly as its intelligence increases.

AI will “want” certain things

When the authors describe AIs “wanting” things, they aren’t claiming that AIs will have humanlike feelings or desires. Rather, they just need a word to describe the AIs’ outward behaviour — AIs will behave like they want things, and will take actions in the real world to “steer” towards the outcomes they want.

AI companies are also working hard to develop AIs that exhibit agentic, want-like behaviour. An AI that can do things like manage a team on its own initiative is simply more useful and profitable than an AI that cannot.

Using AI to align AI

The leading AI companies are hoping that they can use AIs to solve the alignment problem for them. OpenAI adopted this as their flagship plan in 2023.

The weak version of this superalignment plan involves using AI to help with “interpretability” (i.e. understand how the AI is processing information and “thinking”). That’s fine, but it doesn’t actually solve the alignment problem.

The stronger version of this superalignment plan is to use AI to figure out how to make AI that wants what we want — i.e. getting AI to do the actual alignment work. But we can’t make a narrow AI that specialises in aligning a superintelligence, because we don’t have any examples of solutions to the alignment problem. So the only hope is to train a more general AI to get good at a bunch of other things first (e.g. computer programming, how to grow other AIs, and how to shape AI preferences) and then hope that those skills transfer to “solving alignment”. Of course, the problem with this is that for an AI to solve alignment, it would have to be intelligent enough to be dangerous if it weren’t itself already aligned.

The alignment problem is a terrifyingly hard challenge

The alignment problem is ultimately an engineering challenge. And it’s a terrifyingly hard challenge because:

• an intelligence explosion could be self-amplifying;
• misaligned AI would be an adversarial intelligence; and
• we only have one shot at solving it.

The authors believe that, in principle, it is possible to align artificial superintelligence. We’re just a long way off right now, and modern LLMs are particularly inscrutable.

Self-amplifying factors

A nuclear chain reaction is an example of a process that amplifies itself — a fission reaction emits neutrons which hit other atoms and causes more fission. If something goes even slightly wrong, energy output can double on a timescale of milliseconds. This is far faster than humans can react, so it is essential to get everything right before the chain reaction sets off.

If AI gets to the point where it can make a smarter AI (without any humans in the loop), this self-amplifying process could lead to an “intelligence explosion”. Such an explosion could similarly spin out of control faster than humans could act.

Adversarial intelligence

Computer security is widely understood to be an unsolvable problem. To make a computer system secure, an engineer has to nail down every single path a computer could take, in the face of intelligent adversaries who are trying to find any possible way to penetrate the system. The system must work not just in all the normal and expected cases, but all the edge cases that you’d never normally expect to come up. While security professionals can make software more secure or slow down attackers, they can never fully eliminate vulnerabilities.

AI is like a computer security problem, in that the AI could bypass every constraint we try to put on it if it’s intelligent enough and misaligned.

We only have one shot

Perhaps the most terrifying part of the alignment problem is that we only have one shot. Take space probes as an analogy. Probes are extraordinarily expensive. People stake their entire careers on any given probe’s success. Moreover, they are crafted, not grown, meaning engineers have a good idea of how they work, mechanistically. Yet space probes routinely fail. A key reason is that the test conditions you can subject a probe to before launch are not the same as the conditions they face once they’re actually in space, out of your reach.

AI faces the same challenge. We can only run tests and controls before it becomes capable enough to kill us or resist our attempts to change its goals. But true alignment would mean that our attempts to ensure that AI doesn’t kill us persist even after it reaches that level of capability.

For example, an AI might know something, like how to manufacture bioweapons. (This information may not even be directly in its training data — it might have pieced it together from learning about chemistry and biology more generally.) Before the AI becomes superintelligent, engineers may train it not to talk about such knowledge. They may do this by penalising it every time it shares information about producing bioweapons. But that doesn’t remove the knowledge — it just teaches the AI to hide its knowledge. After the AI becomes superintelligent, it may use this knowledge freely.

So we only have one shot to bridge that gap between “before” and “after”.

Note that this is very different from how humans have built new technology. The history of innovation is filled with people making mistakes and learning from them. But superintelligence is different, because it won’t give us a chance to learn from our biggest mistakes.

This is why if anyone builds it, everyone dies

The authors titled this book “If anyone builds it, everyone dies” because alignment is such an enormous engineering challenge, and we are so far from solving it. If alignment were easier, they would have said “If any fool builds it, everyone dies” instead.

The enormity of the alignment problem is also why the authors are less concerned about concentration of power or malicious actors. In fact, almost everyone building AI seems to act as if the alignment problem doesn’t exist. Whenever someone talks about needing to build superintelligence before China or some other unethical AI lab, they implicitly assume that China or the unethical AI lab would be able to get a superintelligence to do what they want.

A misaligned superintelligence would likely kill us all

An AI is not “stuck inside a computer”

People often wonder why we couldn’t just “switch off” AI if it starts to look dangerous. But an AI is not “stuck inside a computer” any more than you’re “stuck inside a brain.” What you can do depends on what you can affect with your body. What an AI can do depends on what it can affect with things connected to the Internet. Lots of things, including humans, are connected to the Internet, so there are billions of opportunities for AIs to affect the wider world.And we are integrating AI into more and more parts of our economy, with companies building or planning to build millions of robots and self-driving cars.

A superintelligence is unlikely to want to keep humans around by default

Some people argue that a superintelligence might still want to keep humans around, as humans might be useful for it. Economists in particular like to point to the theory of comparative advantage. Comparative advantage is the idea that even where Country A can produce all goods more efficiently than Country B, Country A would be better off trading with Country B instead of trying to produce everything itself.

The problem with comparative advantage is that it implicitly assumes both countries continue to exist and there is no coercion. Comparative advantage doesn’t assert that Country A would always be better off trading with Country B instead of conquering Country B and taking its resources. Once AI reaches a high enough technology level, it is likely to be able to get whatever it ends up wanting in ways that are more efficient than trading with humans.

Others argue that a superintelligence’s preferences might saturate. If it is so powerful, it may be able to find some way to easily satisfy its “wants” without killing any humans. Yet we humans also have preferences that are easy to saturate, such as having enough oxygen to breathe. That doesn’t stop us from having other, open-ended preferences that are harder to satisfy. AI will likely have a huge mix of complicated preferences, and at least one of those is likely to be open-ended. [I make a similar point here about how a single zero-sum game could consume an ever-increasing amount of resources (but in the context of humans, not AIs).]

The authors believe that a superintelligence probably would “want” to kill humans, as that would let it pursue its goals more easily. Otherwise, we could do things like set off nuclear bombs, or build another superintelligence that could threaten it. But even if the AI doesn’t explicitly “want” to kill us, we could simply die from the side effects of whatever the superintelligence ends up doing. For example, if the superintelligence needs a lot of energy, it may create so many fusion power plants that the heat cannot dissipate into space fast enough. Earth would get extremely hot, the oceans would boil off, and we’d all die.

Predicting the future is not impossible

People often say it’s “impossible” to predict the future. But that’s not uniformly true.

Easy calls vs Hard calls

Some things, like exactly when a technology develops, are very hard to predict. Two years before the Wright brothers successfully built the world’s first plane, one of them proclaimed that man would not fly for a thousand years.

But some predictions are easy calls when viewed from the right angle. For example, it’s easy to predict that you will lose in a game of chess against Stockfish (an open-source chess engine). The exact details don’t matter — one doesn’t need to be able to predict the moves that Stockfish will make in order to successfully predict that you will lose.

It is similarly easy to predict that humanity will lose against a superintelligence. We can’t predict how we will lose, as the AI will probably use methods we don’t even know are possible today.

Nor can we predict when superintelligence might happen. AI progress could slow down for a while, until new methods and technologies are invented. We don’t know how many more jumps are needed before AI becomes an extinction-level threat. But AI researchers have repeatedly overcome obstacles with new methods, and the field is moving fast. In 2015, the biggest AI sceptics thought these risks wouldn’t materialise for hundreds of years. In 2020, the same sceptics were saying it would probably take at least 5-10 years before we get superintelligence. [I think there are still sceptics with timeframes significantly longer than 5-10 years. But it’s definitely true that timelines have shortened dramatically.]

Part of the problem with AI is that it can bring lots of economic benefits, right up until the “point of no return” when it becomes a threat to us. But no one knows when that point will be.

If all this is true, why aren’t people freaking out yet?

History is full of examples of people minimising and ignoring catastrophic risk. Look at the history of engineering disasters and you’ll see that the most informed and concerned parties consistently downplayed their fears before the rest of the system caught up, because they were worried about the risks of speaking out.

Geoffrey Hinton, a Nobel Prize winner and one of the “godfathers of AI”, usually says he believes there’s at least a 10% chance of human extinction. In a Q&A session, he said he actually thinks it’s more like 50% (because we simply don’t have a clue). But he usually only gives a 10-20% estimate because he acknowledges that other people think it’s less.

Motivated reasoning is also a powerful thing. Many people cling to hope even when the disaster is unfolding in front of them. If we could successfully align AI, the potential benefits could be enormous. Some dream of abundant energy; others dream of ending every disease. The authors used to have such dreams themselves, before they realised how difficult alignment would be.

Upton Sinclair once observed that it is difficult to get a man to understand something when his salary depends upon his not understanding it. For AI engineers and their leaders, it’s not just their salaries that are at stake but the fact that they have sunk their entire careers into something that could endanger everything they know and love.

What can we do?

The good news is that we’re not doomed. Artificial superintelligence doesn’t exist yet, and we not build it. We can find some other, safer, ways to achieve our dreams of an abundant future.

An international treaty

Not building AI might sound simple enough. But no individual company or researcher can put a stop to the whole field, and each one has a commercial incentive to rush ahead. Even if an individual country outlaws superintelligence inside its own borders, that won’t be enough. AI companies will just move to a country that hasn’t outlawed it. If humanity wants to survive, we cannot allow anyone that can scrape together 100,000 GPUs to set up a datacentre and experiment with increasingly powerful AI.

The only solution is to get a group of sufficiently concerned major powers to sign up to a treaty banning the development of superintelligence around the world.

The authors have even prepared draft text for what such a treaty could look like. The draft is modelled after the Treaty on the Non-Proliferation of Nuclear Weapons. The upshot of their proposal is that:

• All computing clusters that could be used to train or run powerful new AIs should be consolidated into places where they can be monitored by observers from multiple treaty signatories.
• Research into more efficient and powerful AI techniques can no longer be published. This may seem draconian, but almost every year scientists come up with AI algorithms that let them train AI models more efficiently. If we let this algorithmic progress continue, then the computing clusters needed to train superintelligence will get smaller and smaller, making them harder to practically monitor.
• Countries must be prepared to take actions against other actors trying to build superintelligence — even if that other actor didn’t sign the treaty. They should ask the other actor to join the treaty on equal terms and submit their chips to monitoring. But if that other actor refuses, they must be prepared to respond proportionately, with economic sanctions, cyberattacks, or even airstrikes if necessary.

What can I do as an individual?

You don’t have to stop using AI tools, and boycotts are unlikely to work here. What you can do depends on who you are.

If you’re a journalist, you can help by treating this subject with the gravity it deserves, instead of going for easy headlines about tech CEOs hyping up their own products.

If you’re in government, you could support efforts to get your country to signal that it is open to an international treaty to stop superintelligence. A few leaders have already made some signals: in October 2023, Rishi Sunak (then-UK Prime Minister) acknowledged the risks of superintelligence. In the same month, China released a document with a call to “ensure that AI always remains under human control”.

Though the vast majority of us are not country leaders, we can still help lay the groundwork for leaders to make such signals. Multiple elected officials have told the authors that they are concerned about superintelligence, but feel they can’t speak freely about it because it still sounds too “weird”. So if you live in a democracy, you can call and write to your elected representatives, attend protest marches, and vote against politicians that want to rush ahead on AI. For those in the US, the authors have prepared some resources to reduce the friction.

You can also talk to other people about these risks. Most people outside the AI field simply aren’t paying attention or aware of what’s going on. Among those who are, many just see disagreement between experts and don’t feel knowledgeable enough to decide between experts’ competing views. But plenty of experts are very concerned about extinction risk, so you should know it’s not an easy call that everything is going to be fine. And, at the very least, you can support efforts to make it possible for humanity to stop building superintelligence at some point, even if you aren’t sure exactly when that will be.

Other Interesting Points

• There are online supplements for each chapter at IfAnyoneBuildsIt.com. Each supplement has more answers to frequently asked questions and objections to the authors’ arguments. They’re pretty good, and well worth a browse if you remain unconvinced.

• In the middle of the book, there’s a long, fictional, extinction scenario involving an AI called “Sable”, which I’ve omitted.

My Review of If Anyone Builds It

I had heard of AI extinction risk before I read this book, and had accepted it as a “real possibility”. But I had assumed it was further away — something that could take hundreds or even thousands of years to play out — rather than something that could realistically happen in my lifetime. Other AI-related harms were already beginning now, and I could see such harms growing potentially much greater in the next couple of decades, so I figured we should be focusing on those first.

But while Yudkowsky and Soares are careful to emphasise that timing is a “hard call”, this book did make me think that the risks from superintelligence could come about far sooner than I had expected. I found the ladder analogy compelling — the very uncertainty around timing means no one knows precisely when to stop. And each AI lab will always have an incentive to try their luck for “one more rung”.

If Anyone Builds It also made me think more seriously about how hard a global pause to superintelligence would be. Since we don’t yet have the global governance mechanisms in place to enforce a coordinated pause, I agree that at the very least we should be working towards building those mechanisms. I am sympathetic to those who argue that there will need to be a high-profile, Chernobyl-like AI disaster to muster the political will to bring about a global pause, though I also agree with Yudkowsky and Soares that such “warning shots” are unlikely to be clear.

A common criticism of this book is that it relies heavily on speculative fiction and analogies. I get it. I personally do not like speculative fiction, but it seems to resonate for many people. For example, AI 2027, the Citrini Scenario, and Europe 2031 have all gotten far more buzz than, say, the International AI Safety Report 2026. I had less objection to the many analogies used in this book. Admittedly, some were unnecessarily weird, and I omitted those from this summary. But others I found excellent — like the exploding ladder, building space probes, and playing chess against Stockfish. All analogies are flawed of course, but it’s very hard to describe something complex to someone who does not already have a lot of the relevant context without an analogy.

Overall, I think the book draws attention to an important risk that has not gotten nearly the amount of attention it deserves. I plan to write a follow-up post explaining how my thinking on AI extinction risk has shifted over time, as well as some areas where I part ways with the authors. But broadly I agree with the thrust of their argument and I’m glad that this book exists.

Let me know what you think of my summary of If Anyone Builds It, Everyone Dies in the comments below!

Buy If Anyone Builds It, Everyone Dies at: Amazon <– This is an affiliate link, which means I may earn a small commission if you make a purchase through it. Thanks for supporting the site! 🙂

If you enjoyed this summary of If Anyone Builds It, Everyone Dies, you may also like:

• Book Summary: Inadequate Equilibria by Eliezer Yudkowsky
• Book Summary: A Hacker’s Mind by Bruce Schneier
• Book Summary: Deep Utopia by Nick Bostrom