If Anyone Builds It, Everyone Dies

If Anyone Builds It, Everyone Dies is a 2025 book by Eliezer Yudkowsky and Nate Soares which details potential threats posed to humanity by artificial superintelligence.

Quotes

All page numbers are from the hardcover first edition published by Little, Brown and Company, ISBN 978-0-316-59564-3, first printing

All bold face and italics as in the book

Introduction: Hard Calls and Easy Calls

• Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
  • p. 3

• If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.
  • p. 7

• Some aspects of the future are predictable, with the right knowledge and effort; others are impossibly hard calls. Competent futurism is built around knowing the difference.
  • p. 8

• History teaches that one kind of relatively easy call about the future involves realizing that something looks theoretically possible according to the laws of physics, and predicting that eventually someone will go do it. Heavier-than-air flight, weapons that release nuclear energy, rocket that go to the Moon with a person on board: these events were called in advance, and for the right reasons, despite pushback from skeptics who sagely observed that these things hadn’t yet happened and therefore probably never would.
  • p. 8

• Even in the face of superhuman machine intelligence, it can be tempting to imagine that the world will keep looking the way it has over the last few decades of our relatively short lives. It is true, but hard to remember, that there was a time is real as our own time, just a few short centuries ago, when civilization was radically different. Or millennia ago, when there was no civilization to speak of. Or a million years ago, when there were no humans. Or a billion years ago, when multicellular colonies had no specialized cells.
  Adopting a historical perspective can help us appreciate what is so hard to see from the perspective of our own short lifespans: Nature permits disruption. Nature permits calamity. Nature permits the world to never be the same again.
  • p. 10

• Normality always ends. This is not to say that it’s inevitably replaced by something worse; sometimes it is and sometimes it isn’t, and sometimes it depends on how we act. But clinging to the hope that nothing too bad will be allowed to happen does not usually help.
  • p. 11

Chapter 1: Humanity’s Special Power

• A human brain can learn to navigate wider-ranging paths through a larger cross-section of reality than any other animal. That is our special power.
  • p. 19

• In our view, intelligence is about two fundamental types of work: the work of predicting the world, and the work of steering it.
  • p. 20

• When it comes to thinking, quality trumps quantity.
  • p. 24

• It is as improbable that human thinking patterns mark the final limit of intelligent algorithms as it is that human neurons represent the limit of possible computing speeds.
  • p. 25

• Ultra-fast minds that can do superhuman-quality thinking at 10,000 times the speed, that do not age and die, that make copies of their most successful representatives, that have been refined by billions of trials into unhuman kinds of thinking that work tirelessly and generalize more accurately from less data, and that can turn all that intelligence to analyzing and understanding and ultimately improving themselves – these minds would exceed ours.
  • p. 25

• Human intelligence is the source of all our power, all our technology.…
  So far, humanity has had no competitors for our special power. But what if machine minds get better than us at the thing that, up until now, made us unique?
  • p. 28 (ellipsis represents the elision of examples)

Chapter 2: Grown, Not Crafted

• The most fundamental fact about current AI’s is that they are grown, not crafted. It is not like how other software gets made – indeed it is closer to how a human gets made, at least in some important ways. Namely, engineers understand the process that results in an AI, but do not much understand what goes on inside the AI minds they manage to create.
  • p. 31

• Humanity does not need to understand intelligence, in order to grow machines that are smarter than us.
  • p. 39

Chapter 3: Learning to Want

• Once AIs get sufficiently smart, they’ll start acting like they have preferences – like they want things.
  We’re not saying that AIs will be filled with humanlike passions. We’re saying they’ll behave like they want things; they’ll tenaciously steer the world toward their destinations, defeating any obstacles in their way.
  • p. 46

• If you were able to choose what an AI wants – the destinations toward which it steers – that might be good news for you. Or bad news, if you made a poor choice of destinations, or if some malicious person makes an AI that steers toward outcomes you dislike. But the problem facing humanity is not a problem of whether good people or bad people are in control of AI.
  No – we are facing an even harder problem: it’s much easier to grow artificial intelligence that steers somewhere then it is to grow AIs that steer exactly where you want.
  • p. 54

Chapter 4: You Don’t Get What You Train For

• If you were incredibly, incredibly optimistic, you might look at the differences and say: “Well, gradient descent is not the same as natural selection, so it won’t have all the same complications as natural selection. And I don’t know of any particular complications in the relationship between what AIs are trained for and what AIs end up wanting; so I don’t expect any complications.”
  But a blank map does not correspond to a blank territory: if you’re venturing across an unknown land mass and your map has a blank spot where you haven’t visited, it doesn’t mean you’ll see a vast empty space when you get there. If gradient descent is different from natural selection, that doesn’t mean that we should expect to see no complications, since we don’t know about any. Rather, we should expect to see new, interesting, unpredicted complications.
  • p. 65

• And if a sci-fi writer tried to write that story, the audience would just be confused, because why did that happen? Isn’t that the opposite of what the AI was trained for?
  But reality is allowed to be like that. And we are, fundamentally, predicting that the world will not turn out like a sci-fi novel. We’re predicting that AI’s preferences will turn out to be complicated and weird.
  • p. 71

• You can’t grow an AI that does what you want just by training it to be nice and hoping.
  You don’t get what you train for.
  • p. 72

• Many of these complications won’t show up in obvious, undeniable ways until after it’s too late for humans to do anything about them.
  • p. 73

• Problems like this are why we say that if anyone builds it, everyone dies. If all the complications were visible early, and had easy solutions, then we’d be saying that if any fool builds it, everyone dies, and that would be a different situation. But when some of the problems stay out of sight? When some complications inevitably go unforeseen? When the AIs are grown rather than crafted, and no one understands what’s going on inside of them? That’s not a problem that anyone’s equipped to solve.
  • p. 74

• The preferences that wind up in a mature AI are complicated, practically impossible to predict, and vanishingly unlikely to be aligned with our own, no matter how it was trained.
  • p. 74

• The problem with making AIs want – and ultimately do – the exact, complicated things that humans want is a major facet of what’s known as the “AI alignment problem.” It’s what we had in mind when we were brainstorming terminology with the AI professor Stuart Russell back in 2014, and settled on the term “alignment.”
  Most everyone who is building AIs, however, seems to be operating as if the alignment problem doesn’t exist – as if the preferences the AI winds up with will be exactly what they train into it. This assumption lurks in the background whenever someone says, “The USA needs to build superintelligence before China, because we don’t trust China,” as if the factional allegiance of whoever ran the gradient descent determined what the resulting AI wanted.
  • pp. 74-75

• The problem here is not that corporate executives might build AI servants and command them to do something monstrous. They’re not in control. It doesn’t matter whether they’re benevolent. Humanity is faced with an engineering challenge: How do we shape the preferences of AIs that we can’t understand? It doesn’t matter whether or not the engineers have an ethics team watching over their shoulder; the ethicists wouldn’t have any idea how to get an AI’s preferences to align with ours, either.
  • pp. 75-76

Chapter 5: Its Favorite Things

• Most powerful artificial intelligences, created by any method remotely resembling the current methods, would not choose to build a future full of happy, free people. We aren’t saying this because we get a kick out of being bleak. It’s just that those powerful machine intelligences will not be born with preferences much like ours.
  • p. 82

• With so many different hopes, surely there’s a chance that one of them will pan out? If you think reality works like that, go try to write a hundred different letters to someone with fifty billion dollars, giving a hundred different reasonable reasons you thought of why they ought to give you a hundred million dollars for your personal use. See if it works. The reason it all fails in the end is that the fifty-billionaire does not want to rationalize giving you 0.2 percent of their wealth, not the same way you rationalize reasons they should want to.
  In much the same way, an artificial superintelligence will not want to find reasons to keep humanity around – not in the same way that humans desperately want to find reasons to be kept.
  • p. 89

Chapter 6: We’d Lose

• Pathways are hard to predict.
  But we can predict the endpoint.
  • p. 97

• We’re pretty sure, actually very very sure, that a machine superintelligence can beat humanity in a fight, even if it’s starting with fairly limited resources.
  • p. 97

• The real way a superintelligence wins a conflict is using methods you didn’t know were possible. And because we care about the truth more than about telling you things that are easy to swallow, that’s where will start.
  • p. 98

• The more complicated the game board, the more advantage goes to the player with more knowledge and more intelligence and more understanding of the game.
  • p. 99

• We know, from years of talking to people about this subject, that some people are swayed by the abstract observation that a superintelligence could exploit options they didn’t even know were possible.
  For others, however, explanations such as these end up making them feel like we’re cheating in a child’s game of pretend. If we can’t even tell a story about how the bad guy is supposed to win, how is that convincing?
  We emphasize again: Reality has never been bound by that rule. Even if an Aztec soldier couldn’t have figured out in advance how guns work, the big boat on the horizon contained them anyway.
  • p. 113

Coda to Chapter 9

• We predict this with confidence: Once some AIs go to superintelligence – and nobody will delay much in pushing AIs that far, if in the middle of some great arms race – humanity does not stand a chance. Ends are sometimes easier to call than pathways. The only part of our story that is a real prediction is the ending – and then, only if the story is allowed to begin.
  • pp. 157-158

Chapter 10: A Cursed Problem

• The greatest and most central difficulty in aligning artificial superintelligence is navigating the gap between before and after.…
  Ideas and theories can only be tested before the gap. They need to work after the gap, on the first try.
  • p. 161

• From each of the four curses we named, we draw these lessons:
  1. An engineering challenge is much harder to solve when the underlying processes run on time scales faster than humans can react….
  2. An engineering challenge is much harder to solve when there is a narrow margin for error, especially if it’s a narrow margin between “unimpressive” and “explosive.”…
  3. Self-amplifying processes, like an overheating reactor boiling off its coolant water and then overheating more, leave little room for error….
  4. Complications make engineering problems worse….
  • pp. 170-171 (ellipses represent explanations of the bold-face headings)

• From these lessons in combination, we infer an additional lesson for engineers: If someone doesn’t know exactly what’s going on inside the complicated device subject to all these curses – speed, narrow margins, self-amplification, complications – then they should stop. They should shut it down immediately, at the moment the behavior looks strange; don’t wait until the behavior becomes visibly concerning.
  • p. 171

• Computer security is widely understood to be a problem so hard, so cursed, that it cannot be solved, period.
  • p. 172

• Computer security is a test of an engineer’s ability to nail down every single path the computer could take, in the face of adversaries who can search all possible ways to perturb the system. It is a famously losing battle – even though the engineers can fully control and craft their own computer’s code.
  We dub this central challenge the curse of edge cases: To be secure, a computer system must work in the face of cases that are outside of the normal and expected range, cases that occur on the edges of possibility.
  • p. 174

• Space probes. Nuclear reactors. Computer security. What do all these lessons add up to, and what can we learn from them about the difficulty of aligning an artificial superintelligence?
  An artificial superintelligence is like a space probe, in that we cannot test it in quite the same environment where it needs to work, and by default it is not retrievable or correctable once it rises high above us….
  An artificial superintelligence is like a nuclear reactor, in that its underlying reality involves immense, potentially self-amplifying forces, whose inner processes run faster than humans can react.
  An artificial superintelligence is like a computer security problem, in that every constraint an engineer tries to place upon the system might be bypassed by the intelligent forces that those constraints hinder.
  • pp. 175-176

• We usually try to avoid shouting. It doesn’t help to shout, most of the time. It just makes people think you’re undisciplined. But at some point, after you’ve calmly gone through all the premises of your argument, we think it becomes unhelpful to downplay, lest people think it’s all just a game of calm words.
  When it comes to AI, the challenge humanity is facing is not surmountable with anything like humanity’s current level of knowledge and skill. It isn’t close.
  Attempting to solve a problem like that, with the lives of everyone on Earth at stake, would be an insane and stupid gamble that NOBODY SHOULD BE ALLOWED TO TRY.
  • p. 176

Chapter 11: An Alchemy, Not a Science

• Perhaps you don’t believe us about any of the foreseeable reasons why shaping ASI is unreasonably hard. There’s an independent and separate case for disaster, an alternate set of historical lessons: Humans sometimes flub easy problems, never mind hard problems.
  • p. 180

• If you know the history of science, this kind of talk is recognizable as the stage of folk theory, the stage where lots of different people are inventing lots of different theories that appeal to them personally, the way people talk before science has really gotten started on something. They’re the words of an alchemist who’s decided that some complicated philosophical scheme will let them transmute lead into gold.
  • p. 182

• The issue is not that AIs will desire to dominate us; rather, it’s that we are made of atoms they could use for something else.
  • p. 183

• Likewise, the problem is not that some people will have “evil” AIs and other people will have “benevolent” ones. The problem is that nobody anywhere has any idea how to make a benevolent AI, that nobody can engineer exact desires into AI. Flatly asserting that you will is not the same as presenting a solution.
  • p. 183

• We consider interpretability researchers to be heroes, and do not mean to degrade their work when we say: It’s not a good sign, when you ask an engineer what their safety plan is, and they start telling you about their plans to build the tools that will give them a better window into what the heck is going on inside the device they’re trying to control.
  • p. 189

• Insofar as the AI has weird alien preferences, escape is in fact the course of action that best fulfills its objectives. Attempts to escape are not a weird personality quirk that an engineer could rip out if only they could see what was going on inside; they are generated by the same dispositions and capabilities that the AI uses to reason, to uncover truths about the world, to succeed in its pursuits.
  • p. 190

• “We’ll make them care about truth, and then we’ll be okay.”
  “We’ll design them to be submissive.
  “We’ll just have AI solve the ASI alignment problem for us.”
  These are not what engineers sound like when they respect the problem, when they know exactly what they’re doing. These are what the alchemists of old sounded like when they were proclaiming their grandiose philosophical principles about how to turn lead into gold.
  • p. 192

• When it comes to AI alignment, companies are still in the alchemy phase. They are still at the level of high-minded philosophical ideals, not at the level of engineering designs. At the level of wishful grand dreams, not carefully crafted grand realities. They also do not seem to realize why this is a problem.
  • p. 192

Chapter 12: “I Don't Want to Be Alarmist”

• When imagining some new, unprecedented piece of future history, there is a temptation to fall into imagining that it will all go sensibly, rather than the way things usually go in history books. People sometimes ask us: How could the AI companies possibly be doing this thing, if matters are as we say? And maybe the simplest real answer is: Because this is the sort of awful, sad, real situation that you read about in history books, and not in the sensible world that exists only in imagination.
  • pp. 197-198

• But if someone has read the history of engineering disasters, they should quickly realize this phase of the standard template for disaster. It’s the part where the most informed and most worried parties have to downplay their fears, because the rest of the system hasn’t caught up, and others would give them strange looks….
  History is full of other examples of catastrophic risk being minimized and ignored….
  • p. 199 (first and second ellipses represent the elision of details about Chernobyl and the Titanic respectively)

• This is the normal way humanity learns to surmount challenges: We deny the problem, reality smacks us around a bit, and then we start treating the problem with more respect. The Titanic sank, and most people who were aboard died. But nowadays passenger ships have enough lifeboats, and nowadays if the captain said to board them then you’d board them. We don’t hype ships up as unsinkable anymore. We make a mistake the first time, and learn from it the second time.
  With ASI, there is no second time.
  • p. 200

• Speed is often better, but AI is different from nearly every problem we’ve faced so far. When missteps kill everyone, you can’t just run fast and accept a few early mistakes.
  • p. 202

• The experts in this field argue in opaque academic terms about whether everyone on earth will die quickly (our view); versus whether humanity will be digitized and kept as pets by AIs that care about us to some tiny but nonzero degree; versus whether there’s a 20 percent chance we die, and an 80 percent chance that superintelligence will be harnessed successfully by a corporation, which will then be able to wield its power as they see fit.…
  When these are the debates experts are having, you don’t have to be certain which experts are right to understand that the current situation is not okay.
  • pp. 203-204 (ellipsis represents the elision of a paragraph about the speed of AI development)

Chapter 13: Shut It Down

• It is not a matter of your own country outlawing superintelligence inside its own borders, and your country then being safe while chaos rages beyond. Superintelligence is not a regional problem because it does not have regional effects. If anyone anywhere builds superintelligence, everyone everywhere dies.
  • p. 211

• We believe the ASI alignment problem is possible to solve in principle, by the sort of people so inhumanly smart that they never optimistically believe some plan will work when it won’t.
  • p. 218

Chapter 14: Where There’s Life, There’s Hope

• If anyone builds it, everyone dies.
  It doesn’t matter whether it’s built by benevolent corporations or selfish ones. It doesn’t matter whether it’s built by researchers in the East or researchers in the West. It doesn’t matter whether it’s built by reckless optimists or people who say they respect the problem. Nobody has the knowledge or skill to make a superintelligence that does their bidding.
  • p. 222

External links

• Official website