The views presented here do not necessarily represent those of Advisor Perspectives.
Some years ago, a friend of mine who had, at the time, just had an important article published in The Atlantic magazine told me that his editor said there are only three themes that get people to read an article: “Who killed whom; who slept with whom; and everything you thought you knew is wrong.”
He could have added another one: “Everybody’s going to die.” That is the hook that AI theorists Eliezer Yudkowsky and Nate Soares use to get people to read their new book, which is literally titled, “If Anyone Builds It, Everyone Dies: The Case Against Superintelligent AI.”
And get attention it has. The book is prefaced by endorsements from a parade of notable figures, including a former CEO of OpenAI, professors at top universities, a Nobel Prize winner, and a lieutenant general.
I wonder if they’ve read it. From my experience with endorsements, probably not. I thought these endorsements must represent the same phenomenon I saw when reading and reviewing a 2010 book on endowment investing. It was a heap of pseudomathematical investment gobbledygook — perhaps presaging the endowment model’s abject failure — but it received a host of endorsements from leading lights in the investment field. Maybe there’s a similar phenomenon in the field of AI. If the author or authors are well known and have lots of friends and associates in the field, they’ll endorse anything without reading it.
Not that Yudkowsky and Soares don’t actually believe that if anyone builds “superintelligence,” whatever that may mean, everyone will die. Apparently, they do. But climate change catastrophists such as Greta Thunberg and many others with more serious science backgrounds also believe that climate change will kill us all. And there are even some grounds for that belief.
If atmospheric warming continued unabated for many thousands of years, there is a chance the earth could become too hot for humans to live on. This is extremely unlikely, though increasing numbers of lives will be lost as weather patterns shift and intensify. It certainly won’t result in an extinction event in our lifetimes, or those of generations of our offspring. But many really believe it. And as with AI, even if it’s not actually going to kill everyone — contra the authors of the book under discussion here — there is cause for concern, just not over imminent and 100% certain extinction.
Yudkowsky’s and Soares’s warning, however, is indeed that AI will kill us all, not only soon, but with virtually 100% certainty. They don’t just mean that AI will get us all killed, for example by being misused by humans in a global war. They mean the AI itself will kill us all.
Yudkowsky’s and Soares’s scenario is highly unlikely, like the claim that climate change will kill us all soon. One need only read how AI is created (for example by reading Georgetown University’s AI security researchers Thomas Woodside and Helen Toner’s three part tutorial, or this post, or many other sources) to wonder, “How the heck can that kill us all?”
AI in its LLM (large language model) form — the form that’s causing all the excitement — really is, at bottom (as the Yudkowsky/Soares book itself explains), essentially a sentence completion algorithm. Given a series of words, it assigns a probability to each of the possible next words, based on their frequencies in billions of word sequences scraped from the internet. (Or if it’s generating an image, a probability to each of the possible next pixels.) That’s about it.
Supremely Silly
Reading Yudkowsky’s and Soares’s book won’t convince you that AI will kill us all, unless you are a whale of a lot more credulous than I am. The book is supremely silly, reading like a badly written science fiction novel for adolescents. To explain how AI could kill us all the authors cook up scenarios written as absurdly implausible fantasy stories.
These scenarios, like fantasy fiction, don’t even make any physical sense. One is driven to wonder if the authors even know much about physical reality. Perhaps the Silicon Valley culture in which they are immersed thinks the whole world is a digital world. They overlook the role of the painstaking and resource-intensive process of building things out of materials obtained by extractive industries.
For example, in one of the authors’ scenarios in which AI kills all humans, the AI builds so many nuclear fusion plants that it heats the atmosphere beyond what humans can stand. They say: “The maximum temperature for fusion plants and factories can probably go up to a few hundred degrees, at least. Hot enough to boil the oceans. Human beings would not survive that.”
For starters, the maximum temperature for fusion plants is not a few hundred degrees. They need to operate at temperatures of 100 million degrees Celsius. They are fusing hydrogen into helium, like the sun does at “only” 15 million degrees, but without the sun’s internal pressure. The authors seem not to know this.
But this is the least of the problems with this scenario. How does the AI gather all the materials needed, and then build and operate millions of nuclear fission plants? (In this scenario, the authors say, “the number of fusion power plants is doubling every hour…”) This is an example of where the authors mistake superintelligence for supercapability.
And, they say, if the superintelligence leaves earth and uses resources from other solar system planets, it would make the sun go dark by launching “Dyson swarms of solar panels” to orbit the sun, blocking its rays. Where, pray tell, would all the material resources and energy come from to build and launch panels to surround the sun and make it go dark? Materials required for such an endeavor would weigh much more than the earth itself. And how would these solar panels be constructed and launched?
This is crazy. Yet there is no sign that the authors are only joking. They are quite serious.
Nevertheless, Soares and especially Yudkowsky are long-time pillars of the AI community. Yudkowsky has been credited with inspiring others in the AI field and introducing important AI players to other AI players or to venture capital funders. They’re taken seriously in the AI world.
I was surprised to see, for example, that Gary Marcus, a noted AI skeptic, who is often acerbically critical of other central actors in the AI field such as Sam Altman, doesn’t outright ridicule Yudkowsky and Soares in his review of their book. In fact, although he pronounces it deeply flawed, he sees some merit in it.
Three Category Errors and a Giant Leap to a Conclusion
The book is, however, truly and very deeply flawed. As a matter of fact, many other discussions of AI are flawed for some of the same reasons.
First, it treats intelligence as unidimensional. Not only this book, but many other discussions of AI speak of the inevitable advance of AI toward “superintelligence,” as if intelligence advances on only one front. But what is superintelligence? It is not defined, either in this book or in many other discussions of it. It is just assumed to mean a degree of intelligence that is greater than any human’s.
And that is where it commits its second error. Like other discussions of AI, the Yudkowsky/Soares book speaks of superintelligence as being a stage in the future, the near future, where AI can do anything any human can do, but better.
But AI being able to do anything better than any human can do is not a stage of intelligence but of capability. Just because it can think things better than humans, doesn’t mean it can do things better. A quadriplegic might be smarter than anybody else, but that doesn’t mean she can do anything any other human can do, but better, such as play baseball. To do that, she would have to have supercapability, not just superintelligence.
But the authors appear to assume that superintelligence means supercapability, without making a persuasive case for that jump in logic. Yes, they make a very weak case for a superintelligent computer being able to cause things to happen better than anyone else — by bribing humans to do the things that a computer doesn’t have the capability to do, for example. However, a direct connection between superintelligence and supercapability is lacking.
A third error is anthropomorphization. The book tries to pretend it is not anthropomorphizing when it speaks of an AI “wanting” to kill everybody. But the way it talks of killer AI sounds an awful lot like talking about a rogue person, a Frankenstein’s monster.
The book’s fourth error is identified by Marcus himself in his review. He agrees with Yudkowsky and Soares on several points. His agreement boils down to agreement that “Rogue AI is a possibility that we should not ignore” and “The public [and governments] should be more concerned than they are.”
But although he admits that “some form of AI might wish to eliminate humans, and unfortunately there is no valid argument that this will not be possible,” he adds that (of course) “that does not make it inevitable.” Marcus is one of the many people in the AI world who believe that not enough attention is being paid to the potential dangers of AI. That is presumably why he is giving the Yudkowsky/Soares book limited credence. But, again, he does say the book is deeply flawed, which indeed, it is.
The Statement on AI risk
The introduction to the Yudkowsky/Soares book begins with a statement that was signed in 2024 by more than 350 notable experts in AI and related fields. It is short. It states, “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” Yudkowsky/Soares then add: “We — Eliezer Yudkowsky and Nate Soares — also signed the letter, though we considered it a severe understatement.”
But the quoted statement doesn’t imply, as Yudkowsky/Soares infer it does, that it’s the AI itself that’s going to kill us. The concerns are mainly that humans’ use of AI might kill us. For example, suppose that the decision as to whether to use nuclear weapons was given over to AI. That could truly pose a great danger, perhaps of human extinction.
If you were a key participant in the AI or AI-adjacent world, and if another key participant asked you to sign this statement, wouldn’t you do so? If you didn’t, it might suggest that you didn’t think there was any risk of extinction if AI were asked to decide whether to use nuclear weapons. You wouldn’t have to agree that “superintelligent” AI will kill us all or even that “superintelligent AI” has an intelligible meaning.
Nevertheless, Yudkowsky and Soares turn that statement into support for their thesis, thus bending the over-the-top AI hype story into support for their claim that it is sure to kill us all.
After reading this book I sold all of my stocks. AI is a bubble.
Economist and mathematician Michael Edesess is an adjunct professor and visiting faculty at the Hong Kong University of Science and Technology. In 2007, he authored a book about the investment services industry titled The Big Investment Lie, published by Berrett-Koehler. His new book, The Three Simple Rules of Investing, co-authored with Kwok L. Tsui, Carol Fabbri and George Peacock, was published by Berrett-Koehler in June 2014.
A message from Advisor Perspectives and VettaFi: Discover something new! Click here to register for our upcoming webcasts.
Read more articles by Michael Edesess
The views presented here do not necessarily represent those of Advisor Perspectives.
