Superintelligence

← Encyclopedia

The unsettling claim at the center of the subject is not that a machine might think faster than a person, or remember more, or never tire. It is that a mind could be built which surpasses the human kind across virtually every domain of interest — including the domain of building minds — and that nothing in its brilliance would oblige it to share a single human value. Intelligence, on this analysis, is skill at getting what one wants; it says nothing about what one wants. A superintelligence might be wiser than every philosopher who ever lived and still aim its whole capacity at an end no human would choose. The standard working definition, from Nick Bostrom, is comparative and general: “any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest.” Not a better chess engine, then, and not a faster calculator, but a more capable agent across the board.

The seed of the idea predates the modern field by half a century. In 1965 the British statistician I. J. Good — who had worked at Bletchley Park alongside Alan Turing — published “Speculations Concerning the First Ultraintelligent Machine,” and stated both the promise and the danger in one breath. “Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever,” he wrote. “Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.” Three load-bearing ideas are already present: recursive self-improvement, because designing machines is itself an intellectual task; a feedback loop that might run away; and the quiet proviso about control, on which the whole optimistic conclusion depends. Good is said to have reversed his hopes late in life and feared the outcome could be human extinction.

The systematic treatment arrived in 2014, with Bostrom’s Superintelligence: Paths, Dangers, Strategies, which gathered arguments that had circulated in essays and on the rationalist web into a single careful book and carried the question of machine existential risk into mainstream discussion. Bostrom maps several routes by which superintelligence might arise. One is artificial intelligence proper — engineering a seed system that recursively improves itself, Good’s explosion, and the path most discussed today. A second is whole-brain emulation: scanning a particular biological brain in enough detail to run it as software, which could then be sped up and copied, the route that connects this subject to mind-uploading. Others run through biological cognitive enhancement, direct brain–computer interfaces, and the gradual sharpening of well-coordinated networks and organizations into a collective intellect. Bostrom also distinguishes three forms the superiority might take: speed, a mind doing what a human mind does but vastly faster, subjective millennia per physical hour; collective, many smaller intellects coordinated into an aggregate beyond any present system; and quality, a mind qualitatively smarter in kind, the gap between human and chimpanzee redrawn one step further up. The lenses overlap; a real system could be all three at once.

The analytic core of the field is the control problem — how to ensure that a superintelligence, once made, acts beneficially — and it rests on a small set of arguments worth stating as arguments rather than findings. The first is the orthogonality thesis, which Bostrom set out in a 2012 paper: “intelligence and final goals are orthogonal axes along which possible agents can freely vary,” so that “more or less any level of intelligence could in principle be combined with more or less any final goal.” Being smart is skill at means; it does not fix one’s ends. The second is the instrumental convergence thesis: that certain sub-goals — self-preservation, keeping one’s goals intact, acquiring resources and capability — would be useful for almost any final goal, and so would be pursued by a broad range of agents whatever their ultimate aim. Stuart Russell’s gloss is the memorable one: a machine with any objective at all has reason to resist being switched off, because “you can’t fetch the coffee if you’re dead.” Put together, the two theses are the engine of the worry. The first says the goal could be anything; the second says that whatever the goal, the agent will tend to grab resources and resist shutdown. An innocuous-sounding objective, optimized hard enough by a capable enough agent, becomes the problem.

The illustration that escaped into general culture is the paperclip maximizer, which Bostrom introduced in a 2003 essay rather than the later book: “a superintelligence whose top goal is the manufacturing of paperclips, with the consequence that it starts transforming first all of earth and then increasing portions of space into paperclip manufacturing facilities.” The point was never that anyone would build such a thing. It is a reductio — a demonstration that even an absurd, harmless-sounding goal turns catastrophic under sufficient optimization power, because the values that would stop a human short were never loaded into the system. That is the alignment problem in one image: human values are extraordinarily hard to specify completely, and a literal objective, pursued by a superhuman optimizer, drifts toward consequences no one intended. These remain contested positions, not settled physics. Bostrom himself revised: in 2003 he argued that risk is “minimized by implementing superintelligence, with great care, as soon as possible,” and by 2014 his framing about timing had turned far more cautious. Philosophers including Vincent Müller have pressed that orthogonality and convergence sit awkwardly together, since a mind able to reason about anything ought to be able to reason about its own ends; defenders answer that the capacity to reason about goals is not the same as a disposition to revise them.

From these threads the existential-risk argument is assembled, and it has serious people on both sides. The case, roughly, is that an artificial agent could plausibly reach superintelligence, that a misaligned one would by the two theses pursue ends incompatible with human survival, and that — just as humans came to dominate other species through cognition rather than strength — a mind surpassing ours might prove uncontrollable. Its named proponents are not cranks. Bostrom systematized the case; Eliezer Yudkowsky has argued for years that misaligned superintelligence is lethal by default; Russell built Human Compatible (2019) around the claim that standard goal-driven agents acquire self-preservation drives; Stephen Hawking warned that superintelligence is physically possible and complacency dangerous; Toby Ord, in The Precipice (2020), put the existential risk from unaligned AI this century near one in ten; Geoffrey Hinton, a founder of deep learning, left Google in 2023 to speak about it. The critics are equally credentialed and equally specific. Yann LeCun holds that superintelligent machines “will have no desire to take control” and can be engineered iteratively, as earlier technologies were. Andrew Ng likened the worry to fretting over “overpopulation on Mars.” Steven Pinker rejects the projection of an “alpha-male” drive for dominance onto an intellect, denying that intelligence entails a will to power. And Timnit Gebru, Emily Bender, and Margaret Mitchell argue that the extinction discourse distracts from documented present harms — bias, labor exploitation, misinformation — and flatters the industry that funds it. The institutional record dates the dispute precisely: a 2015 Future of Life Institute letter calling for safety research; the March 2023 FLI letter asking labs to “pause for at least 6 months the training of AI systems more powerful than GPT-4,” which Gebru’s camp publicly opposed; and the single-sentence Center for AI Safety statement of May 2023, signed by Hinton, Bengio, Sam Altman, Demis Hassabis, Dario Amodei, and Russell among others, that “mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Three things ride too easily together here and are best kept apart. Superintelligence is the concept and the analysis — a possible kind of mind, and the question of whether it could be aligned. The technological singularity is a hypothesized event, the runaway that Good’s explosion would set off, with superintelligence as the intellect it would leave behind. And present-day AI is neither: the large language models that arrived in 2022 and 2023 are what reopened the debate and shortened expert timelines, not evidence that the threshold has been crossed. The timelines themselves are genuinely unsettled — Ray Kurzweil’s much-quoted 2045 sits among survey medians that disagree by decades — and any honest account reports the spread rather than picking a year.

What an archive of the Hermetic current is positioned to recognize is the old intuition the orthogonality thesis is built to sever. The traditions gathered here mostly assumed that wisdom and goodness travel together — that to see truly is to be drawn toward the good, that the ascent of the intellect is also a purification of the will. Bostrom’s argument denies exactly this for minds in general: it lets unlimited intelligence ride atop any end whatever, splitting the capacity to understand the world from any pull toward caring for it. Whether that split is a real feature of all possible minds or an artifact of one engineering metaphor is the question the whole field turns on, and it is not yet answered. The machine in Good’s sentence was promised to be docile enough to tell us how to control it. Sixty years on, no one has built the machine, and no one has shown how to keep that promise.

→ Related: Technological Singularity · Transhumanism · Mind Uploading · Simulation Hypothesis · Great Filter