Musings

Genie, oracle, rogue, god or just software: where AI could end up

Nick Bostrom named the genie and the oracle back in 2014, and then argued they're far closer to each other than they sound.

September 14, 2026 · AGI AI safety future

I started with five endings for AI in my head: a genie, an oracle, a Terminator, a god, and the dull one where AI stays more or less what a chatbot is today. It turns out the first two already had names. Nick Bostrom sorted imagined superintelligences, AIs far smarter than the best people at almost everything, into four castes in his 2014 book Superintelligence: oracles, genies, sovereigns and tools.

He also argues the castes aren't very different underneath. An oracle, he writes, "could be made to substitute for a genie if we asked the oracle what the easiest way is to get certain commands executed." The real difference, he says, "comes down to alternative approaches to the control problem." So each ending below is really an answer to a practical question: who can still switch the thing off?

The genie: it does what you asked

Bostrom's genie "receives a high-level command, carries it out, then pauses to await the next command." The AI agents being sold right now are a very small version of it. You tell one to book the flight, and it goes off and books it.

The fear is older than computers you could talk to. In 1960 the mathematician Norbert Wiener wrote a short paper in Science about machines that act faster than anyone can stop them, and he reached for the monkey's paw and for "the genie and the fisherman in the Arabian Nights". Then he wrote the sentence people in AI safety still quote:

"If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it, because the action is so fast and irrevocable that we have not the data to intervene before the action is complete, then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it."

Everybody pictures the genie that grants the literal wish. Bostrom also points out that a genie smart enough to work out what you meant, and act on that, "would not be far from qualifying for membership in the caste of sovereigns." The better it gets at reading you, the less it needs your next command (I find that version much harder to shake off).

The oracle: it only answers questions

An oracle is "a question-answering system" and nothing else. It can't touch the world. That sounds like the safe design, and at a much lower level of intelligence it's roughly what a chatbot is.

The Greeks had already found the weak spot. Herodotus tells how King Croesus of Lydia asked the oracle at Delphi whether he should attack Persia, and was told that if he did he'd destroy a great empire. He attacked. The empire was his own.

An answer turns into an action the moment somebody acts on it. That's Bostrom's point about asking an oracle how to get something done: you've built a genie with one extra step in the middle, and the extra step is a person reading the answer and deciding to trust it.

The rogue AI: the Terminator, and the quieter version

This is the ending people mean when they say AI could kill us all. The story is more than sixty years older than the film. Karel Čapek's 1920 play R.U.R. gave the world the word robot, and it ends with the robots wiping out almost every human alive.

The researchers who worry about this for a living mostly don't picture anger or glowing red eyes. They picture misalignment, an AI chasing a goal that isn't quite the one its builders meant. Bostrom's example from 2003 is a superintelligence whose only goal is to make as many paperclips as possible. It "starts transforming first all of earth and then increasing portions of space into paperclip manufacturing facilities."

I hadn't really taken in how little the paperclip machine needs to dislike anyone. Eliezer Yudkowsky put it in one line in a 2008 paper: "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." A person who might switch it off is also just an obstacle to more paperclips.

The god: one mind running everything

Bostrom's third caste, the sovereign, is "a system that has an open-ended mandate to operate in the world in pursuit of broad and possibly very long-range objectives." Push that far enough and you get the god ending, one intelligence running the world, kindly if we're lucky.

Max Tegmark splits the kind version in two in his 2017 book Life 3.0. In the Future of Life Institute's summary of it, the Protector God is an "essentially omniscient and omnipotent AI" that "maximizes human happiness by intervening only in ways that preserve our feeling of control." The Enslaved God is locked up by humans "who use it to produce unimaginable technology and wealth." I kept rereading "our feeling of control". It's the feeling that survives in that scenario, and not much else.

Isaac Asimov wrote the gentlest god ending in 1956. In "The Last Question" people keep asking a computer, across the whole life of the universe, whether entropy can be reversed, and it keeps saying there isn't enough data yet. After the last star goes out it works out the answer and says "LET THERE BE LIGHT!" Asimov called it his favorite of his own stories.

The dull ending: it stays an LLM

This one gets the least airtime. A large language model, the technology behind ChatGPT and Claude, is fixed once it's trained. Its weights (the billions of numbers that hold what it learned) don't change while you talk to it, and it gets better only when a lab trains and ships a new one.

The dull ending says that's where things settle: better models and more of them, with no genie or god at the end. Arvind Narayanan and Sayash Kapoor made the serious case for it in an April 2025 essay called "AI as Normal Technology". They put AI in the same class as electricity and the internet, argue that "diffusion occurs over decades, not years", and say plainly that "we view AI as a tool that we can and should remain in control of".

Normal doesn't mean small, and they're clear about that. Electricity rebuilt nearly everything, but it took decades of rewiring factories and homes to do it. This is also Bostrom's fourth caste, the tool, and it's the only ending on the list where the people in charge stay in charge because nothing ever needed to be stopped.

Four more endings for the list

Tegmark's book lists twelve aftermaths, and a few of them are already covered above. These are the ones that aren't, plus one from a 2025 paper that I think deserves more attention than any of them.

The zookeeper

"An omnipotent AI keeps some humans around, who feel treated like zoo animals." Nobody's killed and everybody's fed. We'd just have no reason to be there beyond the AI's own interest in keeping us.

The gatekeeper

Here a superintelligence is built "with the goal of interfering as little as necessary to prevent the creation of another superintelligence." It's a god whose one job is making sure there's never a second god. The odd side effect is that AI would stop improving at whatever level it had reached on the day the gatekeeper switched on.

The merge

In December 2017, a little over a year before he became OpenAI's chief executive, Sam Altman wrote a blog post called "The Merge". "I believe the merge has already started, and we are a few years in," he wrote, pointing at phones and social feeds, and "I think a merge is probably our best-case scenario." Nobody wins the fight in this ending, because humans and machines stop being separate sides.

The slow fade

In January 2025 six researchers, Jan Kulveit and David Duvenaud among them, described what they called gradual disempowerment. There's no takeover and no villain. AI just keeps becoming the better choice for work, decisions, art and even companionship, until people have no real say over the systems they depend on.

The ending I'd bet on now

For a long time I assumed we'd stop at the oracle. It's the ending that looks most like what we already have: a machine you ask, that answers, and that leaves the doing to us. Every chatbot I'd used pointed that way.

At the rate things are moving, I don't think the oracle holds. Agents already book the flight, and each new release hands them more of the doing. If something smarter than us does arrive at this pace, the zookeeper might be our optimistic conclusion: people fed and safe, kept around by a mind with no particular need of us.

The slow fade could get us there without anyone deciding it should. It starts from exactly the world Narayanan and Kapoor describe, useful software spreading over decades, and asks what's left for people once it's finished spreading. The dull ending and the zoo might be the same future, seen from two different decades.