George Orwell and Dune Tried to Warn Us About AI

"Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them." — Frank Herbert, Dune

There is a comforting lie we tell ourselves about dystopia: that we will recognize it when it arrives. That there will be a boot, a face, a wall, a siren. That the moment will announce itself, and history will hand us a clear line — before, and after.

Orwell never believed that, and neither did Frank Herbert. Both men, writing decades apart, arrived at the same conclusion: the dictatorship that actually gets built is not the one that terrifies you into submission. It's the one that makes submission feel like comfort — and by the time you notice, you're no longer sure there was ever a line to cross. You just wake up one day already on the other side of it, scrolling.

I'm writing this a few days after OpenAI released GPT-6 Astra and its own president stood in front of the press and said, without irony, "welcome to the AGI era." Not a research paper hedging its claims. Not a cautious "this may be a step toward." A declaration. And I felt something I suspect a lot of people my age felt reading that headline: not excitement, but a kind of vertigo — the sense of watching a door close from the inside, in real time, and understanding that you were never actually asked whether you wanted it shut.

That's the feeling this essay is trying to name. Not "AI might be dangerous someday." The dread that we are already living inside the thing our ancestors warned us about, and that the utopia they described — the one built from comfort, convenience, and surveillance rather than chains — is not coming. It's here. We're standing in it right now, and the doom isn't on the horizon. It's the ground under our feet.

The Utopia Is the Doom

Orwell's genius in 1984 wasn't inventing surveillance or torture — those were old tools. His genius was showing that a "perfect society," any perfect society, requires the same totalizing administration of human life whether it's built by a tyrant or a well-meaning technocrat. A utopia and a nightmare share the same architecture. They differ only in who's holding the blueprint and how the walls are decorated.

Look at what we've built and called progress. Claude and ChatGPT answer any question before the itch to wonder even forms. Google Maps has erased the internal compass humans spent a hundred thousand years developing. Food, entertainment, validation, romantic prospects, distraction from despair — all of it arrives within seconds, optimized, personalized, frictionless. By almost any definition offered across human history, this is utopia. Nobody had to march us into it. We asked for the app, rated it five stars, and left a review.

And that is precisely why it's so hard to fight. Orwell understood that a creature with nothing left to overcome no longer needs to be free. Effort, struggle, even boredom, aren't bugs in human life — they're the mechanism by which a self gets built in the first place. Strip them out efficiently enough and you don't get a liberated human. You get a well-fed, well-entertained organism that has quietly stopped needing agency, because agency was always slower and more painful than the alternative on offer. We are not approaching that state. We have been living inside it long enough that most of us have forgotten what the alternative even felt like.

Bread, Circuses, and Sedation by Design

Rome understood this two thousand years before Silicon Valley did: keep the population fed and entertained and they will not organize, will not question, will not look up from the spectacle long enough to notice who's writing the laws. Orwell's Oceania ran the same program on its Proles — cheap alcohol, gambling, pornography, sentimental novels, music composed by machines for people who were never meant to think about what they were consuming.

Swap the vocabulary and you have the present. The infinite scroll is the circus. Algorithmic outrage is the arena. None of it is imposed on us by a Ministry of Truth issuing directives. It's produced by an incentive structure that discovered, almost by accident, that human attention is the most valuable extractable resource on the planet — and that people will hand it over willingly if the interface is pleasant enough.

But it isn't only entertainment doing the sedating. It's work itself. Look honestly at the shape of a normal week: the exhaustion isn't incidental, it's functional. A workforce run ragged by long hours, precarity, and the low-grade dread of being made redundant by the very technology it's forced to use every day has no leftover hours, and no leftover nervous system, to organize with. Orwell wrote it plainly: someone with spare time and spare energy might start to think, and someone who thinks might start to ask questions. A population is kept working not because the work is necessary, but because of a vague, correct suspicion that it would be dangerous if it had leisure. That isn't a conspiracy theory. It's a description of an equilibrium — one that doesn't require a single meeting in a single boardroom to sustain itself, because everyone downstream of it is simply doing their job, and their job happens to keep everyone too tired to look up.

The Tower With No Guard, and the Data Underneath It

Jeremy Bentham's panopticon was a prison design: a ring of cells around a central watchtower, built so a single guard could see into any cell at any moment, while no prisoner could ever tell whether they were being watched at that particular second. The genius of the design was that the guard didn't even need to be present. The mere possibility of observation was enough to make the prisoners police themselves.

We built that tower voluntarily and moved into the cells ourselves. Ring doorbells, license-plate readers, fitness trackers logging our sleep and heart rate, social platforms inferring our politics, our fears, and our desires from behavior we don't even register as data. When tech executives say, in plain language, that everyone will be recorded and that this will result in people behaving well, they are describing Bentham's tower with a straight face and calling it safety. "Best behavior" always means the behavior the watcher prefers — not virtue, but predictability, and a population that's predictable is a population that's governable without ever needing to pass a law.

Orwell's telescreen needed government installation. Ours came bundled with a subscription plan we chose, an app we downloaded, a device we paid for with our own money and carry into every room of our own home.

Situational Awareness, and the Part Nobody Wants to Say Out Loud

In 2024, a former OpenAI researcher named Leopold Aschenbrenner published a long essay called "Situational Awareness" that quietly did more to explain the next decade than almost anything else written about AI. Its argument, stripped of its technical scaffolding, was simple and cold: the compute and capital being poured into frontier AI is scaling on a trajectory that, if it continues, produces systems capable of outperforming humans at nearly all economically valuable cognitive work — and it argued this wasn't a distant hypothetical, but a matter of a few years. GPT-6 Astra landing with OpenAI's own leadership invoking the word AGI is not a coincidence sitting outside that trajectory. It's a checkpoint on it, arriving roughly on schedule.

The essay's more unsettling argument wasn't about the technology at all. It was about what happens to power once a handful of labs — and the handful of governments that will not sit by and let those labs stay independent — realize what they're holding. Aschenbrenner's own prediction was that a project this consequential would not stay in the hands of a private company for long; that national security states, the US chief among them, but China and Russia moving in the same direction, would treat frontier AI the way they once treated the atomic bomb — as a strategic asset too decisive to leave outside state control. Whether or not that exact scenario plays out on his timeline, the underlying logic is the one Orwell already gave us in You and the Atom Bomb: when a weapon is cheap, power diffuses. When it's expensive enough that only an industrial state can build it, power concentrates into the hands of two or three actors who then spend the rest of history in a permanent, silent standoff over the rest of us.

That is the part of this story that should actually keep you up at night — not that a chatbot got smarter, but that the small number of people who control the compute, the capital, and increasingly the compliance of governments, are converging on the same closed circle Orwell called Oceania, Eurasia, and Eastasia. Superstates that don't need to conquer each other, because they've already conquered their own populations, quietly, with our full cooperation.

Dune's Older, Colder Warning

If Orwell diagnosed the mechanism of control, Herbert diagnosed something further upstream: the danger of surrendering cognition itself. Ten thousand years before the events of Dune, humanity fought and won the Butlerian Jihad — a holy war against thinking machines, sparked by the discovery that when people handed their reasoning over to computers, something in them didn't just relax. It atrophied. The Orange Catholic Bible that survives that war carries a single commandment above the rest: thou shalt not make a machine in the likeness of a human mind.

This isn't a warning about robots turning violent. It's a warning about a species that stops training its own judgment because a machine does it faster, and doesn't notice the muscle disappearing until it's already gone. In 2025, MIT researchers monitoring brain activity in people writing essays with AI assistance found measurably reduced neural engagement compared to people writing unaided. That's not a metaphor. That's a body doing less of what it used to do, recorded on a scanner. Ask yourself honestly how many routes you could navigate without turn-by-turn directions, how many arguments you've fully worked through in your own head instead of asking a chatbot to do it for you, how much of your own thinking you've already, quietly, outsourced.

Orwell's Winston Smith was dangerous to the Party specifically because he still had a private, unassisted interior life. The Party's entire project was colonizing that space. AI doesn't need to colonize it by force. It only needs to become the default first move whenever a thought requires effort, until the interior space stops being exercised, then stops being missed, then — this is the part that should actually frighten you — stops mattering, because the systems making the decisions that shape your life were never going to consult your interior life in the first place.

What Are You Worth, If the Machine Is Better

I want to say something directly to anyone reading this who feels what I feel: that the ground is moving, that a degree bought with years of debt and effort might be worth less than the model that graded the essays for it, that the entire premise of "build skills, get better, earn your place" is being quietly rewritten while you're still mid-sentence in your own life plan.

That fear is not irrational. It is, in fact, one of the most rational things a person my age (GenZ) can feel right now, and anyone telling us otherwise either hasn't been paying attention or has a professional incentive not to. Orwell would recognize this exact moment — a generation raised to believe effort leads somewhere, discovering that the connection between effort and reward is being severed on a civilizational scale, on purpose, by systems that were never designed with your flourishing as the objective. Herbert would say you are watching, in real time, exactly what happens to a society that forgets its own commandment and hands its thinking over to the machine, believing it would set them free.

But notice the shape of the fear itself: it assumes your worth was always a function of economic output — that you are only as valuable as what you produce relative to a model that produces faster. That premise was fed to you. It wasn't discovered by you, and it isn't true simply because a hundred systems are currently built as if it were. Orwell's answer to Winston's despair was never "don't worry, it'll be fine." It was smaller and, in a strange way, harder: hold onto the parts of yourself nobody can price. A private thought. A relationship that isn't optimized. A skill you keep for no reason except that it's yours. That isn't a solution to the doom. It's the only form of resistance available to someone standing inside it, and it happens to be the one thing no model, no matter how capable, can be handed on your behalf.

What's Actually Being Asked of Us

Neither Orwell nor Herbert offered an escape hatch, and it would be dishonest to pretend this essay found one they missed. What both of them offered instead was a diagnosis precise enough to act on, even from inside the utopia, even while the storm is already overhead rather than approaching. The fight was never going to look like a war. It was always going to look like a thousand small, mundane surrenders — a question you didn't bother to sit with, a route you didn't bother to learn, a doorbell that watches the street so you don't have to, a feed that decided what mattered today so you didn't have to decide for yourself, a job that exhausts you exactly enough to make sure you're too tired tonight to ask who benefits from your exhaustion.

None of these, on their own, is a boot on a face. That's precisely why they work, and precisely why the dread is correct. We are not waiting for the dictatorship. We are already inside the utopia it was built to resemble, and the two, as both writers tried to tell us, were never actually different buildings.

"Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them." — Frank Herbert, Dune