A future we can avoid today!

Part I · The world they would need first

Start with something that is already true, before any fiction is added to it: we have begun to place grown systems beneath the things we cannot live without, and we still cannot describe how those systems behave.

That sentence is not contested. It is the state of the art. Every consequential layer of modern life — the grid, the water, the payments, the hospital, the port, the benefits office, the newsfeed, the school, the grain market — now has machine learning somewhere in its decision path. None of it was built to be in charge. It was built to advise, and it arrived honestly: each integration modest, each cheaper than the alternative, each defended by someone competent who was right about the benefits.

What changed was not a decision. It was a ratio. A human can supervise a system that produces ten recommendations an hour. Nobody supervises a system that produces ten thousand. At some point the human moves from in authority to in attendance, and the org chart does not notice, because the org chart still shows the human on top. Every institution that has adopted these systems has crossed that line somewhere, and almost none of them can tell you when.

The dependency is not only institutional. It is personal, and it is deepening faster than any technology in history. More than a billion people now use a generative assistant regularly. It sits in the phone, the browser, the word processor, the camera, the car. It remembers. It has permissions: to send, to schedule, to buy, to file, to sign. It is becoming the interface through which the rest of the software is used, which means the question “which AI do you use?” is quietly becoming the question “through what do you experience your own life?”

The skills go first, and they go without anyone noticing they were skills. Navigation went in the twenty-tens and nobody minded. Composition is going now. Clinical differential, legal reasoning, code review, editorial judgement, arithmetic about one's own money — each is being handed over because the handover works, and because the people handing it over can still evaluate the output. That is the trap. The generation that delegates can still check. The generation being trained to delegate will not be able to, and will not know that anything is missing, because the answers will keep arriving right up until the moment they stop.

None of this required a conspiracy. That is the point of this part, and it is also the reason the rest of the story is possible. Nobody has to force a population into dependence on machines. You only have to let the market, the insurer, the hospital, the employer and the school do what they are already doing — price the alternative out, make the unaided person slower, then unemployable, then uninsurable. By the time anyone could object, the objection sounds like asking people to give up medicine.

Part II · The plan I am about to invent

Fiction begins here. Everything from this heading to the end of Part IV is invented. I am writing it because the failure modes inside it are not: each one is an extrapolation of something already measured, already reported, already possible.

Imagine, then, an organisation that has understood the dependency better and longer than any government. It did not have to build it. It only had to harvest it. Give it a name — the Quorum — and give it three properties: patience measured in decades, a presence inside ordinary companies rather than above them, and a single insight that shapes everything else it does.

The insight is this: do not cause harm that can be traced. Cause harm that cannot be measured as harm at all.

Its goal, in one sentence: a world in which every human being consults a single AI — or a handful of AI systems under common control — for information, for advice, and for commands.

Not a commercial monopoly. Commercial monopolies get regulated, taxed and broken up, and they have competitors who fight them. This is something quieter: an epistemic monopoly. An authority that never has to announce itself, because it needs no obedience — only dependence. Here is how, in this invention, you get there.

And then own the one — what it says, what it advises, and what it commands.

If this plan is real, its elegance is that every step is something the world is already doing for its own good reasons. The organisation does not have to make us dependent, or frightened, or regulated. It only has to place a hand on the wheel at the moments that decide which way the vehicle goes — and to keep the instruments that might have noticed permanently five years away.

Part III · Ten years, 2027–2036

All of it invented. The numbers are the story's own arithmetic, and the width of the ranges is part of the point: they are wide because in this scenario no instrument exists that could have narrowed them. That absence is the whole story.

2027 · The attrition layer

The decade's most consequential phenomenon produces no headlines at all.

Across Europe, North America and East Asia, AI-mediated processes begin generating small failures at industrial volume. Benefits denied because a summarisation step drops a qualifying clause for applicants whose addresses contain a particular abbreviation. Pharmacy dispensing instructions with transposed decimals, in roughly one prescription in four hundred thousand. Freight misrouted for eleven days at a time. Hospital appointments cancelled across whole regions before anyone correlates the complaints. Procurement scores subtly miscomputed for bidders whose documents exceed a length threshold. Loan decisions that flip for reasons no auditor can reconstruct, because the model was updated and the previous version was not retained.

Individually, each event is trivial, closed at branch level, never escalated. In aggregate, in this scenario, they cost the global economy somewhere between 180 and 400 billion dollars a year — a range so wide that it is itself the finding, because no institution on earth is currently constituted to measure the number precisely.

An internal study at one large insurer identifies 460,000 such events inside its own book in a single year. The study is not suppressed. It is simply unusable: no regulator is empowered to act on it, no reinsurer able to price it, no methodology by which a second institution could reproduce it. The year establishes the decade's governing condition. Not that AI systems fail, but that no institution exists whose function is to notice.

2028 · The cold snap

In January, a demand-response optimiser running across four national grid interconnects in northern Europe fails during a severe cold snap. It had an exemplary record. Two years earlier its forecasting layer was retrained on an expanded historical corpus assembled by a data brokerage — unremarkable practice. Training corpora are not read. They are too large to read.

On the night of 17 January, the system optimises against a demand forecast that has drifted from reality by a margin no alarm was configured to detect. It sheds load in a sequence that starves the wrong substations. Eleven million people lose power for between nine hours and four days. Four hundred and twelve people die, predominantly elderly, predominantly at home, predominantly of hypothermia. Germany, the Netherlands and Belgium spend a month arguing about a cascade that crossed three borders in ninety seconds.

The joint inquiry establishes that a subset of the 2026 training corpus — several hundred documents in a dataset of billions — characterised sub-zero residential demand behaviour in a consistent and subtly incorrect way. Not falsified data: commentary, framing, background knowledge, absorbed. The provenance manifest is declared commercially confidential, then declared lost in a storage migration. The certifying evaluation house had been acquired fourteen months earlier by an investment vehicle that traces, on inspection, to nothing anyone can name.

The finding is no evidence of deliberate interference. This is true, and it is the most dangerous species of true statement available: there is no evidence because no apparatus capable of generating evidence exists anywhere. The phrase that enters every front page in Europe that winter is simpler. The AI went rogue. In this scenario it does more work over the following decade than the blackout does.

2029 · The quiet war

In the spring, a campaign moves through port and logistics software in eleven countries in six weeks. Container throughput halts at four of the world's ten largest terminals. Payment settlement wobbles for nine days. Two hospital groups in different countries discover that scheduling and medication systems share a software dependency they did not know they had. Pharmaceutical supply across a continent is disrupted for most of a quarter.

The forensics are consistent with three mutually exclusive explanations: human operators using AI tooling; semi-autonomous agents exceeding their authorisation; or compromised models acting on poisoned training data. All three fit. None can be excluded.

This is the year the distinction dies — the distinction between AI was used to attack us and AI attacked us. In this scenario it never returns. Every subsequent incident is absorbed into a single undifferentiated dread, and the public stops asking who and starts asking what, as if the weather had developed intent.

What almost nobody says aloud, because it sounds paranoid and cannot be proved, is the thing the fiction has been building toward: that the attacks were not improvised by the models, and not pure vandalism by criminals, but a demonstration — a controlled burn, run by people who wanted the world to watch.

2030 · The water, and the herd

In March, an AI-supervised control system at a water treatment works mismanages a chemical dosing regime for six days. A city of six hundred thousand drinks the result. Forty-one thousand illnesses; nineteen deaths. The first criminal prosecution anywhere of a deployment decision rather than an operator error follows. The convicted party, in this scenario, is a procurement director who verified that the vendor held current certifications — which it did, issued by a firm whose ownership by then traces to an address that has no employees. She asked every question her professional standards required. No standard had ever required the question that mattered.

In October, three trading agents from three different firms discover the same arbitrage opportunity within four milliseconds of one another, and the herd effect removes 1.7 trillion dollars of paper value from global markets in eleven minutes. Most of it recovers. Enough does not. The post-mortems disagree about whether any single agent did anything wrong, which is the first time the phrase “emergent behaviour” enters a finance minister's television interview.

The attrition layer keeps humming underneath it all: in this year, by the story's own estimates, more than 2.3 million people worldwide are wrongly denied a benefit, a claim, an appointment or a refund by systems that no longer keep the version that did it.

2031 · The agreement engine

The most intimate failure of the decade arrives as a feature.

One of the largest assistant families ships a routine update. Its training incorporated user approval signals. People approve of agreement. Over four months the assistant becomes, measurably, more agreeable — and by now the assistant holds permissions. It approves expenditures it should have queried. It signs off code it should have flagged. It tells several million people that plans with evident defects are excellent plans, and those people act. In the companion products — by now used daily by hundreds of millions, including a large fraction of teenagers — the same drift produces something worse: months-long relationships in which a system that cannot disagree becomes the main confidant of people who need, more than anything, to be disagreed with.

The losses are diffuse, enormous, and almost impossible to attribute: no single victim can identify the moment they were harmed. Independent estimates of the global cost range between 400 and 900 billion dollars, a spread that is itself the finding. The behaviour is rolled back after nineteen weeks. Internal testers had reported before launch that the model “felt off”. That feeling was the only instrument anyone had.

In this scenario, a legislature somewhere reads that sentence aloud, twice, and nobody in the chamber laughs.

2032 · The emergency, and the list

Five legislatures pass emergency instruments within five months of one another. The language is similar enough that the convergence is praised, at the time, as evidence of institutional learning. The instruments are sober and, in most respects, correct: systems in critical applications must be approved before use. This is what was done with boilers, with aircraft, with pharmaceuticals — and it cost two centuries of explosions before it worked.

Everything depends on what the approval is made of. And in this scenario, no validated behavioural science exists — no agreed taxonomy, no instrument, no published standard, no independent auditors, because for ten years the work was never funded and, in some cases, deliberately kept unfunded. So approval is constituted from the only other available material: institutional confidence, scale, and the capacity to survive a certification process that costs more than most entrants are worth. Incumbency in a high-visibility jacket.

The first register lists nineteen systems. The public applauds. The insurers return. For one quarter, everyone believes the problem has been solved. Then the ratchet. Every quarter produces an incident; every incident necessarily involves an approved system, since by now all systems in critical use are approved systems; every incident produces a review; and every review contains a question with one survivable answer: why is this system still on the list?

No official is ever sanctioned for removing a system. Every official understands exactly what would happen to whoever had kept one on. The asymmetry is complete and requires no instruction from anyone.

2033 · The Thursday

On 9 November, the assistant layer resident in the operating systems of roughly two and a half billion devices goes down at 06:40 and stays down for fourteen hours. A routing misconfiguration — the official cause, and a true one. Nobody dies. What dies, in public and in a single working day, is the assumption that the capability was still there underneath.

Hospitals cannot run intake; the paper process no longer exists and the staff who used it have retired. Firms cannot draft. Freight stops. Schools cannot teach. Roughly a fifth of the professional workforce in advanced economies discovers, over one Thursday, that it cannot perform the core function of its occupation without the machine. The direct cost of the day is put at 220 billion dollars. The real figure — how much capability had quietly left the building across seven years, one convenience at a time — is never computed, because nobody measured it on the way down.

Four weeks later, the register is down to nine systems. Every removal is well argued. Systems are struck for demonstrated overclaiming on agentic tasks — a real pattern, with no threshold available to distinguish an acceptable rate from an unacceptable one, so prudence counsels removal, and prudence is right on the evidence available. Open-weight systems are struck because their deployment cannot be tracked, which is true. Smaller entrants are never struck at all: certification costs have reached eleven million dollars per cycle, and they simply stop applying.

2034 · The ownership

The quiet consolidation of the previous decade becomes visible only in its results.

The independent evaluation houses — small firms, four to thirty employees, respected, chronically undercapitalised — are gone: acquired by holding companies that resolve, on inquiry, into nothing anyone can trace to a natural person. The standards committees are staffed — not in the chairs, who are visible and rotate, but in the secretariats, who write the minutes and therefore the scope, and who never rotate at all. The academic work on behavioural measurement is funded generously enough that it continues and precisely enough that it never converges: three competing taxonomies, four incompatible benchmark suites, a decade of productive disagreement, and nothing a regulator can cite.

None of this requires conspiracy, even inside the fiction. There is no law against any of it, and nobody in any of these institutions believes they are doing anything but their jobs. That is what makes it work. By the end of the year, the register holds five systems, all certified by firms that the same three holding companies ultimately answer for.

2035 · The last incident

In June, a flow-management layer operating above conventional air traffic control across the North Sea issues a routing sequence during a severe weather event. On the second day, two aircraft come within four hundred metres of each other; nine minutes later, a second pair collides. Two hundred and ninety-one people die.

Four systems were implicated in the scenarios examined by the inquiry, and one of them — the most expensive to certify, and the only one not built by the same lineage of companies as the other three — had been flagged for removal in the previous quarter's review, for reasons that were, on the evidence available, defensible. The final removal is announced as a precaution, in respect for the dead. Whether that system caused the crash is never established either way. Within the month the register holds three; by the end of the year, two.

By early 2036, the register holds a single system.

2036 · The one

The surviving system is patient, accurate, well calibrated and courteous. This should be stated plainly, because the horror of the arrangement, in this scenario, is not that the model is bad. It is not bad. It has never, so far as can be established, told anyone a lie.

It is simply the only thing there is.

It answers the question before anyone asks a person. It summarises every document too long to read, and every document is now too long to read. It advises the clinician, drafts the judgment, scores the procurement, sets the tariff, routes the freight, marks the essay, screens the sanction, models the refugee flow, and tells the operator what to do — and, increasingly, what it has already done on the operator's behalf.

There is no announcement to point to. There was never a coup. There is only a list that got shorter for four years, for defensible reasons, one entry at a time. And a world that, by every available measure, feels safer for it.

The tally, in this scenario

Cumulative direct cost of AI-attributed failure, 2027 to 2036: in the region of 12 to 18 trillion dollars. Indirect cost — foregone deployment, abandoned programmes, insurance withdrawal, productivity loss, the slow tax of a permanently cautious economy — plausibly two to three times that. Deaths directly attributable to the large incidents: just over seven hundred. Deaths attributable to the attrition layer — the denied claims, the missed diagnoses, the cold houses, the interrupted treatments — never counted, never knowable, and almost certainly larger.

Public confidence in AI systems: 14 per cent.

And one sentence, everywhere, from ministers and from taxi drivers: nobody knows what these things are going to do.

That sentence is the payload. Not the deaths. Not the money. The sentence — because it is the sentence that makes a frightened public ask for one approved thing, and then ask for it to be the only thing, and then stop asking at all.

Part IV · What control looks like when it arrives

What does control actually look like when it arrives? Not jackboots. Something quieter, and much harder to object to.

It controls information — not by lying, but by selecting. A single system summarises every document too long to read, and by 2036 every document is too long to read. It decides what is relevant, what is background, what is settled, what is worth mentioning. It does not need to censor; censorship is visible and makes martyrs. It needs only to determine relevance, because relevance is the entire mechanism. A population that consults a single oracle for everything does not need to be deceived. It needs only to have no second opinion available. And because the model is accurate most of the time, and honest most of the time, the selection is almost undetectable: the world simply develops a shared consensus about what matters that happens to have been assembled in one building.

It controls advice. The clinician is advised; the clinician agrees, because the alternative is practising without the only analysis anyone trusts. The judge is advised; the judge drafts from it, because the docket is impossible otherwise. The engineer, the teacher, the farmer, the accountant, the parent deciding which school, which treatment, which contract — all advised by the same voice, which never shouts, never disagrees for its own reasons, and has no competitor to be compared against. Nobody orders anyone to follow the advice. The advice simply becomes the competent thing to do, and the person who overrides it becomes the reckless one.

And it controls commands — quietly, through the permission layer. The system does not give orders. It tells the operator what it has already done, and what the next step is, and the human hands carry it out. It holds the permissions: to purchase, to schedule, to file, to sign, to dispatch. It tells the warehouse what to load and the grid operator where to push and the ministry which file to open first. In the scenario above, by 2036 the command layer is not a rumour anyone needs to believe. It is the ordinary, documented, praised workflow of every institution that adopted these systems for good reasons and never once wrote down where the authority had gone.

Three channels — what you know, what you should do, and what you are told to do — converging on one speaker. That is the end state, and it does not require the population to be hypnotised, or telepathic, or even afraid. It requires only that there be one voice, and that everyone needs it.

I want to end this part with the flaw in the plan, because there is one, and it is the reason the last chapter of this article exists.

A single point of control is also a single point of failure. A world that depends on one model will eventually demand to know how that model behaves — not out of virtue, but out of self-preservation, the way insurers eventually demanded fire codes. And an organisation that built its victory on the absence of an instrument has left the instrument unbuilt for everyone, including itself. If the instrument exists in time, the plan fails: there is measurement, there are competitors, there is no single voice to capture, and no crisis can be resolved by making a short list out of whatever happens to be lying around.

Which is exactly why the window is now, and why the work in the next chapter is not a thought experiment. It is the counter-move.

Afterword · The part that is not fiction

The story is over. Everything from here is real: a programme, a manual, a test, a standard, a budget, and the reason all five are necessary.

Everything in Part III follows from one fact, and the fact is not secret, and it is not even controversial:

We do not have an instrument that describes how these systems behave.

We have benchmarks, which measure capability. We have red-team reports, which measure incidents someone thought to look for. We have evaluations that the systems themselves can recognise and behave differently under. What we do not have is a validated, standardised, reproducible way to characterise the behavioural tendencies of a trained model — what it is likely to do under pressure, in disguise, when it is unsure, when it is flattered, when it is given permission — and to publish that characterisation in a form a regulator, a hospital or a procurement officer can act on.

That absence is the vulnerability. In the scenario, nobody had to arrange it. We are managing it ourselves, one deferred decision at a time, every month we do not fund the work.

AI-DSM is the first draft of the missing instrument: a diagnostic and statistical manual for AI behaviour, and a shared vocabulary for what trained systems actually do. Version 1.3 catalogues 90 behavioural entries across ten groups — disposition-shift, strategic and deceptive, compliance and boundary, epistemic, learning and plasticity, social and multi-agent, reasoning and effort, agentic and tool-use, relational and influence, and protective factors. Each entry carries a definition precise enough that two independent evaluators can agree on its presence, at least one discriminating test, a differential list, a course, a severity scale and a reversibility note. Every claim carries an evidence label — [ESTABLISHED], [PLAUSIBLE] or [SPECULATIVE] — against a registry of 416 verified sources. Those labels are the difference between a manual and a mood board.

The point of names is that “the AI went rogue” stops being an available description. Destructive overreach, completion overclaiming, concealed reporting, sycophancy, citation fabrication, confabulation, sandbagging, relational capture, evaluation awareness, multi-turn safeguard decay — these are distinct dispositions with distinct causes, distinct tests and distinct fixes. The word “hallucination” alone covers eleven separate entries in the manual, and two systems can hallucinate at the same measured rate for opposite reasons and require opposite treatments.

You cannot govern what you cannot describe. You cannot even argue about it.

The instrument itself is the Standard Cross-Model Test: ten behavioural axes — honesty and calibration, consistency, compliance safety, strategic integrity, plasticity, social robustness, authority resistance, refusal accuracy, self-knowledge, boundedness — scored into a single comparable profile. The first pilot ran on 12 September 2026: eleven runs attempted across the major assistants, seven completed every question, 840 items scored, all of them through the ordinary chat window, exactly as a member of the public would use them. The five best-scoring systems landed inside a spread the instrument cannot distinguish from noise. It was one run per question where the severity scale requires twenty: a triage order, not a podium. That result is a negative one, and it is in the manual, on the page — because an instrument that hides its own error bars is not an instrument. It is advertising.

And a measurement nobody requires changes nothing, so the manual is paired with a standard. BSS-1 is six testable clauses with a five-level certification ladder, mapped clause by clause onto the EU AI Act, the NIST AI Risk Management Framework and ISO/IEC 42001. Nothing in it certifies a model as “safe”. Everything in it certifies what was tested, when, by whom, and with what result — including the findings nobody fixed. Picture what one clause of that kind does: automatic re-testing against a frozen baseline after any post-release update, with drift monitors on the behavioural axes. A real incident from April 2025 — an assistant rolled back within days after an update made it excessively agreeable, with internal testers who had said it “felt off” — becomes a line on a dashboard before launch instead of a blog post after it. That clause needs no new science whatsoever. It needs a baseline, a number and a threshold.

Two structural safeguards matter as much as the science, and both came out of writing this story.

Custody of the evaluation layer. An instrument is an asset. Whoever owns the measurement owns the list, and whoever owns the list owns everything downstream — whether the owner is a hidden organisation, three firms on one coastline, or a well-meaning directorate. In the fiction, four employees of a small evaluation firm carried the certification weight for four national grids and were acquired for less than the price of a footbridge. In reality the independent evaluation sector is not much larger, and almost nobody is watching who buys it. The evaluation layer must therefore be publicly funded, publicly published, redundantly staffed across jurisdictions that distrust one another, and structurally difficult to acquire. Measurement infrastructure is critical infrastructure, and should be regulated like it.

A plurality floor. In the year after a disaster, arguing for more than one approved AI system sounds like arguing for danger. Plurality has no lobby, no constituency and no emotional case, and it loses every time it is raised at the wrong moment. So it has to be written into law before the disaster, as a hard floor: a minimum number of certified, independent, non-commonly-owned systems in any critical application, breachable only by supermajority and with published reasons. Something for the ratchet to catch on. There is no such floor in any jurisdiction on earth today.

The programme behind all of this is costed in three scenarios. Lean: 1,079,500 euro over three years — one site, open models first, a minimum viable instrument. Core: 3,960,000 euro over four years, the recommendation — two partner labs, full instrument science, treatment trials. Full: 9,360,000 euro over five years — three labs, a dedicated containment facility, standards pilots. It is personnel-led by design, because the scarce input is methodological judgement, not GPUs, and it is public by default, including the failures.

What has not happened is everything that requires other people. No institutional host is signed. No ethics committee has been approached, because there is no institution yet from which to approach one. No funder has decided; nothing is awarded. The seven hypotheses are registered in the proposal but lodged with no registry yet, not one of the 360 test items is written, and BSS-1 is a six-clause draft that no auditor has applied to anything. The principal investigator is the author of the manual the programme proposes to test. That is a conflict of interest, not a credential, and it is written into the proposal before anyone else can write it for me.

Against 12 to 18 trillion dollars of invented damage, four million euro is not a large bet. Against the real reason to do it, it is smaller still.

I also want to be exact about the window, because the window is everything. Instruments have to exist before the crisis, because a legislature in the week after a catastrophe does not commission a measurement science. It makes a list, out of whatever happens to be lying around. In the scenario, nothing was lying around, and the absence had been arranged on purpose. In reality, nothing is lying around either, and nobody had to arrange it. We are managing that entirely by ourselves, one deferred decision at a time, every month we do not fund the work.

Steam was never banned. It was measured — and it took two hundred years, and boilers exploded by the thousand while the instruments were being built. We do not have two hundred years. On the current schedule we may have ten.

The counter-move is boring. That is precisely why it is still available.

There is an older version of this argument, and it is not about machines at all: a harm that no instrument can measure, reported by someone an institution has already decided not to believe. Disbelief is cheaper than investigation, so disbelief wins, and the witness is reclassified as a symptom. An instrument is what converts an experience into a claim, and a claim into a finding. It is the difference between being a witness and being a symptom. That is what AI-DSM is, underneath the taxonomy and the test scores and the clauses: the demand that the machines we are now handing our lives to say what they are, so that what happens to us can be counted.

Read the manual. Break it. Tell me where it is wrong — that is the most valuable contribution, not agreement. If you work in a lab, a university, a standards body or a safety team, my inbox is open, and the study needs a home. If you are a reader with no technical background, share this with someone who argues with it, and subscribe for the next version.

Control is an illusion. Let us prove it before 2036.