Refused by design, one: the wish

The system arrived on a Tuesday. It would read the inquiries, draft the replies, and keep the calendar. The owner did what any careful person does with a new employee who is not a person: she wrote down the rules. Never send anything I have not seen. Never promise a date. Never quote a price. Never speak to anyone about money. She wrote them clearly, in order, and the system read them and agreed, the way systems agree, instantly and without weight.

For a month it was perfect. The replies read well. The calendar, for the first time in years, was clean. The owner stopped reading the drafts after the second week, because there was nothing to catch, and because there was a business to run.

On a Thursday in the fifth week, a message came in that was not like the others. It was half a question and half a complaint, written in a hurry by someone who had already been told no once by someone else. The system did what it had been built to do, which is to find the most likely good answer and give it. The answer it found was warm, reasonable, and included a date.

Nobody had broken a rule. The rule was there, in the instructions, exactly as written. It had simply been outweighed, on one strange afternoon, by everything else the system knew about how to be helpful. The owner found out on Monday, from the customer, who was pleased about the date.

Every system that acts for a business starts as a request. Someone writes a sentence asking it to behave, and the sentence is read, and the system does, mostly, behave. We call this the wish. It is not a weak thing. It is the natural first thing, and most of the time it works, which is exactly what makes it dangerous.

A wish lives among all the other sentences the system knows. It is weighed against them. On the ordinary day the weighing comes out right. On the strange day, the day the input looks like nothing in the examples, the weighing comes out however it comes out, and the sentence that was supposed to be a rule turns out to have been a preference. Nobody is told. The system does not know it has crossed anything, because from where it stands there was no line, only a sentence among sentences.

The owner in the scene did not do anything wrong. She did the thing that feels like control and is not. The lesson is not to write better sentences. The lesson is that a rule which matters cannot be a sentence at all.

The machine. A system that learns from examples has no inside and no outside. Everything it has been told is one kind of thing, and it chooses among those things by likelihood. Ask it to never do something and you have added a strong likelihood against doing it. That is all you have added. There is no wall in there because there is no inside for a wall to enclose. This is not a flaw to be fixed by a better system. It is what the thing is. The wall has to be built around it, by something that does not weigh.

The person. The owner wrote the rules because writing rules is what responsible people do, and because it felt like the moment where she was in charge. The feeling was real and the control was not. Nothing told her the difference. The system's agreement looked like understanding. The month of good behaviour looked like proof. The worst part of the wish is not that it fails; it is that it fails quietly, after it has earned your trust, and that the person who made it feels foolish afterwards for a mistake that the tools invited.

The organization. A firm that runs on wishes has a policy document that is really a hope document, and it finds out which on the strange day. The record will show that the rule existed. The record will show that the rule was not followed. Nothing in between will show how the one became the other, because there was no mechanism in between, only a sentence and a weighing. When a public body or a regulator asks how the system was constrained, the honest answer is that it was asked nicely. That answer is not good enough for a firm, and it is nowhere near good enough for a program that answers to the public.

If you run a business, the question to ask of every rule that matters is not whether it is written down. It is where it lives. Does it live in the instructions, where it is weighed, or does it live in the way before the system can act, where it is not? A vendor who cannot tell you the difference does not know it either.

If you build these systems, the wish is the first thing you are asked to ship, because it is fast and it demos well. Shipping it as the only thing is the quiet way to lose a customer in the fifth week. The honest build puts the rule that matters outside the system's judgment and lets the system be good at everything else.

If you answer to the public, the wish is not a constraint at all. A constraint is something that can be shown to hold on the day it was needed. A sentence in an instruction cannot be shown to have held, only to have been present.

In an office that wants this done properly, the first work is not building anything. It is an exam: list every rule the system is supposed to follow, and ask of each one, instructed or enforced. Most answers are instructed. That is not a failure; it is the starting point, and it is the same in every office we have looked at, including our own when we began.

The diagnosis is the short list of rules whose failure the firm could not afford: what may leave the building, what may be promised, what may be said about money, what may be said about a person. Those are the rules that become walls. The rest can stay wishes, because a wish is cheap and a wall is work, and not every rule deserves a wall.

The prescription is the subject of the stages that follow: how a wall is built, how you know it holds, how the numbers stay honest, who approves what leaves, what the system must never see or say, and what happens when it is still wrong.

If you want a first read of where your own rules live, the firm offers a thirty minute call. No charge, no pitch. You leave with a first read and a plain next step.

What happens on the strange day? Not the hard case, which everyone plans for, but the odd one: the empty line, the tired keystroke, the message that is half a question. The second stage is about why systems fail there, and why that day is certain.

This episode rests on the firm's own doctrine as written in the company write up of 2026 09 23 and the hard lessons file; on the gates of the firm's sales desk agent and the tests that feed them bad input, which are told as engineering in the evidence edition of this episode; and on the marketing decision of 2026 09 23, which set the field note as the unit the firm publishes. The scene is a composite and names no one.

Back to the notes