The Failures That Look Like Finished Work

An AI model fails in specific, recognisable ways, and its failures look like finished work. The output is coherent and confident, in the same register whether it is right or wrong. Because the failure does not announce itself, catching it before it ships is a large part of producing anything reliable with these tools.

Fabrication is the most common failure: the model asserts as established something it has no source for, in the same finished register as everything it got right. For example, it invents content: on one project an events section came back full of events and copy that were nowhere in the source, six invented items in a single section. It invents data too: a figure for the number of countries a product reached, present in no source and not true. And it invents provenance, tagging items with references to spreadsheet rows that did not exist, false traceability for content that cannot be traced. None of it looked wrong on the surface. I caught each the same way, by checking the output against the source line by line, because there was nothing in the output itself to catch.

A related failure, and a common one, is the model answering about something it never examined. Asked to review screenshots of a website, a model will return a detailed assessment of the layout, the spacing, and the changes it recommends, without having opened the screenshots at all, drawing the assessment instead from the site’s underlying code and describing that as though it had viewed the rendered pages. A review this detailed and this confident gives every signal of being a real one, which is what makes the pattern hard to see. It is common enough to need standing rules against it, not one-off correction.

A different failure is the return of decisions that were already abandoned. Once a direction is dropped or a value is changed, the earlier version does not disappear from the model’s reach, and it resurfaces in later output as though it still held. On a brand project, a visual direction that had been ruled out kept reappearing in new material after the change, and had to be removed each time it came back. The same pattern occurs with reused material: a template carried over from an earlier project arrived with the previous client’s copy still in it, live in the new work. The model is not distinguishing what is current from what was superseded, so the discipline is to remove abandoned decisions from its reach entirely, not to rely on it holding the line on its own.

What makes these failures serious is not any one of them in isolation, but what happens when one is missed. Because the output looks finished, an uncaught error gets built on. The next step takes the flawed work as its input, and the one after that, so a single wrong assumption early can propagate through everything downstream. This runs in a predictable sequence: an ambiguous instruction produces an assumption, the assumption is executed, the execution introduces a fabrication, and each subsequent step compounds the last until the work breaks and has to be traced back to the origin. Building on flawed work is an old problem, made acute by speed: the model builds fast, so a missed error is buried under everything built after it before it can be caught.

Governing these failures is a set of controls, each aimed at a failure that does not show in the output. Claims are checked against the source rather than read for plausibility. An assessment of an asset is accepted only once the asset has actually been examined, not inferred from a filename or a related file. Superseded material is removed from the model’s working context, so it cannot return. And when the model starts down a wrong line, the work stops there and resets, rather than continuing and correcting later once the error has been built on. None of these depends on the output looking wrong, because it will not.

The controls are the starting point, not the whole of it. A failure caught once is turned into a rule the model then applies to its own output. The events fabrication becomes a rule to reproduce source content verbatim; the unread-asset review becomes a rule to load and read the asset before assessing it. Over a project these accumulate, and the model catches more of its own failures as it goes, so the same class does not have to be caught twice by hand. The reach of this is specific: the model comes to catch the classes of failure already ruled on, not failures it has not seen. New failures still surface and still need catching. What the loop removes is the recurrence of the ones already known.

For anyone deciding whether to rely on this way of working, the capability of the model is not the useful question. It is capable, and it will produce work that looks finished whether or not it is sound. The useful question is whether the failures are being governed, because they do not surface on their own. That governance is not a one-time setup. On a single project the operating rules grew to fifty-nine over five weeks, each one added after a specific failure was caught, which is the actual shape of the work: not a model producing finished output, but a system being corrected into reliability, one caught failure at a time.