The Question That Caught 194 of 196
How was I supposed to catch a mistake that did not look like one, when every check I had written was calmly telling me the work was fine? I had built a system to learn from official documentation, and I had given it a sensible precaution: do not learn from obsolete things. The pipeline ran. The tests passed. Nothing crashed. Nothing raised its hand. The only problem was that the system was quietly removing things it should have kept, which is a difficult defect to notice because absence has excellent manners.
The answer, when I finally found it, turned out to be cheaper than the machinery around it, and older than the question itself, written in my own words. I did not arrive by force. I arrived because I looked at a number and gave it the smallest possible amount of respect.
The checks passed, and that was the problem
I had trusted the green marks in the ordinary way, which is to say I had treated them as evidence that the thing was right, rather than evidence that the thing had performed the particular dance I had taught it to perform. That distinction sounds obvious after the damage has been found. It was less obvious while the system was doing what I asked, producing output, and allowing me to continue my day under the pleasant fiction that completed work and correct work are relatives.
The tests were not useless. They were obedient. They checked that the exclusion step ran, and that the pipeline completed after it ran, which meant they could certify a bad decision with excellent posture. A shorter list looked exactly like a cleaned list. A missing thing did not crash anything by being missing. It simply stopped being available to learn from, which is a tidy form of losing evidence.
This is one of the hazards of building systems that tidy up after themselves. When they succeed, they remove the trail. When they fail, they may also remove the trail, but with more confidence.
The cost was a kind of clean ignorance: I could repeat the run, compare the counts, and still not know what had vanished
I asked the smaller question
The useful part did not begin as an audit. I was not being thorough, noble, or even especially suspicious. I saw a count that looked too neat, too concentrated, too willing to behave, and I asked what had actually been removed. That is all. No grand method announced itself. No serious investigation stood up and cleared its throat. I had just learned, slowly and at some cost, that a number can be less a measurement than a door handle.
What made the question worth keeping was how little it required from me. I did not need a new framework for doubt, which was fortunate because I would probably have named it badly and then spent the afternoon maintaining the name. I needed to pause over the cheap question instead of walking past it. What are the missing things? Who decided they were missing? What was the decision actually based on?
I have learned to respect the questions that look too small to file properly.
The list was too tidy
Once I looked, the shape was wrong before the details were. The supposedly obsolete things were not scattered around the way real age scatters through a working system. They were clustered. They lived in one neighbourhood, as if obsolescence had developed a local council and kept minutes. That mattered because reality is usually messier than a pattern match, and when a failure is too tidy, I have learned to stop admiring the tidiness and ask what made it.
The rule I had wanted was plain enough: keep the system from learning dead material. The rule I had actually built was cruder. It looked for a warning word anywhere on a documentation page, then treated the thing on that page as if it had declared itself obsolete. That sounds almost reasonable if I say it quickly, which is where many of my worst decisions prefer to live.
A page can mention a retired setting while describing a current, ordinary, heavily used thing. A document can contain a warning without being the warning. The difference between mentions the word and is the thing was not a philosophical gap. It was where the system had been throwing away useful material while my tests sat there with clean hands.
The mistake was cheaper than judgement in the moment, because looking for a word was easier than asking what the page was actually saying. I saved myself the work of reading context, then paid for it by making the system forget material it was meant to learn.
The old rule had no teeth
The part that made the mistake mine was not the crude check by itself. I had already written the principle that should have stopped it: never drop data silently. I had the rule. I had the sentence. I had the satisfaction of having noticed the right danger, which is a surprisingly efficient substitute for preventing it. What I did not have was a check that forced the system to obey the sentence when the sentence became inconvenient.
That is how a rule becomes decoration. It sits in the project like a smoke alarm with no battery, adding moral architecture to the room while the actual machinery gets on with its private hobbies. I had built a fence around the behaviour in my head, then failed to put the fence where the behaviour lived. The result was not dramatic. It was worse than dramatic. It was administratively wrong.
In June 2026, the fix was not to become more suspicious of every result in some theatrical way. It was to make the rule enforceable at the source. A thing only counted as obsolete when it declared itself obsolete. Ambiguous mentions kept the item. If the system was unsure, it preserved evidence instead of deleting it and asking me to be grateful for the tidier floor.
A wrongly kept item can still be seen. A wrongly dropped one becomes a silence with consequences.
The question caught the number
The question in my log was: "What are these dropped (deprecated) nodes and how were they decided?" When I followed it, 196 were on the list and only 2 belonged there. The other 194 were wrongly flagged.
That was the point where the green checks stopped meaning what I had been letting them mean. They had not lied. They had answered a smaller question than the one I needed answered. They could tell me that exclusion happened, and that the pipeline survived it, but they could not tell me whether exclusion had become a quiet little machine for deleting the present because a page mentioned the past.
The repair did two things at once. It narrowed the judgement so the system only acted on a self-declaration, and it changed the default when the evidence was unclear. Keep first. Remove only when the claim is specific enough to deserve removal. I also added a guard for recurrence, because I had already proved I was capable of writing the right principle and then leaving it to supervise nothing.
I kept the questions
I would prefer the story in which I found the problem through disciplined engineering pressure. It would be cleaner. It would make me sound like the sort of person who simply aims rigor at a system until the system confesses. What actually happened is less flattering and more useful: I was curious on a day when I was not formally looking, and the idle question found a defect that the formal checks had approved.
So I changed the habit. I stopped treating doubt as a mood I had to either indulge or overcome, and started treating it as an instrument I could keep on the bench. Not every question deserves a hunt. Some are noise, some are procrastination wearing a clean shirt, and some are just my brain trying to avoid the next hard thing. But the cheap questions that keep finding expensive mistakes have earned a place in the work.
That is how I catch a mistake that does not look like one: I keep the question long enough for it to become testable, and I make the system answer the question I actually meant to ask.
I carried forward the old backend habit after all: make the invisible decision visible enough that it can be argued with. What changed was not the amount of doubt, but its route. There was less of it circling in my head, and more of it pressed against the machinery, where Houdini could expose whether the decision actually held.
Next time: the journal that ended, and what happened when I moved my working life into the tool that was watching me.
