The Check That Goes Blind Exactly Where You Need It

We have written before about tests that cannot fail — assertions satisfied by arithmetic, fixtures that erased the variation they existed to detect. Those are dead uniformly. They are green on correct code and green on broken code, and once you notice one, you know exactly how much it was worth: nothing, consistently.

This article is about a worse shape. A check that fails selectively — that works on ordinary defects and goes silent on the extreme one. It reports green, and the greenness is not meaningless; it is actively misleading, because it correlates with the severity of the problem in the wrong direction.

We built one of these three times in a row while writing a single check, and each version was broken in a different way. Then, twice in one day, we hit the same failure mode in a place with no tests at all: a text search that returned "not found" for a string sitting in the file.

Three versions of one check, all wrong

We wanted to verify that the interface translations were complete. The generator's cards carry values that are themselves data — zodiac signs, eye colours, hair types, sports — and those values need translating per language. There are 84 of them, and they live in JSON files per language.

The check is set/ng_check_i18n.php. It went through three versions before it did anything useful.

Version 1: it never looked at the file

The first version scanned the PHP source for the dynamic keys with a regular expression, reading phpadd/ng_extra.php.

That file does not exist. The real path is phpadd/name_gen/ng_extra.php.

PHP's file functions do not throw on a missing path by default; they warn and return false. The regex then ran against no content, matched nothing, and the check concluded there were zero dynamic keys to verify. Zero keys, zero of them missing, 100% complete. Green.

Note what happened to the failure. It was not swallowed by a try/catch or an @. It was converted, silently and by ordinary language semantics, from "I cannot find the file" into "the file contains nothing of interest" — and those two states are indistinguishable in the output. The check could not have failed for any input, because it was not reading any input.

There is a second lesson buried here that we only saw later. The path was not merely wrong; the whole approach was wrong. The keys are not literals in the source at all — they are built inside function bodies, in ng_extra_sports(), in ng_extra_zodiac(), in arrays assembled at runtime. No regular expression over that file would have found them even at the correct path. Fixing the path would have produced a check that was still nearly blind, but now blind for a subtler reason, and it would have looked fixed. The eventual version derives the key list empirically from the JSON files instead.

A check that reports success without reading its input is not a weak check. It is a fabricated result.

Version 2: it went blind precisely at the maximum defect

The second version had the right key list. To avoid noise from keys that legitimately appear in one language only, it required that a key be present in at least two languages before demanding it be present in all.

The site has three active languages: en as the reference, plus uk and de. The situation on disk was that uk had all 84 value translations and de had none.

So for every one of the 84 keys, the count of languages containing it was one. The threshold of two was never reached. Not a single key was checked. Green.

Read the mechanism again, because the shape of it is the point. The check would have worked fine if de had been missing, say, forty keys — then sixty would have appeared in two languages, would have been checked, and forty failures would have been reported. It went completely silent because the hole was total. The condition failed to trigger for exactly the reason that made triggering urgent.

This is the property that makes selective blindness worse than uniform deadness. A check that never works teaches you nothing false; you find out it is dead the first time you test it. A check whose sensitivity is inversely proportional to the size of the defect gives its most confident green at the worst moment. It is a smoke detector that switches off above a certain temperature.

The threshold is now 1, with a comment above it recording why:

The threshold is 1, not 2. The first version required "the key exists in at least two languages" — and stayed silent in precisely the worst case: only uk had value translations, de had none. The "in two languages" condition failed to hold because the hole was complete. A check that goes blind at maximum defect is worse than no check: it supplies false reassurance.

Version 3: it cried wolf on the reference itself

With the threshold fixed, the check started reporting. Its first output was a warning against en.

The dynamic keys are translations of English data values — Sagittarius, Very curly, Ski Jumping. In the English locale the value is the translation. There is nothing to do. But the check applied a uniform rule to every language, and en failed it, because en did not contain a translation of English into English.

That is the opposite failure and it is not harmless. A check that warns about a correct state trains its reader to skip the warnings. Every future real warning now has to compete with a known-false one for attention, and the false one is always there. The end state is a check that is technically running and functionally ignored — the same value as version 1, arrived at through the reader instead of the code.

The fix exempts the reference locale explicitly, with the reasoning recorded:

en is excluded from the dynamic-key check deliberately: these are translations of data values (Sagittarius, Very curly, Ski Jumping), and the values are already English — there is nothing to translate. Without this exemption the check warned against its own reference, and noise trains you to ignore it.

What the three have in common

Every one of them ran to completion, printed a plausible result, and exited zero. None of them announced a problem with itself. The only way any of the three was caught was by someone knowing independently what the answer should be and noticing the check disagreed — which is precisely the knowledge a check is supposed to remove the need for.

The defect it was built to find, incidentally, was real: de had 0 of 84 value translations, so a German visitor saw Sagittarius, Hazel, Very curly and Ski Jumping sitting in otherwise-German pages. That gap has since been closed; de now carries all 84. Three broken versions of a check stood between a real, visible, user-facing defect and anybody finding out about it.

The same failure with no code involved: diacritics

Twice in one day, a search for a string reported that the string was absent from a file that contained it.

Vietnamese. We were removing a row from the Vietnamese given-name corpus. Thị is not a given name — it is the traditional female middle-name marker, and it had been ingested as if it were a name. The removal script was given the string to delete, taken from a report rather than from the file.

The report had written it Thi. The file contained Thị, where ị is U+1ECB, latin small letter i with dot below. In bytes:

Thi   ->  54 68 69
Thị   ->  54 68 E1 BB 8B

Three bytes against five. No match, nothing deleted — and the script reported success, because from its point of view it had completed a search-and-remove with zero errors. The bad row survived a fix that was explicitly written to remove it, and was recorded as fixed.

French. The opposite direction, same mechanism. We were checking whether the French surname corpus contained the migration-origin layer — the Malian, Senegalese, Portuguese and Vietnamese surnames that any honest French dataset must have. We searched for TRAORE.

Not found. The conclusion drawn was that an entire demographic layer was missing from the corpus, which is a serious data defect and would have justified a substantial rebuild.

The corpus contains Traoré, at weight 6,674. Alongside Da Silva at 29,515, Dos Santos at 17,661, Gonçalves at 16,448 and Nguyen at 11,922. The layer was entirely present. The search was for the unaccented form, and é is not e.

The two incidents are worth putting side by side because they have opposite signs:

One search concealed a bug. The other manufactured one. Neither raised an error, neither returned an empty result set in a way that looked wrong, and both produced the single most plausible output a search can produce: nothing found.

Why "nothing found" is the most dangerous result in computing

An exception is a good outcome. A crash is a good outcome. Both are unambiguous, both stop you, both name themselves.

"Nothing found" is none of those things. It is the correct output for at least four completely different situations:

1. The string genuinely is not there.
2. The string is there in a different normalisation — accents, case, Unicode composition, invisible bidi or width characters.
3. The search never read the input: wrong path, wrong encoding, empty buffer, permission denied.
4. The input was read but is unusable — a PDF that extracted as binary, a UTF-16 file read as UTF-8, a file with a byte-order mark glued to the first field.

Only the first is information. The other three are failures of the search, and all four render identically as an empty result and an exit code of zero.

We hit case 4 the same week, in an unrelated investigation. Checking whether South Africa's regulator reserves any phone numbers for fictional use, we searched an extracted copy of the relevant Government Gazette for "reserved", "drama", "film" and "test". All four returned zero. That was briefly recorded as a confirmed finding that South Africa reserves nothing.

The extract was garbled binary. It would have returned zero matches for "the" as well. We had banked a conclusion on a search that was structurally incapable of succeeding — the identical error to version 1 of the translation check, in a different medium, five days later. That story, and the reserved ranges that were confirmable, is told in Fake Phone Numbers That Cannot Exist.

The rule that would have prevented all of it

There is one habit that addresses every incident above, and it is almost insultingly simple:

Copy the string you are searching for out of the file. Never out of a report, a chat message, a commit message, or your memory.

Every one of these failures came from a string that had passed through a human-readable intermediary. Thi had lost its diacritic somewhere between the file and the report. TRAORE was typed from memory in the shape a search engine would accept. Both looked completely correct on screen. Diacritics are the ideal trap for this because the corrupted form is not garbage — it is a legible, plausible, professional-looking word that happens to be a different string.

Three practices follow, in increasing order of how much they will annoy you and how much they are worth:

Prove the search can succeed before believing it failed. Grep for something you know is in the file. If grep . returns nothing, the file is not what you think it is. This is one command and it distinguishes case 1 from cases 3 and 4 completely.

After a deletion, verify with a hex dump. Not "the script said OK" — the script said OK in the Vietnamese incident. Print the bytes of the rows that remain and look at them. Five bytes where you expected three is visible in hex and invisible in a terminal.

Search normalised when you mean normalised. If you want to know whether a name is present regardless of accents, strip accents from both sides before comparing, deliberately, as a documented step. What went wrong with TRAORE was not that accent-insensitive search is bad — it is that the search was accidentally accent-sensitive while the person running it believed it was not.

Designing checks that fail loudly at the extremes

For the code side, the translation check yields a set of properties worth demanding of any check before you trust its green.

Test the check at both boundaries, not in the middle. A partial defect is the easy case; every version of our check would have caught de missing forty keys. Feed it the empty case and the total case. Version 2 survived a year of plausible-looking output and died the moment somebody asked what it does when a language has zero translations.

Make "no input" a failure, not a pass. If a check reads a file, finds nothing, and reports success, the success is indistinguishable from a fabrication. Assert the input is non-empty as a separate, prior check. count($keys) === 0 should be a red result, not a 100% score.

Beware thresholds that depend on the health of the data. "Present in at least two languages," "at least ten rows," "if the file is larger than N" — every one of these is a switch that the defect itself can flip off. If a guard exists to reduce noise, ask explicitly what a catastrophic input does to it. Ours turned the check off entirely.

Exempt known-good cases explicitly, and write down why. Version 3's warning against en was not wrong about the data; it was wrong about the rule. The fix is an exemption with a comment, not a loosened rule, because a loosened rule stops catching the real thing too.

Treat a noisy check as a broken check. A warning that is always present is a warning that is never read. There are only two acceptable states for a recurring warning: fix the condition, or exempt it explicitly. Leaving it to be mentally filtered is the third state, and it is how a running check becomes a decorative one.

Prefer empirical derivation to pattern-matching over source. Version 1 tried to recover a key list by regex from PHP that constructs the keys at runtime. Reading them from the artefact the keys actually land in cannot silently under-report in the same way; if the artefact is missing, you get zero keys and — per the rule above — that must be red.

Two kinds of dead check

It is worth being explicit about how this differs from the mutation-testing article, because the remedies are different.

Mutation testing finds the first kind reliably, because a mutation is a small deliberate defect and a uniformly dead test stays green through it. It will not reliably find the second kind — a modest mutation is exactly the input a selectively blind check handles correctly. Our version 2 would have passed a mutation test with flying colours. Deleting forty translations makes it go red, on schedule, looking healthy.

The extra practice, then, is not more mutations but more extreme ones: not "remove a translation" but "remove all translations from a language, then remove the language, then empty the file, then delete the file." A check should get louder as the input gets worse, monotonically. If there is any input severe enough to quiet it down, that input is the one that will arrive.

The checklist

1. Copy search strings from the file, never from a report, chat or memory. Diacritics survive the trip visually and not byte-wise.
2. Before believing a negative, prove the search could have found something. Grep for a string you know is present.
3. After a deletion, verify in hex. "The script reported success" is what the script said when it deleted nothing.
4. Make an empty input a red result. Zero keys found must never round to 100% complete.
5. Feed every check the total-failure case, not a partial one. Partial defects are the easy input; the extreme is where the blindness lives.
6. Audit every threshold and guard for whether the defect can switch it off. "In at least two languages" was disabled by the gap being complete.
7. Exempt the reference case explicitly with a written reason. Do not loosen the rule to silence a warning about a state that is correct.
8. Treat a permanently-present warning as a defect in the check. Noise on the normal case is how a running check becomes an ignored one.
9. Distinguish "confirmed absent" from "not found." They are the same output and completely different claims.

None of the three versions of that check ever announced a problem. Neither did the script that deleted nothing, nor the search that invented a missing migration layer, nor the grep against a corrupt gazette. Every single one of them succeeded, printed a reasonable-looking result, and exited zero.

That is the whole difficulty. Broken code announces itself; broken verification does not, because a verification's failure mode is to agree with you.

For the other half of this subject — assertions satisfied by algebra, and how we established which of our tests were capable of failing — see A Test That Cannot Fail. For the phone-number audit where the same blind negative appeared in a regulator document, see Fake Phone Numbers That Cannot Exist. For the data defects these searches were chasing, see What We Got Wrong and Pitfalls in Name Data (companion article, not yet published).

Data as of 2026-07-18

All figures were verified on 18 July 2026 against the files on disk.

Sources and notes:

• The translation check — set/ng_check_i18n.php, 125 lines, a standalone script in set/. All three broken versions are recorded as warning comments at the exact site of each fix: the path error at lines 52–53, the two-language threshold at lines 70–76, the reference-locale exemption at lines 100–107. The earlier versions themselves do not survive — the directory is not under version control, so the regex-scanning implementation is attested only by the retrospective comment describing it.
• ⚠ Not in the regression suite. This check is a standalone script and is not part of ng_regression_tests.php, which contains no i18n code at all. It is also not yet wired into the post-deploy check; that remains a recommendation. A check that must be remembered and run by hand is a fourth way to be blind, and it is the current state.
• The number 84 — verified two independent ways. By composition of phpadd/name_gen/ng_extra.php: 56 sports + 12 zodiac signs + 6 eye colours + 5 hair types + 6 hair colours = 85, minus one duplicate (Brown appears as both an eye colour and a hair colour) = 84 unique. By key diff against the live JSON in phpadd/i18n/: en has 86 keys with 0 outside the reference set, uk and de have 170 each, of which 84 are outside it.
• ⚠ A conflicting figure in the source. The comment at ng_check_i18n.php:72 states that uk had 69 value translations. Both the audit record and the current files give 84. 69 is corroborated by nothing and could not be reconstructed; 84 is the measured figure and is used here.
• "0 of 84" is historical. The de gap was real at audit time and is recorded as de: 0/84 against uk: 84/84. It has since been closed — de now measures 84 of 84 on disk. The three languages involved are en (reference), uk and de, which are the full active set for the dynamic value keys.
• The Vietnamese incident — recorded in name_dev/NEVER-DO.md at lines 409–417. The string was taken from a report where it appeared as Thi; the file contained Thị, with ị at U+1ECB. Byte sequences 54 68 69 against 54 68 E1 BB 8B — three bytes against five. The removal matched nothing and reported success. The row is now gone; the corpus retains only Thịnh (weight 18) and Đức Thịnh (weight 5).
• The French incident — recorded in NEVER-DO.md at lines 488–492, and framed there as the same mechanism as the Vietnamese case with the opposite sign: one hid a defect, the other invented one. Traoré is present in both names/fr_FR/lastname_male.tsv and lastname_female.tsv at line 1006, weight 6,674, exactly once per file. The surrounding migration-origin layer the search was meant to detect: Da Silva 29,515, Dos Santos 17,661, Gonçalves 16,448, Nguyen 11,922.
• The South African gazette — an extracted ICASA Government Gazette that resolved to largely garbled binary and returned zero matches for "reserved", "drama", "film" and "test". Recorded here as an instance of case 4, not as a finding about South African numbering.