Project Spark ships with roughly 3,800 PHP tests, 950 JavaScript tests, a 14-check automated audit, and static analysis, north of 4,700 automated checks on every release. The surprise isn’t the number; it’s what the tests did: writing coverage for previously untested code surfaced more than two dozen real production bugs that had been silently misbehaving, some for years.
Key takeaway: Test coverage isn’t only insurance against future regressions, it’s a discovery tool. Every sustained campaign of writing tests for “working” code in this project uncovered live bugs that no user had reported, because the failures were silent.
How a test suite gets to 4,700
Nobody plans a suite that size on day one, and nobody should. Project Spark’s suite grew the way sediment does, in layers, each with a reason:
- Feature tests arrived with features, the ordinary way.
- Regression pins arrived with bugs, every production incident left a permanent test behind.
- Coverage campaigns arrived in deliberate sweeps during the project’s mature phase: pick a class or module with no direct tests, write coverage for its documented behaviour, repeat. These sweeps were run as sustained sprints across hundreds of maintenance releases, often five to ten tests at a time, and they are where most of the bug discoveries in this post came from.
- Structural pins arrived when we noticed the same kind of failure recurring: two lists drifting apart, a forbidden pattern creeping back. Rather than testing an instance, these test an invariant.
The gate matters as much as the count: every release runs the entire stack, PHP tests, JavaScript tests, the 14-check audit script, static analysis, syntax lint across multiple PHP versions, and a documentation-drift detector that fails the build when the docs’ claims stop matching the code. A suite that runs occasionally is a report; a suite that gates every release is a contract.
Three kinds of tests that earned their keep
1. Regression pins: every bug becomes permanent knowledge
Every user-reported bug ends life as a named test asserting the exact reported scenario can never recur, the doubled Hebrew brand suffix (Part 7), the mangled emoji (Part 8), the importer mutations (Part 6). The pin uses the exact reported input, the real Hebrew string, the real emoji, the real punctuation, so the test encodes the incident itself, not a sanitized approximation of it.
Some pins even assert source patterns: one test verifies a buggy comparison can’t be reintroduced by literally checking the code doesn’t contain it; others assert that a dangerous capability remains deleted (Part 10) or that a security escape flag is still present (Part 13). Pinning the shape of the code sounds crude, and it is, usefully so. It catches the well-intentioned revert, the “simplification” that removes a guard, the merge that resurrects an old branch. Over hundreds of releases, those are not hypothetical events.
2. Self-syncing structural tests: lists that can’t drift
Any place two lists must stay synchronized eventually drifts: registered endpoints vs. their handlers, options created at install vs. cleaned at uninstall, meta keys written vs. read. Our answer was tests that parse the code and enforce the sync automatically, a parser walks every registered AJAX/REST/CLI endpoint and verifies its callback exists, is public, and is gated; another cross-references every stored option against uninstall cleanup, every scheduled task against its removal, every stored key against a reader.
When we first built the endpoint parser, it revealed the hand-maintained test list had silently drifted 50+ entries out of date, the very test meant to guarantee coverage had itself decayed. That discovery generalized into a rule we now apply everywhere: never hand-maintain a list that code can derive. A hand-maintained list is a second copy of the truth, and second copies drift. The defect class is now structurally impossible across all four dispatch layers of the plugin, because the tests read the registrations themselves.
The same family caught quieter failures: a meta key that was read in two places but written nowhere, dead code in the best case, a feature silently returning empty in the worst (it was the worst: a whole schema feature had been reading a key under a slightly different name than the one being written, for its entire production life).
3. Byte-fidelity certification
Covered in Part 8: hostile-character round-trips through every write surface, turning data-corruption disputes into five-minute diagnoses. Its role in the wider culture: it is the template for certification tests generally, tests that do not check a feature but prove a property, end to end, in a form you can show a client.
Bugs that tests found (not users)
A sample of what coverage campaigns surfaced in “working” code: a robots.txt parser silently dropping every rule after the first per user-agent block; a regex that failed to match the canonical YouTube embed URL format, so video schema had silently not been generated for dozens of releases; a redirect-map builder destroying links on sites with plain permalinks; day-lists in opening hours dropping two of three days; a PHP 8 fatal hiding in a defensive code path that production inputs never reached, until they would have; cache invalidation that cleared the database copy but not the object cache, so busy sites kept serving stale data; and a family of interface defects where status styling silently never applied because an emitted class name had no matching stylesheet rule.
Notice the common signature: every one of these was silent. No error, no complaint, no log entry, just quietly wrong output. Users cannot report what produces no symptom, and monitoring cannot alert on output that is merely incorrect. Silent bugs have exactly one natural predator, and it is a test that asserts what the output should be.
The economics deserve a moment, too. Several of these defects had been live for years across busy production sites, degrading real outcomes, missing rich results, dropped crawl rules, broken internal links, the whole time. The cost of each test that found one was an hour or two of writing. The cost of each bug, compounded over its silent lifetime, is unknowable but certainly larger. Coverage campaigns look like overhead on a budget line; measured against the defects they surface, they are among the cheapest engineering work a mature product can buy.
The audit reflex: every fix hunts its siblings
The culture’s most productive habit: after fixing any bug, immediately audit the codebase for the same defect class elsewhere. One truthiness bug became a five-iteration sweep that found eleven more instances (Part 15). One cache-invalidation bug pointed directly at three sister sites with the identical flaw. One missing write-compensation for slashes turned out to exist on four other write surfaces. Fixing the instance is the minimum; fencing the class is engineering.
The reflex has a second half: every sweep ends by pinning the class, not just the instances, a structural test, a source-pattern check, or an audit-script rule that fails the build if the pattern returns. That is the difference between having fixed eleven bugs and having made the twelfth impossible.
The forgotten gate: documentation drift
One member of the release pipeline surprises everyone who hears about it: a documentation-drift detector. It reads the project’s own documentation, the counts, the claims, the described behaviours, and fails the build when the documents stop matching the code. Endpoint totals, test counts, references to removed features: all verified mechanically on every release.
The reasoning is the same as for the structural tests: documentation is another hand-maintained copy of the truth, and copies drift. In a long-lived project, stale documentation is not a cosmetic problem, it is how the next developer confidently does the wrong thing, and how a future maintainer “restores” behaviour that was deliberately removed. Treating documentation accuracy as a build gate sounds bureaucratic and costs almost nothing; the detector is a small script, and the failures it catches are exactly the ones nobody would otherwise notice until they had misled someone.
The same release discipline bundles everything into one verdict: PHP tests, JavaScript tests, the audit checks, static analysis, multi-version lint, drift detection. A release is not “tests passed”; it is “every gate passed.” The distinction keeps quality one-dimensional, green or not, which is the only dimension a maintenance-mode project can afford to argue about.
What it costs, honestly
Tests were the majority of development effort in the project’s mature phase, deliberately. For software that’s feature-locked and in maintenance mode (Part 20), the test suite is the product’s warranty. Every future change, by anyone, is checked against every lesson ever learned.
The costs are real and worth naming: suite runtime grows and needs occasional pruning of redundancy; tests themselves can be wrong (we found a “focus trap” that existed only inside its own test file, Part 17, and test fixtures that accidentally reproduced the very bug they were meant to catch); and structural pins add friction to legitimate change, which is precisely their job and occasionally annoying. The return is a specific kind of confidence: a one-line fix in year three ships in an afternoon, because 4,700 checks stand between the change and the client.
For readers who want to start: do not aim at a number. Adopt two rules and let the suite grow itself, every bug becomes a named regression pin before the fix merges, and every synchronized list gets a test that derives it from the code. Those two rules, applied for a year, build the suite this post describes.
Next: Part 15, The ‘0’ That Broke Everything
FAQ
How many tests should a WordPress plugin have?
Enough that every user-visible behavior, every past bug, and every synchronized structure is pinned. This project’s ratio landed around one test per ten lines of production code, high, and worth it for maintenance-mode software.
What is a regression test?
A test written from a specific bug, asserting the exact failure scenario can never recur. Named after the bug, kept forever.
What is a self-syncing structural test?
A test that parses your own code (endpoint registrations, option lists) and enforces consistency automatically, so paired structures can’t silently drift apart.
Can writing tests find existing bugs?
Reliably, yes. Covering previously untested code in this project surfaced dozens of live, silent production bugs, parsers dropping data, regexes missing valid inputs, error paths that would fatal.
Why do silent bugs stay hidden for years?
Because they produce no symptom: no error, no crash, just quietly wrong output. Users cannot report what they cannot see, so only a test asserting correct output can find them.
Where should I start testing an existing codebase?
Start with two rules: pin every new bug with a named regression test before fixing it, and write structural tests for any two lists that must stay synchronized. The suite grows itself from there.
