· javascript
Your CI Retries are Hiding Real Bugs
If you start a Playwright project with npm init playwright@latest, the config it writes retries failing tests twice on CI.
That’s a sensible default for a browser suite. Now say a product bug makes one test fail one run in five.
With two retries, the test fails all three attempts only 0.2 × 0.2 × 0.2 = 0.8% of the time, so the build stays green in 99.2% of runs. I planted such a bug in a small service and ran its test suite 41,200 times.
Retries hid the bug about as often as the arithmetic predicts, while the test reports flagged the bug’s test as flaky in about one green build in five. Below are the numbers and the quarantine policy I’d build on them: Who owns a quarantined test, how long it may stay and how it gets out.
