Key takeaway: Flakiness destroys the property that makes tests valuable — that a red build means something is broken. Tolerating it converts your suite from a gate into a delay.
The Damage Is Cultural, Not Computational
The first time a test fails spuriously, someone investigates. The fifth time, they rerun. By the twentieth, rerunning is reflex.
At that point the suite no longer communicates anything. A genuine regression produces a red build indistinguishable from noise, gets rerun, passes on retry because the failure was intermittent in the other direction, and ships.
The measurable cost — CI minutes, waiting time — is real but secondary. The primary cost is that the team has been trained to disbelieve its own tests.
Where Flakiness Comes From
| Cause | Signature |
|---|---|
| Fixed sleeps instead of waiting for a condition | Fails on slow CI, passes locally |
| Shared mutable state between tests | Fails only in certain orderings |
| Real network or clock dependency | Fails randomly, often at boundaries |
| Unspecified ordering treated as ordered | Fails when the underlying store changes |
| Test-to-test leakage | Passes alone, fails in the suite |
| Concurrency without synchronisation | Fails under CI parallelism |
sleep(2) is the most common single cause. It encodes an assumption about timing that holds on a developer laptop and fails on a loaded CI runner. Replace it with an explicit wait for the condition — element present, queue drained, status changed — with a generous timeout.
Order dependence is the most insidious, because it hides until someone adds a test or enables parallel execution. The diagnostic is to run the suite with randomised ordering; if failures appear, tests are sharing state they should not.
Time and timezone assumptions cause failures that cluster around midnight, month boundaries and daylight saving transitions. Injecting a fixed clock removes an entire category permanently.
Handling Them Deliberately
The most important rule is to stop the bleeding immediately. A test that fails intermittently should be quarantined — moved out of the blocking suite — within the day, with a ticket and an owner. Leaving it in place while awaiting a fix is what trains the rerun reflex.
Automatic retries are worth naming as a trap. They make the dashboard green and let flakiness accumulate invisibly, which converts a visible problem into a hidden one. If retries are used at all, record every retry as a first-class metric so that flakiness remains measurable.
Track flakiness explicitly: rate per test, and pass-rate-on-first-attempt for the whole suite. That second number is the health indicator. A suite that passes on first attempt ninety-eight percent of the time is trusted. One at eighty percent is not, regardless of how green it looks after retries.
The Bottom Line
Quarantine flaky tests immediately, fix the root cause rather than adding retries, and measure first-attempt pass rate as a health metric. A smaller suite that is always right is worth far more than a comprehensive one nobody believes.
