TecLeads TecLeads Blog
2026-11-11 · 6 min read

How We Kill Flaky Tests Before They Poison the Pipeline

Workstation with two monitors showing source code
qatest automationflaky testsquality engineeringci-cd

At 2 a.m., a red pipeline is supposed to tell you something useful. Instead, someone reruns the failed job, watches it pass, and approves the release. Nobody investigates because the same test failed twice last week and recovered both times.

That is the real cost of flaky tests. It is not the wasted five minutes. It is teaching engineers that red does not mean stop.

Once that habit forms, a genuine regression gets the same treatment as a timing glitch. Rerun. Merge. Hope. Killing flakiness is therefore not housekeeping. It is part of release safety, and it needs the same ownership as any other production defect.

[ ] Prove the test is flaky before touching it

A test that fails once is a failed test. A test that passes on retry is suspicious, but one retry is not enough evidence to diagnose the cause.

Capture the test name, commit SHA, runner image, browser or runtime version, worker count, seed, duration, and failure output. Preserve screenshots, traces, logs, and the JUnit XML. Without that material, the investigation becomes folklore.

Run the test repeatedly against the same commit and environment:

pytest tests/checkout/test_payment.py \
 --count=50 \
 --maxfail=5 \
 -vv

For Playwright:

npx playwright test tests/checkout.spec.ts \
 --repeat-each=50 \
 --workers=1 \
 --trace=retain-on-failure

Then repeat with normal parallelism. If it fails only with several workers, you have learned something useful. Shared state, resource pressure, or an ordering dependency is now more likely than a bad assertion.

[ ] Make the failure reproducible, not merely frequent

Increasing retries is not reproduction. It is buying more lottery tickets.

Start by controlling the variables that commonly move underneath test automation: random seeds, clocks, time zones, locales, network responses, database state, queue consumers, feature flags, and worker scheduling. Print those values into the job output so a failed run can be reconstructed.

For pytest, make the random order visible and reusable:

pytest --random-order-seed=731942 -vv

When time affects the result, inject a clock or freeze it at the test boundary. Do not scatter longer sleeps through the suite. A sleep hides the race on a quiet laptop and returns when CI is busy.

Also run the failing test in isolation, immediately after its usual predecessor, and in a shuffled suite. Tests that pass alone but fail together are often leaking database rows, environment variables, files, ports, browser storage, or mocked global state.

[ ] Fix the waiting model

Most UI flakes are not mysterious. The test is waiting for the wrong event.

This is fragile:

await page.waitForTimeout(2000);
await page.click('.submit');

This expresses what the test actually needs:

await expect(page.getByRole('button', { name: 'Submit' })).toBeEnabled();
await page.getByRole('button', { name: 'Submit' }).click();
await expect(page.getByText('Payment accepted')).toBeVisible();

Wait for observable state, not elapsed time. Prefer stable selectors based on roles, labels, or explicit test IDs. A selector built from generated CSS classes is a request for future pain.

The same rule applies below the browser. Wait for the queue message to be acknowledged, the database record to reach the expected state, or the API to return a specific response. Poll with a bounded timeout and useful diagnostics. Never loop forever, and never treat any HTTP 200 as proof that the right work completed.

[ ] Remove shared state and ordering assumptions

A reliable automated test owns its data and cleans up after itself. Better still, create disposable state and let the environment disappear when the job ends.

Generate unique identifiers for users, orders, buckets, and files. Do not make every worker update [email protected]. Parallel execution will eventually turn that shortcut into a race.

Database cleanup deserves care. A broad DELETE in teardown can erase another worker's fixtures. Scope cleanup to records created by the current test, use transactions where the application permits them, or allocate a schema per worker.

Check for hidden ordering assumptions by shuffling the suite in CI. If test B requires test A to run first, they are not independent tests. Either combine them into one intentional workflow or give test B its own setup.

[ ] Treat retries as evidence, not a cure

Retries have one defensible job: collecting evidence while keeping a known flaky test visible. They must not turn the pipeline green without consequence.

Record the first failure even when a retry passes. Track the test as flaky, assign an owner, and set a removal date for any quarantine. A quarantine with no owner is where broken QA work goes to become permanent.

Use a policy such as:

flaky_test_policy:
 retries: 1
 record_initial_failure: true
 require_owner: true
 quarantine_expiry_days: 14
 block_release_on_expired_quarantine: true

The exact values can differ. The point is that quarantine is a short containment measure with an exit condition, not a second test suite everyone ignores.

[ ] Separate functional flakiness from performance noise

Performance testing fails for different reasons, but teams often respond with the same rerun reflex. Shared CI runners, cold caches, noisy neighbours, autoscaling, and background maintenance can all distort latency results.

Pin the load generator resources. Warm the system deliberately. Record infrastructure changes alongside the result. Compare distributions and error rates, not one convenient average. If performance thresholds decide whether a release proceeds, run them in an environment stable enough to support that decision.

A noisy performance test is still a test defect. It should not be promoted as proof that the application is fast.

[ ] Put flake control into the pipeline

Flaky test work fails when it depends on somebody remembering to check a dashboard. Wire the controls into CI.

The pipeline should retain failure artefacts, flag pass-on-retry events, detect tests whose runtime has drifted, reject expired quarantines, and publish ownership beside the failure. This is where quality engineering earns its keep. Automated testing, performance signals, and release confidence should be part of the delivery path, not a report assembled after the release meeting.

Review the flake queue regularly. Fix the tests that fail most often, but also watch for broad patterns. Ten unrelated browser tests timing out in one week may indicate overloaded runners, a DNS problem, or a shared test environment falling over. Fixing each assertion separately would miss the system failure.

If you only do one thing this week

Change retries so a pass-on-retry creates a visible failure record with an owner and deadline.

That single change stops flaky tests from disappearing into green pipeline output. It also gives the team a real queue to work from, with evidence attached. Once engineers trust that every red result will be investigated, they stop merging around failures. That trust is the point.


If this is on your plate, TecLeads does exactly this as part of our QA & Test Automation work. If you'd like a second pair of eyes on your setup, book a 30-minute call or explore what we do.

📍 Tech Pulse · today's quick question 🟡 Level: Intermediate Cybersecurity

What is a zero-day?

Pick an answer to see how other engineers voted.

Want a hand with this?

TecLeads helps engineering teams ship faster and more securely.

Book a 30-minute call

← All posts