A flaky test passes and fails on the same code, and the usual reaction is to hit "re-run" and move on. Our earlier post on tackling flaky tests covered the general strategies: tracking history, isolating tests, mocking dependencies. This one is about the step in between, the one that decides which of those strategies you need. Almost every flaky test comes down to one of seven causes, and the run history usually tells you which one before you open the code. Below are the seven, each with a short example, the pattern it leaves in the run log, and the fix.
A flaky test gives you two sources of evidence, and each answers a different question. The run log tells you when it fails. Take the last ten or so runs and ask what every failing run has in common that the passing runs do not: a different runner, a different start time, a different test that ran just before it, a different number of parallel workers. The code tells you why. Once the log has narrowed it down, the code shows the line that depends on whatever changed.
Reading only the code leads to guessing. Reading only the log tells you what correlates with the failure, but not what to change. You need both. Here is a small example, a test that failed three times in its last six runs:
| Run | Result | Ran after | Runner |
|---|---|---|---|
| 6 | pass | none | runner-1 |
| 5 | fail | removes a coupon | runner-1 |
| 4 | pass | none | runner-2 |
| 3 | fail | removes a coupon | runner-2 |
| 2 | pass | none | runner-1 |
| 1 | fail | removes a coupon | runner-1 |
The runner column is a decoy: both runners pass and both fail. The "ran after" column explains every failure. The test fails exactly when removes a coupon runs right before it, which points at test order (cause 3 below), and the code will show the test leaning on something the previous test cleaned up.
The test sleeps for a set time and hopes the app is ready by then. Two seconds is plenty on a quiet laptop and too short on a busy CI runner.
await page.click('#pay');
await page.waitForTimeout(2000);
expect(await page.textContent('#status')).toBe('Paid');
What the log shows: failures cluster on slower or more heavily loaded runners, or at the busiest times of day.
The fix: wait for the thing you actually need, not for time to pass. A web-first assertion keeps checking until the text appears or the timeout runs out:
await page.click('#pay');
await expect(page.locator('#status')).toHaveText('Paid');
Like a fixed wait, but worse: nothing waits at all. The test acts or checks before the app has finished, often because an await is missing or a value is read before a request lands.
page.click('#save'); // no await
const rows = await page.locator('tr').count();
expect(rows).toBe(11);
What the log shows: the same pattern as a fixed wait, failures that follow how slow the run was. The difference is in the code: there is no sleep to blame, just a check racing the app.
The fix: await every action, and assert with a retrying assertion instead of reading a value once:
await page.click('#save');
await expect(page.locator('tr')).toHaveCount(11);
The test depends on something a test that ran before it left behind or cleaned up. It passes in the usual order and fails when the order changes, when it runs alone, or when the suite is split differently across machines.
test('removes a coupon', async () => {
await deleteCoupon('SAVE10');
});
test('applies a valid coupon', async () => {
await applyCoupon('SAVE10'); // assumes SAVE10 still exists
});
What the log shows: a "ran after" or "shard" column lines up with the failures. They follow one specific test, or happen when the test runs first or on its own.
The fix: every test creates the data it needs and does not assume any other test did, or left it alone. If every test passes on its own and with the order shuffled, you have removed this whole class of flakiness.
This one sounds like test order but is different. Tests running at the same time use the same account, database row, file or global variable, and step on each other.
// every parallel worker logs in as the same user
await login('qa-user@example.com');
await addToCart('mug');
expect(await cartCount()).toBe(1); // another worker also added a mug
What the log shows: failures appear when several workers run at once and disappear when the suite runs serially. A "workers" or "parallel" column gives it away.
The fix: give each worker, or each test, its own data: its own account, its own row, a unique file name. Locking the shared resource works as a stopgap, and our post on parallel Playwright tests that share state walks through that approach, but isolation scales better.
The test depends on the clock, the date, the timezone or the locale. This one deserves a closer look, because sometimes the test is right and the app is wrong:
// app code
const nightsBetween = (checkIn, checkOut) => (checkOut - checkIn) / 86_400_000;
// test
const checkIn = addDays(startOfToday(), 30);
const checkOut = addDays(checkIn, 2);
expect(nightsBetween(checkIn, checkOut)).toBe(2);
The stay always starts 30 days from today, so the dates move every day. On most days the two nights are exactly 48 hours apart and the test passes. When the stay crosses a daylight saving change, the gap is 47 or 49 hours, nightsBetween returns 1.958 or 2.042, and the test fails. Twice a year, for a day or two, and only on runners set to a timezone that has daylight saving at all. A runner on UTC never fails.
What the log shows: a date or timezone column. Failures bunch around month ends, clock-change weekends, midnight UTC, or one runner whose timezone differs from the others.
The fix: stop reading the real clock in the test. Pin the date, or pass it in: Playwright has page.clock, Jest and Vitest have fake timers, and most languages have an equivalent. Then add a test pinned to a clock-change weekend. Here that test fails every time, which tells you the bug is in nightsBetween, not in the test. Counting calendar days instead of dividing milliseconds fixes it.
The test calls a real service over the network that it does not control: a payment sandbox, a third-party API, a mail provider. The service rate-limits you, goes down for a minute, or returns a different number tomorrow.
const rate = await fetch('https://api.exchange.example/rates').then((r) => r.json());
expect(convert(100, 'EUR', rate)).toBeCloseTo(108.2);
What the log shows: timeouts, 429 or 503 errors, and failures that arrive in bursts when the other service is struggling.
The fix: stub the service at the network boundary so the test controls the response, and cover the real integration separately with a small number of contract or smoke checks that are allowed to fail without blocking a release.
The test uses random inputs, or relies on an order nobody promised.
const { rows } = await db.query('SELECT name FROM users'); // no ORDER BY
expect(rows[0].name).toBe('ada');
Without ORDER BY, the database is free to return rows in any order. It often comes back the same for weeks, then changes after an update, a vacuum or a different query plan. The same family includes iterating a hash map or set whose order the language does not promise, and unseeded random generators.
What the log shows: no environment column explains it. The failing runs printed different values from the passing ones, so the evidence is in the failure output, not in the conditions.
The fix: make the order explicit (sort, or compare as sets), and seed every random generator, logging the seed so a failure can be replayed.
When a test goes flaky, work through these in order:
A retry is a reasonable stopgap, not a fix. If you have to keep a flaky test in the suite while you work on it, quarantine it: move it out of the blocking path, give it an owner and a deadline, and track it. A quarantine with no owner is just a place where tests go to be forgotten. Our post on quarantining tests with a pytest pattern shows one way to set that up.
Reading a run log next to the test and naming the cause gets faster with repetition, and real incidents are a slow way to get it: they arrive one at a time and rarely come with the answer. That is why I built Light Trap. Full disclosure, it is part of QArcade, my own project: a set of practice games for software testers. Each Light Trap round gives you flaky tests with their code, the error the failing runs printed, and a run log like the one at the top of this post. You name the cause and, on harder tiers, point at the line the fix changes. Then it shows you which column gave the answer away.
Light Trap needs a free account. If you would rather look around first, the QArcade homepage has a short round you can play without signing up.