Dealing with Flaky Tests

For the past day or so, I have had my head down, tear­ing my hair out, try­ing to de­bug a sig­nif­i­cant block caused by many en­gi­neers’ arch neme­sis: a flaky test. In my case, it’s ac­tu­ally been sev­eral flaky tests.

A flaky test is any test for your code that only some­times fails. It shows up when a part of your test re­lies on some­thing non-de­ter­min­is­tic. Non-de­ter­min­is­tic com­po­nents can in­clude:

• Fetching re­sources over a net­work con­nec­tion.
• Reading from a ran­dom num­ber gen­er­a­tor.
• Using a non-de­ter­min­is­tic asyn­chro­nous sched­uler.
• Animations (often use the GPU).
• …and many more.

If your ap­pli­ca­tion in­cludes code that looks any­thing like these, you are at risk for flaky tests.

My use-case hap­pened to be es­pe­cially vul­ner­a­ble to these, since our in­te­gra­tion tests run against a wide va­ri­ety of text ed­i­tors. See my orig­i­nal blog post for more de­tails.

Flaky Tests SUCK

It’s hard to ex­press how frus­trat­ing a flaky test can be if you have not ex­pe­ri­enced a bad one your­self. In short, they slow de­vel­op­ment time, cause un­in­ten­tion­ally brit­tle soft­ware, and they can cre­ate a lot of frus­tra­tion for users, since the is­sues they al­low into pro­duc­tion are in­con­sis­tent. They make all the ef­fort put into pro­duc­ing tests, wasted.

Tests are de­signed to catch bugs be­fore they get into pro­duc­tion. If a test ex­pects cer­tain be­hav­ior out of non-de­ter­min­is­tic sys­tems, these tests can pass lo­cally and in con­tin­u­ous in­te­gra­tion, but fail on a user’s de­vice.

This is a prob­lem pri­mar­ily for users, but also for de­vel­op­ers. It’s good if a flaky test fails in CI. After all, that is how you know it ex­ists. But it is very bad if a flaky test fails in CI and the orig­i­nal au­thor of the test is no longer around. In that case, it be­comes dif­fi­cult to de­ter­mine which non-de­ter­min­is­tic sys­tem is caus­ing the fail­ure, and thus how to fix the un­der­ly­ing bug rep­re­sented by the flak­i­ness.

Prevention

The best way to get around flaky tests is to make it as dif­fi­cult as pos­si­ble to cre­ate them in the first place.

When I first started us­ing Playwright about a year ago, I did­n’t think too much about its con­fig­u­ra­tion. Boy was that a mis­take.

I re­ally like Omakase soft­ware, and I had (mistakely) as­sumed that Playwright was an ex­am­ple of it. Thus, I as­sumed a de­fault con­fig­u­ra­tion for my pro­ject and let it be. While Playwright is a phe­nom­e­nal tool, I ended up be­ing wrong. The de­faults I re­ceived from my pro­ject tem­plate were not op­ti­mal, and they would sub­tly waste a lot of my time over the next year. You see, there is con­fig­u­ra­tion for Playwright that can pre­vent most flaky tests from en­ter­ing your code, and they aren’t set up by de­fault.

To avoid most is­sues, change these two set­tings:

export default defineConfig({
    retries: 0, // Was 4 by default.
    repeatEach: 2, // Was 0 by default.
});

In my pro­ject, I added a small nudge to dis­cour­age peo­ple from re­mov­ing these:

export default defineConfig({
	/** Extremely important to avoid flaky tests. DO NOT CHANGE or I will kill you. */
	retries: 0,
	/** Extremely important to avoid flaky tests. DO NOT CHANGE or I will kill you. */
	repeatEach: 2,
});

I jest, of course.

What do they do? They are pretty self-ex­plana­tory. retries: 0 will con­fig­ure Playwright to never retry a test if it fails. If it fails, it is re­ported as a fail­ure and your con­tin­u­ous in­te­gra­tion will stop, forc­ing a hu­man to deal with it. repeatEach: 2 will run each test twice. A po­ten­tial flaky test might pass on the first round, but is much less likely to pass twice. Some flaky tests may still sneak through. You may want to con­fig­ure it more ag­gres­sively in some stages of your CI/CD pipeline.

I have no­ticed that AI agents are es­pe­cially prone to pro­duc­ing flaky tests, since they will of­ten sim­ply run a test again and again un­til it passes.

Debugging

As for de­bug­ging, I can pro­vide no bet­ter ad­vice than I typ­i­cally do:

• Use a de­bug­ger.
• Use git bisect if nec­es­sary (or have an agent do it).
• Work slowly and me­thod­i­cally. Slow is smooth, smooth is fast.