Flaky tests hurt!
Selenium has completely changed the game for endless software companies and became a critical key for agile development.
But, as we’ve written in a previous article, Selenium isn’t flawless, and one of the most common Selenium hurdles are recurring flaky tests. Selenium flaky tests are so common that they not only impact companies’ bottom line, but lead to ‘tests’ result blindness’ that impact quality, and also frustrate the testers.
Flaky tests significantly slow down testers’ productivity, as they struggle to differentiate between a test error that implies a bug, and a false positive.
So, what can you do? Concrete tips!
Here I decided to share a couple of tips that I’ve found very useful for my everyday life as a Selenium coder:
- Use waits wisely, it is better to use explicit waits instead of implicit waits
- Avoid using Thread.sleep()
- Avoid using driver.manage().timeouts().implicitlyWait(X,TimeUnit.SECONDS);
- First of all, try to find the id, short fast safe and unique
- Then try to find className
- Then name, tagname, linktext, etc.
- Xpath and CSS should be the last options, as they are less reliable
- Avoid using locators that are likely to change
- Try adding automation locators so UI developers won’t change them (for example “aut-click-btn”).
- That way, If a UI developer tries to change the class name, he/she will know to change any values but the automation locator, which has a prefix of “aut-“
- Avoid using hardcoded Strings in automation tests
- If possible, get Strings from the same place the UI developers get them, so if requirements for those Strings changed, it wouldn’t affect your tests.
For example, get your Strings from the i81n file (if exists)
- If the first run ends with, say, 20 failures, you need to automatically generate a new run that contains only the failed tests. This way you get a new run with only 20 tests, and now you will most likely reduce the number of failures, in case they are flaky
- Keep editing the machine resources until you achieve the ideal CPU and RAM to run your tests without slowness problems
- You should restart your machine when you feel it affects your tests
- Try to determine the ideal number of machines that suits the number of tests
- Keep trying to edit the number of parallel threads until you get an optimal number (sometimes a lot of threads cause slowness)
- Your number of threads should be configurable and suit the number of tests
I know that Selenium flaky tests can be a pain. I totally get it. But don’t give up. The good news is that there’s a huge community of Selenium coders out there who are more than willing to give back and share their wisdom.
Besides, we at Shield34 have created a product that deals with exactly this, and you can use it for free.