About Shield34

This content is brought to you by Shield34: Automatic AI deep reporting. Eliminate flaky tests. Integrate seamlessly with existing tests & write new tests 500% faster. Learn more

With a huge market share, Selenium is currently the uncontested king of the web-automation universe.

More than 25,000 software companies are using Selenium framework, and since its first release in 2004, it has been continuously developing to support a massive number of languages, platforms, and browsers, as well as to assist the widespread use of the Agile methodology.

There’s probably not too many professionals in the automated testing domain who haven’t used it one way or another. The fact that you can find 55K questions (!!) about Selenium in StackOverflow says a lot about its popularity.

As stated in this Microsoft blog, Selenium has clearly become the industry standard.

 

via GIPHY

 

However, with great power comes great responsibility. And unfortunately, while being extremely popular and flexible, Selenium has some major drawbacks that can dramatically hurt its potential and reduce ROI.

 

Faults that are are too critical to be ignored??

While every test automation coder with some experience working with Selenium could easily make a list of its faults, not all drawbacks are equal. I will list ‘only’ three of them – but they are extremely painful ones if you ask me, and could lead to major losses to any company running test automation processes at scale.

 

  • Flaky tests: Selenium’s possibly most critical fault is the relatively high percentage of flaky tests. Not being able to be sure whether a certain result is an indication of a real problem or not is a cause for a lot of pain. No one is happy about a team playing a game of hide-and-seek with bugs that don’t actually exist (you can find common reasons for why flaky tests happen here).

    Flaky tests waste company time (waste its money), increase workload, delay releases and build up a lot of frustration, which is never a healthy thing.Besides, since it’s been reported that in many cases more than 20% of Selenium  tests come up with false-positive results, testers tend to take the dangerous step of ignoring these results altogether. Which of course misses the whole point in QA testing…Agile became so popular because it’s designed to increase efficiency and therefore support the business KPIs and grow the profitability. Flaky tests that delay releases are hurting Agile, and thus impacting your bottom line.

 

  • Testing leads to the natural need to trace the root cause of an error, which completes the cycle. After all, one main goal in automated testing is to locate issues as fast as possible – and fix them as fast as possible. If testing is devoid of root cause analysis, it’s not as good as it can get, right? Again, it’s not a tiny fault. As with the other drawbacks, struggling to find the root cause could delay version releases and increase workload.

 

  • Selenium tests that overcome the above usually do not run fast enough, In fact, you can say they are slow. In the stressful world of software, this is a major fault. Being limited due to slow test runs is not something you can ignore.

 

 

 

 

The ROI suffers – How can we work it out?

Can an organization work around these issues, and somehow still continue using Selenium effectively? Well, there are a few ways it can, by not settling for these faults and finding ways to optimize tools and processes.

When dealing with the question of how to measure the ROI of automated testing in order to optimize it, several ways come to mind. Here, for instance, you’ll find a very thorough analysis of six different approaches. Generally speaking, this resource suggests calculating the cost savings from reducing redundant tests, defect and knowledge leakage, and from ensuring proper test coverage, while weighing it all against the investment (hours needed for development, execute and maintain each test) that go into these practices. “The ROI in test automation can be huge”, states article, and rightly so. Accordingly, companies should aspire to minimize the significant negative impact of Selenium faults on its ROI.

 

Ways to optimize your Selenium automation ROI

 

 

Do nothing and hope for the best

So, what do companies usually do? Well, not all companies were created equal, and different companies have different strategies for dealing with these kinds of drawbacks. Some companies simply do nothing. They’re probably aware that Selenium is a mixed bag, but they choose to bury their heads in the sand. They probably have their own reasons.  

Add your own code

Other companies take a more proactive approach and struggle to independently develop code improvements for the Selenium platform. Why did I write a struggle? This is a highly expensive and complex mission which can become a real burden on a company’s budget and workload (consider that ongoing maintenance is needed).

 

Leave Selenium behind

A third approach is to use Selenium-based external test automation tools that are mostly designed for “codeless experience”, thereby forcing users to leave Selenium coding behind and move to their realm.

These tools are abundant, but I think that preventing coders from using Selenium like they are used to is a disadvantage. Besides, by their very nature, these tools can’t support the optimization of existing (legacy) tests.

 

Add a Selenium optimization layer & keep coding as before

A new and emerging approach is to use add a transparent optimization layer on top of Selenium and keep working as before. That’s the approach we follow here at Shield34. We believe it offers the best alternative for Selenium coders. This approach brings quite a few benefits when compared with the previous approaches. First, it eliminates flaky tests (we use AI for that). Second, it speeds up test execution. Third, it automatically creates root cause analysis reports, thereby reducing the need for redundant search. And finally, it optimizes legacy tests, as opposed to external solutions, which forces users to recreate them if they want them to be optimized as well.

 

 

To summarize

Selenium poses a real opportunity, but it includes challenges that impact companies’ bottom line. Since there are clear ways to optimize it, and the impact on the ROI should be very loud and clear, I think it’s a no-brainer.

Ran Finkelstein

CEO @Shield34 - Ran is an experienced entrepreneur, and prior to managing Shield34 he co-founded GetAlert, was a senior director in Extreme Reality, Juniper networks and more.

ran@shield34.com

Featured Articles