As the sun’s rays begin to light the room, the soft chirping of birds slowly wakes you to another bright and hopeful day of…
Whoops! Wrong story. Let’s see now… *shuffling papers* Ah, here it is.
BEEEEP. SNOOZE. Another day begins. You open one eye and look at the glaring lights on the clock. Time to start a new day. Maybe you jump out of bed full of energy, work out, and head to work fully refreshed; or maybe you stumble through your morning routine, chug down your caffeine fix, and hope today will be just a bit less frustrating than the day before. If you are the former, you are a rare gem.
In many ways, being a Selenium automation tester isn’t so different from any other high-tech job. Wasted time in meetings, overdosing on coffee, spending too much time choosing where to get lunch… the usual.
The work itself is where things get, well, flaky.
Morning time, second cup of coffee
Once you get to the office, make yourself that second cup of coffee and acknowledge the existence of your coworkers, you sit at your desk, crack your knuckles, and take a deep breath. Will the flaky monster strike again, or will work today be smooth sailing?
Does Selenium have something against us?
Before anyone starts preparing their speech on why Selenium is great – we agree. It’s open source, free, and knows how to “act like a real web user” for the testing we need. And there’s more.
Millions of companies use Selenium thanks to its capabilities, which is why Selenium coders are so needed. Thanks to Selenium automation testers, companies can quickly test and launch software versions and keep agile. That being said, it’s not easy dealing with the downsides of automated testing.
Back to your typical day.
There’s a delay in the latest software update because of a test that came back with an error. Everyone is already pretty stressed out about the release, and now it’s up to you to try and figure it out.
Unfortunately, you can’t seem to reproduce the bug fast enough in order to understand what its root cause is and get it over with. After spending hours on the error, you learn that it was probably due to a timeout caused by a network failure. Come on, not this again! Now you need to decide. Did that cause the error for sure? Should you keep digging, in case there really is a bug in the code, or can you tell everyone to ignore it and it was just yet another flaky test?
Another cup of coffee. Make it a double…
Lunch time already?
it’s almost noon. And as usual, you’re already starving. “What’s the update on that error?” you are asked, again. Everyone wants answers now, and you have to either delay lunch (unthinkable!), or grab something to eat at your desk. You’ve been at it a while, you already know that the while the first error was a ‘false positive’ error, a second error result is because of a bug, only you don’t know why it’s happening. It’s not your job you think, let the developers deal with it, I’m just the guy who is in charge of the testing, but, hmm… that’s not how it works in your company.
Without a proper root cause, you’re running blind… You have no way of knowing why it’s happening. But Selenium doesn’t report the root cause of an error. Expert help is needed, or some heavy digging. You’re starting to feel frustrated and hungry.
Add to that the fact that tests are running too slowly for you to get things done as fast as you’d like, and you’re back at the kitchen making another cup of coffee. Execution times are often longer than expected, and Selenium testers need to make sure they use the right wait points or risk much slower run times.
Ideally, you’d run parallel testing, but Selenium doesn’t support that out-of-the-box. The whole point of Selenium is to use automated testing to support an agile environment, but Selenium doesn’t seem to care about your agile KPIs… you’re on your own buddy.
What time is it now?
The day passed by. The version isn’t out. You think it’s dark out, but you’re not sure because you haven’t looked up from your computer for way too many hours. Cups of coffee? You lost count after the fifth one. And the bugs? Too many of them are still alive and kicking, staring back at you … a mystery.
Is there hope?
Let’s put humor aside. Basically, automated testing can be super frustrating due to a significant percentage of false-positive results, slow runtime, lack of automatic root-cause analysis and more.
What do Selenium coders do to ease the pain?
- They give back the community
You’re probably familiar with communities like StackOverflow, Github or alike, sharing code pieces, hacks and tips with other testers, and searching for others who have faced similar problems. One of the benefits of using an open source tool like Selenium is the amount of online support available from the community.
- External tools
There are external test automation solutions available that help speed up test execution and solve the issue of flaky tests. While these tools are usually based on Selenium, they won’t let you keep coding Selenium like you’re used to, and their users are required to learn a whole new method. These tools may come handy, but keep in mind that they will most likely not optimize your existing tests, and you will need to recreate them.
- DIY Selenium code optimization
Some companies bring in experts who work hard to make tests more robust and reliable by adding rules to the tests. But this is a long, tedious and extremely expensive process that does not suit everyone.
- Add external Selenium optimization later
Those who wish to keep coding Selenium while enjoying the benefits of optimized tests can now use a new kind of solution – Selenium optimization layers, like what we develop here at Shield34. The idea is to add a transparent layer to your code, one that does not interfere with anything you do, but stops flaky tests, generates automatic detailed root-cause reports, and lets you write and run tests 500% faster.
Tomorrow can be one of those days. Birds chirping, bees humming, stress-free mornings, lunch on time with colleagues, and clocking out before dark. Well, let’s not take it too far, but generally speaking, there’s room for improvement.
You’re not alone here. Software companies clearly understand the automated testing brings a huge promise but it’s far from being optimized. Not only Selenium coders are having a hard time. Agile organizations are struggling as a whole.
We created Shield34 out of our own experience and after realizing that things must change. We were not the only ones: an entire industry was born – the industry of test automation solutions.
My advice to you is – don’t keep things as they are. Proactively work to optimize the way you run automated tests; you don’t have to enjoy ~20% flaky tests, slow runtime and lack of root-cause analysis.