Top Five Challenges in Test Automation

Test automation is an essential subset of software testing. By using automated testing, we can expedite the process of software validation and increase testing coverage. However, there are a lot of challenges in applying test automation for applications under test (AUT).

Without overcoming these challenges, testers might face countless nightmares that can cause software automated testing failure. This topic aims to outline the top five challenges that have the highest impact on the overall automation test effort and project success. Hopefully, the earlier these challenges are understood, the better solutions are prepared to deal with them.

1. Effective Communicating and Collaborating in Team

This is perhaps a challenge not just in test automation but also in manual testing teams. However, it is more complicated in test automation than in manual testing because it requires more communication and collaboration in the automation team. Indeed, test automation is an investment. Therefore, like any other investment, to get the whole team members involve in identifying test automation objectives and setting targets, we need to spend significant efforts on communication and provide huge evidence, historical data, and we even do a proof of concept. In addition, to have clear purposes and goals, we necessarily keep the entire team on the same page. Unlike manual testers, we, automation testers, not solely talk with developers, business analysts, and project managers about the plan, scope, and timeframe but also discuss what should and shouldn’t be automated with manual testers, developers, and technical architects.

Moreover, we have to present the cost and benefit analysis along with the Return on Investment (ROI) analysis to the higher management team. Absolutely, without management team support, the whole test automation effort will be put at risk. Hence, how we communicate and collaborate among these teams and others effectively is a big challenge. Clearly, ineffective communication and collaboration can easily turn test automation experiences into a nightmare.

2. Selecting a Right Tool

Nowadays, there are a variety of testing tools, ranging from free and open-source tools like Katalon and Selenium to commercial tools like TestComplete and supporting different testing types and technologies. Each tool tends to support particular situations. Vendors of testing products have a tendency to exaggerate the ability of their products. Vendors often assume that they have a “secret sauce” for all automation tastes. This causes misconceptions and confusions for us to select an appropriate testing tool satisfying our needs. Plus, many of us do not do enough research before making a decision about tool selection, and we tend to buy popular commercial tools quickly based on an inadequate evaluation. Remember that a sufficient assessment includes defining a set of tool requirements criteria based on the AUT and the experience of experts who have already used the tools considerably.

Unfortunately, people do not have enough resources to fulfill this requirement. No matter what kind of process and testing methodology we have, if a tool does not match our technical and business expectations, we will give up using it. Eventually, test automation will be failed and not be applied in testing activities any longer. In my point of view, choosing a test tool is as complicated as getting married to a person. If you marry with an inappropriate person, you tend to break up sooner or later. Similarly, without a suitable test tool, we will deadly end up with failed test automation effort.

Read more: How to Select the Right Automation Testing Tool?

3. Demanding Skilled Resources

Some people claim that test automation can be just handled by manual testers or any technical testers because many test tools already support recording test scripts and playing back them so easily and quickly. This is a huge myth. In fact, test automation requires the necessary technical skills to accurately design and maintain test automation framework and test scripts, build solutions and resolve technical issues. Automated testing resources need to have strong knowledge of the framework’s design and implementation. To fulfill these job requirements, obviously, these resources need to have both strong programming skills and solid test automation tools. On the other side, others believe that developers can entirely manage and take the test automation responsibilities.

However, how developers can write correct test scripts at testers’ views and end-users’ needs is a big concern although they easily develop a piece of codes in accordance with the test automation framework. Certainly, we can utilize our resources within our test automation process to be more effective. However, skilled resources are always of importance in test automation effort.

4. Selecting a Proper Testing Approach

Automation tests not simply require a right tool to create scripts but also need a correct testing approach. This is one of the biggest challenges for test automation engineers. Technically, it’s vital for testers to find an appropriate test automation approach. In order to do so, they have to answer several important questions: How to reduce effort in both implementation and maintenance of test script and test suite? Will automation test suites be having a long lifetime? How to generate useful test reports and metrics? With adopting the Agile development in recent years, the application under test often changes through development cycles.

Don't miss: Agile testing methodology - A complete guide for agile testers

Therefore, how to design and implement automation test suites to correctly identify these changes and keep up-to-date quickly with reasonable maintenance effort. It is ideal to have a test automation solution that can detect these issues to automatically update and re-validate the test without any human intervention. Definitely, it’s not easy to address these difficult questions.

5. High Upfront Investment Cost

Talking about test automation, most of us agree that automated regression testing is crucial and useful in most Agile contexts. But when turning into the cost, we have many concerns. As a matter of fact, the initial phase of test automation is usually expensive. It’s necessary to analyze, design, and build a test automation framework, libraries or reusable functions, etc. In some cases, it is required to take into account licensing costs, facilitating and operating costs such as hardware and software costs.

Moreover, even though we can use free open-source tools to reduce the licensing costs, we might spend significant efforts on learning, training, and maintaining them. Furthermore, we also take hidden costs into consideration. How to account for hidden costs such as meeting, communicating, and collaborating. As a result, how we can ensure that these things cannot affect our decisions. Although there is a huge payoff in the long run after running some regression testing cycles, convincing the stakeholders to have a consensus about this investment is a big challenge. Actually, just due to budget constraints, many people tend to give up test automation even though they agree with an executable goal and high ROI.

Challenges in Test Automation: Survey Key Results

Katalon and KMS Technology conducted the survey with ToolsQA as the media sponsor to understand and offer evidence on challenges being faced by worldwide test professionals and their organizations with test automation. The infographic below shows key results of the survey about test automation types, tools selection criteria and the severity of the issues.

Conclusion

Absolutely, these challenges are not the only ones found in test automation, yet they are common. If we don’t have solutions to overcome them, the test automation project can easily result in failure.

Test automation is nowadays dominating in the Agile development contexts. Many test automation projects, to some extent, have been brought out tremendous benefits and successes when people are aware of and control test automation challenges effectively. There are still controversies about whether we should promote test automation in testing activities or not, from where I stand, test automation might have a huge payback, contributing to result in valuable investments.

Going to Test Automation with Katalon Studio now! (Trusted by +100 companies)

Source

HCMC Software Testing Club http://www.hcmc-stc.org/blog/top-five-challenges-in-test-automation

About the author

Thao Vo is a co-founder and member of Board of Directors at HCMC Software Testing Club. He is currently a QA Architect at KMS Technology. Contact him at thaovo@kms-technology.com.