Automation Testing 101: What, Why and How

What is automation testing?

In the software testing world, there are two types of testing techniques – manual and automated. Both kinds aim to execute the test case, then compare the actual outcome with the expected result.

To put it simply, manual testing is a testing technique performed by human effort to ensure the software code does anything it is supposed to do. So, what is automation testing? On the contrary, it is the practice of running tests automatically, managing test data, and utilizing results to improve software quality.

If you are familiar with testing, you understand that successive development cycles require the execution of the same test suite repeatedly. This process can be extremely repetitive and time-consuming if you perform it manually. However, by leveraging a test automation tool, it is easier to write the test suite, re-play it as required, mitigating human intervention, and improving testing ROI.

Why do we automate a test?

This is one of the questions I ask when I interview candidates for a Test Automation role and to my surprise, many candidates seem to miss the main and most important reason to automate a test. Some of the answers I get from candidates are quite credible, but still not the answer that I’m looking for. Some of the answers I get to the above question are:

Increase Test Coverage

This answer is quite valid, but how do we define coverage? If we have 100 tests, how can we measure the percentage coverage?

With a mature test automation practice in place, you could be running hundreds of tests in a relatively short period of time.

Because of this, we can create more test cases, more test scenarios and test with more input data for a given feature and thus gain more confidence that they system is working as expected.

However, in testing and especially test automation, more tests don’t really mean better quality or more chance of finding bugs.

In a post by Martin Fowler, where he discuses Test Coverage, he mentions

“If you make a certain level of coverage a target, people will try to attain it. The trouble is that high coverage numbers are too easy to reach with low quality testing. At the most absurd level you have AssertionFreeTesting. But even without that you get lots of tests looking for things that rarely go wrong distracting you from testing the things that really matter.”

Save Time

This answer is also true as you can spend valuable time doing interesting exploratory testing while the automated tests are running. However, for a brand new feature that has been developed, it could actually take longer to write automated scripts than to test the feature manually in the first instant.

So, it is important to note that to save time from automated tests, it requires an initial increased effort in scripting the automated tests, making sure they are code reviewed, and that there are no hiccups in the execution of automated tests.

Find More Bugs

This answer worries me sometimes as I have never seen any metrics that suggests there were more bugs found by automation than manual / exploratory testing. Automated tests generally check for any regression in the system after new code has been implemented.

There is always more chance of finding bugs in new features than in existing functionality. Furthermore, there are other reasons why automated tests fail to find defects.

Replace Manual Testers

This is probably the worst answer I have heard in regards to why we automate a test. There is a clear distinction between what a manual tester does and what an automated test checks. Automated testing is not testing, it is checking of facts.

In order to be able to automate a test, we have to know the expected outcome so that we can check for the valid or invalid outcome. This is what gives us true or false, positive or negative, pass or fail.

Testing, on the other hand, is an investigation exercise, where we design and execute tests simultaneously. Many things can behave differently where only an observant human tester can notice.

Good manual testers will always be needed because of the different mindset and the ability to question the system.

Improve Quality

Although automated tests are capable of giving us quick feedback and alert us about the health of an application, so that we can revert any code change that has broken the system, automated testing on its own does not improve quality. Just because we have a mature test automation in place does not guarantee that no bugs escape into production.

We can improve quality by ensuring correct practices are followed from start to finish of a development cycle. Quality is not an afterthought; it should be baked in right from the beginning. It is not enough to rely on automated tests to get a picture of the quality of the product.

Benefits of automation testing

Now that we have gone through what automation testing is, it is time to glance at several benefits of automation testing to help you eliminate the ambiguity on whether automation testing is the right choice for your team. The following are highlighted points on why automation testing is so important:

Simplify test execution

With the automated testing tools, test scripts can be reused as often as you need, thus saving both time and effort. Imagine using manual testing, you have to write a single code line for the same test case, over and over again.

Reduce human intervention

Utilizing automation tools, you can run automated tests unattended (overnight) without human intervention. Once written, the tests can be reused and executed unlimited times without additional cost. The tests are also available 24/7, unlike manual testers!

Speed up test

The speed of test execution and test coverage increases, thus shortening the software development cycles.

Increase test coverage on multiple platforms

Automation testing grants you the ability to perform testing on multiple platforms in parallel without creating abundant test cases in different browser versions.

When to opt for automation testing?

While QA teams turn their testing strategy towards a more inclusive automation approach to increase efficiency and coverage of the testing process, there are still testers wondering if automation testing is the right choice for them.

Automation is an integral part of a development cycle, so it is essential to determine what you want to achieve with automation before switching into it. A test should meet some criteria in order to be automated – otherwise, it may end up costly investment rather than saving.

Ultimately, it is substantial to remember the goal of automation is to reduce your time, effort, and money. Take the below criteria into account before making your own decision

• High Risk – Business Critical test cases
  

  Some test cases may contain severe risks, which will have a negative impact on the business. The negative impact includes costs, customer dissatisfaction, poor user experience.
  In case the whole testing process is run by a manual tester, even by the most experienced one, there is always a higher possibility of error-prone codes. Running an automated test is considered as a better way under risk-based testing, where higher priority should be put to prevent these unexpected errors.

• Repetitive test cases
  

  There is no sense in applying automation testing tools for the tests that can only be run one time. Under these circumstances, repeatable tests can be run on-demand, resulting in a reduction of the cost per test run and the time to complete a development cycle.

• Functional test cases
  

  Functional testing is also an excellent time to take advantage of automated testing. You can quickly and seamlessly detect the real-time performance of the functional requirements. This approach allows you to achieve accuracy, interoperability, and compliance at ease.

Recommended Article: Types of Automation Testing: A Beginner’s Guide

5 Steps to get started with Automated Testing

Step 1: Defining the Scope of Automation

The scope of automation means the area of your Application Under Test that will be automated. Make sure you have walked through and know precisely your team’s test state, the amount of test data, also the environment where tests take place. Below are additional clues helping you to determine the scope:

• Technical feasibility
• The complexity of test cases
• The features or functions that are important for the business
• The extent to which business components are reused
• The ability to use the same test cases for cross-browser testing

Access our FREE course on Katalon Academy: Create and Execute Test Cases.

Step 2: Selecting a Testing Tool

After determining your scope, it is now the time for you to pick up a tool for automation testing. Of course, you can select it from a wide range of automation tools available in the market. Yet, it solely depends on the technology on which the application tests are built. Each type of tool or framework may serve different demands, therefore having a thorough understanding of multiple tool types is also a prominent factor in choosing your best tool.

Step 3: Planning, Designing, and Development

At this stage, you will create an automation strategy and plan. This plan can include the following items:

• Your selected automation testing tool
• Framework design and its features
• A detailed timeline for scripting and executing test cases
• In-scope and Out-of-scope items of automation
• Goals and deliverables of automation testing process

Step 4: Executing Test Cases and Build your reports

Once finishing all of the preceding steps, it is time to take action! You can write the scripts, run the test automatically, either by running the code directly or by calling an application’s API or user interface. After your execution, the test report provides a consolidated summary of the testing performed so far for the project.

Step 5: Maintaining previous test cases

No matter how well you manage the automation testing, test maintenance is unavoidable if you want to expand your collection of reusable test scripts. Once your automated tests have been scripted and running, they still need updating if the application changes the next time.

Start Web Testing with Katalon Studio now!

Conclusion

To sum up, this article provides you with an introduction to automation testing, the benefits of automation testing, and how to start your journey with it. We believe this is the best way to fulfill most of the testing goals with practical resources and time in an Agile world. But be careful before choosing the type of automation that fulfills the requirement of the application because no one can meet 100% requirements.