Automated vs. Manual Testing: When and Why?

Testing is the backbone of the Agile software development methodology. The efficiency of the testing process directly impacts the quality of the deliverables and the goodwill of the organization. Buggy software can lead to poor user experience, delayed project timelines, diminished brand value, and revenue losses.

 A strong testing process consists of several well-planned activities with definite timelines and a set of clear expected outcomes. The execution plan in the testing process, either manual or automated, has always received greater significance and seems to drive the entire process definition. Having a well-defined testing process helps in the following ways, directly and indirectly:

 Gives enough time to iterate the test execution in manual and automated mode

• All test artifacts are created and captured along with a clear Requirements Traceability Matrix
• Scheduling and utilization of testing resources, both in terms of people and environments, becomes effective
• Defect isolation, reporting, and reproducing become easier
• The feedback cycle of identifying and addressing a bug is small and quick
• Test coverage across multiple testing verticals becomes clear and indicates the quality of the software; it includes functional testing, black-box testing, smoke testing, integration testing, regression testing, performance testing, and more
• The strategy of Test data creation and refresh is concise, clear, and repeatable

 Once a test process is in place, it is easier and simpler to automate the execution of the tests and reduce the manual interventions in the test cycle. However, it is impractical to assume that the manual test effort can be completely reduced to zero, and neither is it advisable. It needs to be repurposed correctly.

Test Execution

Test execution takes the maximum amount of time in the entire test process. To ensure maximum and complete coverage, test execution is done iteratively and repetitively. It is planned and performed multiple times with the same and different data sets under different categories of testing, such as functional, non-functional, regression, and environmental. Repeating the test execution several times gives delivery teams complete confidence about the quality of the deliverable. Moreover, it is less expensive to identify a bug during testing and address it, rather than let users discover it during their operations.

The key to reducing test execution time is to automate the activity of test execution. It is important to understand that the basic difference between automated testing and manual testing is in the way the test cases are executed. Irrespective of execution style, the steps of test planning, test scenario identification, data strategy definition, and expected outcome identification need to be performed beforehand.

 The automation of testing includes the following aspects of the test process:

• Test data preparation: Using scripts created automatically by tools like Katalon Studio, or written in SQL, Python, Shell, or other similar languages
• Test cases execution: Using tools such as Katalon Studio, Katalon TestOps, Selenium, or others
• Test results verification: Using tools like Katalon Studio and Katalon TestOps, or scripts such as Groovy, Ruby, Python, and others
• Test results visualization: Using advanced reports in Katalon TestOps, JavaScript-based visualization toolkits, or HTML5 based reporting toolkits

In the case of automated testing, there is an additional step of preparing the scripts for test cases using a tool or programming language.

Comparisons

Whenever automated and manual testing are compared, the following factors are considered; however, automated testing is better on all of these fronts:

• The accuracy of the execution and results
• The velocity of execution of the test cases
• The repetitiveness of the execution steps

Cases for Automated Testing

The automation of testing is useful in modern-day delivery, both, for quality and velocity. However, it does not come for free. During the preparation phase, test teams need to write or record test scripts along with data creation, results verification, and data tear-down steps. Automated testing is worth the effort in order to save the time and cost in most of the cases, such as those listed below:

• Agile development teams require continuous feedback on the quality of their daily deliverables and use them for performing nightly builds. Continuous testing activity is configured within the CI/CD pipeline and executed along with the nightly builds as well as the on-demand builds. It includes unit testing, system testing, integration testing, and regression testing of the build. Test results are shared immediately with the development team to address the issues found.
• At the end of a sprint or a release cycle, the release candidate needs to undergo various types of tests. If performed manually, it’s very time-consuming. Having automation ensures that multiple rounds can be run simultaneously or back-to-back to capture the results. This ensures complete coverage of new functionality, easy regression, configuration verification, and more.
• Another good case for automation is when there are multiple configuration options available in the software and each needs to be tested. Although the test cases remain the same, the test results may change due to changes in the environmental and software configuration. Running the same test cases manually is time-consuming as well as error-prone.
• When concurrency, load, and endurance need to be tested for software, it is best to use automated testing. Performing the same thing manually would require several people to perform the same activity over a long period of time. It is a waste of time, resources, and money. Automated test cases can be run in parallel from the same machine and produce the same output.
• Automation of the security tests against threats and attacks is very important to prevent security attacks, DDoS attacks, malicious code injections, brute force attacks, vulnerabilities, cross-site scripting, script injections, impersonation attacks, and more. Quite a few of these attacks cannot be simulated manually and always require automation.

Cases for Manual Testing

While test automation is very important for a project, manual testing holds its place in the entire mix.

• The initial rounds of testing done during the verification phase of the first release candidate are done manually. During this phase, the process of testing and the efficacy of the release candidate is established. Only after the initial manual testing is completed successfully, can the automated test cases be executed confidently.
• During the support phase of software, multiple releases of the software are made. While all forms of repetitive testing are performed using automation, there are cases where the software may encounter a P1 bug and require an immediate quick fix. In such cases, while the regression is performed using automation, the fix of the P1 bugs must be tested manually to ensure completeness.
• Testing the usability of software requires observing the behavior of the software, focusing on the ease of use, and assessing user irritation during use of the software. Usability testing is done manually because of the need for human intelligence. With the advancement of the AI/ML (artificial intelligence and machine learning) stream, the analysis of emotion is also becoming automated.
• Random testing of the release candidate is also performed manually. It is also referred to as “Monkey” testing. Random pieces of functionalities are chosen for testing under this process. It unveils the robustness of the software. Since there is no defined path of testing, either in terms of test data or test steps, it is performed manually by testers. Any bugs identified during such a test result in thorough regression testing.
• Sometimes specific scenarios need to be tested manually. For example, in an integration environment, the software needs to interact with a gate system. It involves hardware delay because the gate responds only after reading the RFID tag, which takes a bit of time. Although such scenarios can be configured in automated test scripts, it is always good to test them manually as well to ensure that the process is completed smoothly.

Conclusion

For testing software, neither automation nor manual approaches are going to work individually. The best approach is to prepare a mix of both. Delivery teams should automate as many test cases as possible and use manual testing as a way to randomly check the most critical scenarios. While the initial setup of an automated test suite involves a lot of initial effort, it benefits the entire delivery process in the long run. Automation helps to save time and budget and manual testing helps to give an intermittent push to the quality assurance. Manual testing lays the process down that can be automated subsequently. Both of the testing approaches are complementary to each other.