Stop Test Suite Execution with ‘X’ Failed Tests | Fail-fast Principle

The Fail-fast Principle in Software Testing

In software development, a fail-fast system is a tool that will notify when a failure is likely to occur. The philosophy of the fail-fast principle is formed around the idea that a system should stop normal operations rather than trying to continue working around a flawed piece of code. 

The fail-fast principle makes software bugs easier to detect earlier in the development cycle. It allows for faster strengthening of the code under development, limits the number of bugs that make it into production, and reduces the intrinsic cost of letting bugs in the code.

In short, it is better to fail fast, fail often in the development than later when the product is about to be released.

One of the benefits of test automation is the ability to test early, often, and anywhere in the development cycle. Katalon Studio 8.1 delivers a vital feature that aims to increase the operations and cost-efficiency of your automation cycle. Leveraging the fail-fast principle,  Studio 8.1 provides users with the ability to stop Test Suite Executions when a particular failure threshold is reached. 

This new feature will help make more efficient use of testing devices and improve infrastructure costs, especially for cloud users.

Fail Fast, Fail Often in Katalon Studio – Terminate Execution Conditionally

Failure thresholds in KS 8.1 allow users to terminate the Test Case or Test Suite Collection execution when there is a significant number of test cases in an execution that has failed, enabling them to provide feedback earlier in the cycle rather than waiting until the test cycle is completed.

To leverage this feature, we have included a new command option, -maxFailedTests=T,  where T is the maximum number of failed executions allowed before the test is stopped. Once the Test Execution stops, Katalon Studio will generate the report containing all the insights. This feature is only available for tests executed through the CLI or CI using the runtime engine.

First, find the option to set the threshold within the Generate Command for Console Mode dialog box –  set the level of failure tolerance by ticking the box “Terminate the execution once the total number of test failures reaches this threshold” and typing a number in the text field.

Generate the CLI command, which in this case will include the -maxFailedTests= parameter in it.

Run the execution and once it reaches the threshold, there will be a message in the CLI indicating that it had stopped following the rules. 

You can also take a look at the summary of passed and failed tests generated by Studio in the Test Cases Table within the reports:

To learn more about how to leverage this feature in your testing environment, check out the Katalon Studio 8.1 release notes.

Access Pass/Fail Trends with TestOps

After selecting your failure threshold, connect to TestOps, Studio’s built-in test reporting and orchestration platform, to examine your pass/fail patterns.

Step 1. Set up an Organization and Project

First, log in to analytics.katalon.com with your Katalon account.

Step 2. Integrate with TestOps in Katalon Studio

Open Katalon Studio, go to Project > Settings (similar for both Mac and Windows).

• Click on Katalon TestOps > check the Override authentication box > choose your Organization (if you cannot find your Organization, click Fetch Organization)
  
• Insert your Katalon email and password

Click on Enable Katalon TestOps Integration > choose your Team > Project > Apply and close.

Step 3. Push and view Katalon Studio Test Suite Results on TestOps

Note: TestOps will automatically collect all of your test runs by Test Suite. Assemble your Test Cases into a distinct Test Suite to ensure that executions are thoroughly documented. In the left menu, navigate to TestOps > Test Runs > select a Test Run.

Having been navigated to TestOps, you can view the pass/fail trends. 

Wrapping Up

We are thrilled about Studio 8.1, and we hope you are too. We encourage you to download Studio 8.1 to start taking advantage of its many new features and improvements. 

For a complete list of new features, improvements, and fixes, please visit the release notes. As always, please post any questions, ideas, or concerns on our community site, we are eager to hear from you.