Katalon Studio 8.1 – Ways to Handle Flaky Tests Smarter

Hello Katalians! Time flies when you’re busy creating awesome products! Back in May, we delivered Katalon Studio 8, which included features to provide better scalability options and a better user experience. 

And because we are not in the business of keeping you waiting for the latest and greatest, today, we are excited to announce the release of Katalon Studio 8.1 (KS8.1). This release comes jam-packed with a broad set of new features as well as dozens of improvements and fixes to continue providing you with a top-notch testing experience. Read on to learn more about what’s included in this release.

What’s New?

Studio 8.1 delivers features that aim to increase the resource and cost-efficiency of your automation cycle, provide better integration with Azure Test Plans, and enhance mechanisms to tackle test flakiness. 

Improved retry failed executions immediately to spot flaky tests 

A flaky test is a test that will pass or fail regardless of the changes done to the underlying code. Test flakiness is caused by issues with new code, external factors like faulty infrastructure setup or network glitches, or an issue with the test itself. The unreliable nature of test results yielded by flaky tests is a never-ending war and it represents one of the most important challenges in the test automation industry because, without accurate results, DevOps teams can’t make accurate decisions about their product releases.

To tackle test flakiness, developers rely on the following steps:

1. Identify the flaky Test Case
2. Troubleshoot the cause
3. Edit and improve the Test Case accordingly

Katalon Studio users have multiple options to tackle test flakiness: they can retry all Test Cases when the Test Suite finishes; retry only failed Test Cases when the Test Suite finishes, or retry a failed Test Case immediately after its first failure. The latter is by far the most popular option among Katalon Studio Enterprise users. 

At Katalon we understand how important it is for developers to be able to make data-driven decisions. In this release of Katalon Studio, we have improved the logic of our reporting mechanisms so they are more intelligent and intuitive to provide better insights on test results. Studio 8.1 groups all executions into one report and makes it much easier to find inconsistent results or false positives. 

Additionally, we have enhanced our test result third-party integrations with tools like TestOps and qTest so the report shows only one final set of results rather than multiple sets of results for each test case. This feature allows users to easily identify failed or passed test results faster.

To learn more about the reporting enhancements included in this release, check out the release notes for Katalon Studio.

Less execution wait-time with Terminate Execution Conditionally 

One of the benefits of test automation is the ability to test early, often, and anywhere in the development cycle. In Studio 8.1 we leveled up these benefits by allowing users to stop Test Cases when they detect an issue early in the cycle by establishing a failure tolerance threshold. 

This new feature allows users to terminate the Test Case or Test Suite Collection execution when there is a significant number of test cases in an execution that has failed, enabling them to provide feedback earlier in the cycle rather than waiting until the test cycle is completed. Failure thresholds will help make more efficient use of testing devices and improve savings on infrastructure costs, especially for cloud users.

To leverage this feature we have included a new command option, -maxFailedTests=<T>,  where T is the maximum number of failed executions allowed before the test is stopped. Once the Test Execution stops, Katalon Studio will generate the report containing all the insights. This feature is only available for tests executed through the CLI or CI using the runtime engine.

To learn more about how to leverage this feature in your testing environment, check out the Katalon Studio 8.1 release notes.

Azure Test Plans integration

Building on the promise to always provide the best integration with your existing testing platform, Studio 8.1 delivers a better way for users to map Studio test results to release views in Azure Test Plans.

In Azure DevOps users have the ability to get a full view of the quality of a release, including passed and failed tests as well as the number of runs. Studio’s enhanced integration with Azure Test Plans includes a new parameter, “Release Definition ID,” which can be used to map test results with releases in Azure DevOps. 

Starting in Studio 8.1, users can enable this feature from the Project Settings menu and add a numerical ID to the “Release Definition ID” parameter. Alternatively, testers can use the new command option “–info-adoReleaseDefID=<DefinitionID>” and based on the specified Definition ID and its stage, pass them to the Test Run properties on Azure DevOps.

Tidbits 

• Enhanced Kobiton integration: Mobile test automation keeps getting better! Starting in Studio 8.1, users will be able to customize the remote server protocol and device name in Kobiton to improve their testing experience. See our documentation on mobile testing to learn more. 
• New built-in keyword for conditional waiting: Studio 8.1 introduces a new keyword “waitForElementNotPresent” for mobile testing, allowing users to set conditional waiting when executing the test script. This feature drastically cuts down on the execution time of the automation suite while also preventing flaky tests. See [Mobile] Wait for Element Not Present for more details.
• New APIs for plug-in development: In this release, Katalon Studio provides a list of new APIs to offer resources for building integration plugins. These APIs are allocated to plugins that have access to the JRE location, Running mode (IDE or Katalon Runtime Engine), Test Suite/Test Suite Collection, and JUnit report location via the plugin platform. 

Wrapping Up

We are thrilled about Studio 8.1 and we hope you are too. We encourage you to download Studio 8.1 to start taking advantage of its many new features and improvements. 

For a complete list of new features, improvements, and fixes, please visit the release notes. As always, please post any questions, ideas or concerns on our community site, we are eager to hear from you.