Over the past few months, we have slowly rolled out and enhanced our AI Visual Testing solution during the trial period. This month we are proud to announce the general availability (GA) of Katalon AI Visual Testing that includes additional enhancements.

The following reviews all the combined AI Visual Testing features in the GA.

The Need for Visual Testing

Modern businesses depend on digital experiences for both customers and business users. Modern test automation helps to validate functionality within these digital experiences, but validating the UI/UX continues to be an intense process of:

Manually, or through automation, capturing images
Manually, or through partial automation, comparing for differences

While tedious and time-consuming, failure to include visual testing as part of your quality processes could lead to:

Bad first impressions due to variations or broken UI layouts, such as buttons and/or text that are accidentally displaced
Loss of sales given limited or no usable UI Interactions due to non-visible elements on pages that functional automation tests cannot assert
Frustration or anger triggered by improper use of language that somehow slipped through various review rounds, etc.

All of these are actual problems that have impacted businesses in real life, resulting in thousands or millions of dollars in lost revenue, customer dissatisfaction, and negative brand image.

Visual Testing and AI

Automated functional tests are helpful in ensuring software behaves as expected with any recent code changes, but they cannot detect visual changes to the application such as layout, colors, fonts, misplacement of elements, unexpected visual overlays, etc. Visual verifications can be done with the naked eye, but the effort needed will quickly swamp the available capacity of all teams, considering the typical large count of pages and elements that modern applications contain. This method of manual visual testing is also error-prone since there is a high probability to miss details during mundane, repetitive tasks.

Automated Visual testing is an alternative tool that enables teams to create visual regression tests with minimal effort. Basic pixel-to-pixel image comparisons can be automated to help filter through the hundreds or thousands of images that need to be manually compared. Unfortunately, this often still leaves hundreds of images that fail since pixel-based testing can generate failed tests even when the visual difference isn’t noticeable by the user.

Katalon has offered pixel-based visual testing for a long time now and has listened to customers' feedback to create a new visual testing process that leverages AI to add an additional automated layer of image comparisons to further filter out false positives on visual testing. Its seamless integration with the Katalon platform means that users can start using it out of the box without having to worry about complex setups or third-party configurations.

AI Visual Testing

AI Visual Testing has arrived in TestOps. Katalon’s new AI expands on our existing pixel comparison visual testing with two additional AI-enabled visual tests: content-based and layout-based. These visual testing options greatly improve the speed and efficacy of UI quality testing. They allow software quality teams to automatically analyze, differentiate, and identify valid visual UI differences from minor visual infractions, reducing manual visual validation efforts by hundreds or thousands of hours per year.

Layout-Based AI Visual Testing

Instead of simply identifying when individual pixels are not the same, layout-based AI Visual Testing analyzes the layout of images and visual objects in relation to each other and compares those relative differences between baseline and test results. When the relative differences are significant, the test fails. However, when an image, object, or line of the text differs by only a few pixels or lines, where a standard visual test would flag a failure, AI Visual Testing intelligently deduces that there is only a minor difference in layout and passes the test. This strategy significantly reduces false positives and wasted manual review effort.

Content-Based AI Visual Testing Katalon Platform

Content-Based AI Visual Testing

When visual quality is rated by the completeness and accuracy of text content rather than the layout and existence of images and UI elements, content-based AI Visual Testing is the best option. Content-based AI Visual Testing prioritizes the review and validation of text within a UI regardless of font, layout, color, or changes in its background. Like the layout-based comparison, this also increases the efficacy of content testing and reduces manual validation efforts by hundreds or thousands of hours per year.

Content-Based AI Visual Testing

Support Multiple Visual Baseline Collections

Different test cases/test suites cover different parts of the application, hence they need different sets of baseline images during visual testing. The users can now choose to use different baselines when scheduling a test run.

Support Multiple Visual Baseline Collections | Katalon Platform

You may view baseline collections as well as their images via the “Visual Testing” tab.

Visual baseline collections | Katalon Platform

Visual Baseline Collection

Ignore Zones in Visual Tests

Many UIs now contain dynamic content such as ads that make visual testing more complicated. Katalon’s Visual Testing now lets you create ignored zones within baseline images to prevent differences in these dynamically generated UI elements from triggering failed tests. This is a powerful new feature that addresses one of the main challenges of visual testing.

[NEW] – Apply/Delete Ignored Zone(s) for All Images in a Collection

For use cases where the area(s) that need to be ignored exist in all the baseline images, this release adds the ability to apply and delete ignored zones to all baseline images in a collection. This offers a big-quality-of-life improvement for testers that experience this use case often.

[NEW] - Specify Baseline Collection ID as a Parameter in Katalon Command

For users executing visual tests directly from the CI/CD pipeline using Katalon Command, a new parameter called testOps.baselineCollectionId has been added to the command line, which triggers the visual test run. Users can either input an existing baseline collection ID to the parameter or input -1 to the parameter to generate a new baseline collection.

TestOps Enhancements

Incomplete Status in Test Runs

Prior to this release, Test Run status in TestOps could only be Passed or Failed. This was inaccurate for test runs that were canceled or those that were still running or importing. Now test runs that have been canceled or are still running are placed in the Incomplete status.

Incomplete Status in Test Runs | Katalon Platform