How to Design and Implement a Top-Notch Test Automation Strategy

Every new software, method, or tool comes with certain growing pains and takes some time to get used to, although it’s almost always worth the adoption effort, a major part of which is testing. 

Test automation dramatically improves your processes, saves you time and resources, and ultimately leads to higher-quality software. But you can’t just jump into it and expect the automation to produce the results you want. You need a clear strategy in place if you want your releases to go smoothly. 

This article explains how to design and implement a top-notch test automation strategy so that your testing efforts are fully worth it and your software quality reflects a well-thought-out plan that enables you to fully optimize your product before your customers start to use it. 

I Don’t Have a Strategy—What Can Go Wrong?

First off, let’s discuss what happens if you don’t have a test automation strategy. Not having a test automation strategy is something that’s been, er, tested many times already in the real world, so we have a pretty good idea of what happens without test automation strategies in place. 

1. Inability to show business value

When teams look to implement a new test automation or other solution, such as performance testing or browser-based testing, quite often they don’t bother to take the time up front to examine the business reasons for doing it. Yes—the tech is cool. Yes—it’s something that will potentially help the business. But if you don’t take time to tie that into real business value, you run the very high risk of the project getting canceled—or not even approved in the first place—for lack of demonstrating ROI or potential ROI. 

2. Lack of vision

Without a plan of action, it’s very tough to have a vision. Automation projects tend to pivot. You may need to bring in a new application to automate as part of your framework, or potentially even switch the framework technology itself. If you don’t know what your original vision was, it’s very easy to end up postponing or scrapping the project when faced with a glitch. With a vision, you have a documented way to assess the bigger pivoting questions that come up and act on them accordingly. 

3. Technology efficiency loss

Without a clear test automation strategy, you also run the risk of choosing the wrong testing automation technology for your project, in what we call “technology efficiency loss”. Your test automation technology needs to match the application you’re building. If you don’t take the time to write that down and have a strategy around that, you will likely wind up trying to shoehorn technology into a solution for which it shouldn’t be used. 

4. Being unprepared for the “testing squeeze”

Agile development processes were supposed to have done away with testing squeezes by now, but we all know that hasn’t happened. Testing squeezes are still very much a reality of software development, and given that they’re still happening, you need a test automation strategy in place to decide on what to cut first. Which scripts should you cut? What was the most important thing to test? What was tied to business value? Without understanding that or having something like that in the strategy, you’re left scrambling to decide what to cut and where to look, and often you will just end up cutting everything—removing it all to worry about it later. 

What is a Test Automation Strategy?

Now that you know you need a test automation strategy, let’s define what a test automation strategy actually is. 

Simply put, a test automation strategy is a microcosm of your larger testing strategy. A lot of the same techniques you followed to develop and build your overall testing strategy will be the same for your testing automation strategy. Accordingly, your test automation strategy should sit on the same level as your other system and performance testing because you’re using a lot of the same data points to drive toward the answer of what to automate, how to automate it, and which technology makes the most sense to use. 

In short, a test automation strategy lives inside your larger testing strategy and uses the same kinds of processes and tools to determine who you’re testing for, what the users do, what the testers do, what the developers are doing, and all of the associated metrics. 

Download Now: Getting Testing Done On-Time | 15 Questions for Agile QA Leads

What’s the Purpose of a Test Automation Strategy?

You may still be wondering about the true purpose of a test automation strategy. What’s the goal?

First and foremost it’s to inform on the risk, capabilities, and functionality and arrive at a reliable, repeatable process of informing on those things. 

Second, it’s a way to communicate your goals and plans. 

It’s also a discussion starter that can sometimes be a springboard for a new proof-of-concept or a new technology to bring into your company. 

Finally, it’s an auditing tool that allows you to go back and look at what you planned to do and compare it against what was actually done. 

What Does a Test Automation Strategy Look Like?

Don’t worry about format. It doesn’t need to be a 70-page Word doc. It can be a living document such as a mind map. In fact, mind maps work quite well for test automation strategies.  You can start with ideas or rough sketches as a way to lead into the bigger test automation strategy document that you build out over time. But again, format should be the least of your concerns. 

( Maybe you want to take the Tester’s Checklist too!)

Steps to Start Creating a Test Automation Strategy

Now that you understand the value, purpose, and look of a test automation strategy, let’s dive into the actual steps to get started on one. 

Step 1: Define your high business value tests

The first step is to define what’s most important to test by defining what we call the “high business value tests” – ie, the flows that could potentially cause the business to fail if they stopped working. Consider Knight Capital Group, the trading company that went from fully functional to belly up in just 45 minutes and lost $485 million due to the failure of a single software book (ie, a high business value test that wasn’t considered). Be sure to work with your business to understand what your high business value tests are because it will allow you to understand if the solution you’re proposing fits your critical scenarios. It will also help you demonstrate true business value with your automation framework, which will dovetail nicely into the ROI conversations. 

Step 2: Identify your risk

A key part of any test automation strategy is knowing what to test first and what to test last. You should use a risk-based approach to determine this testing automation priority. You can determine the risk, or priority, of each thing you want to test by figuring out its business impact and adding that to the probability of it failing. Obviously, the things with the highest business impact and highest probability of failure should be highest on your priority list, while the things with the lowest business impact and lowest probability of failure should be at the bottom. This will also help quite a lot with the aforementioned testing squeeze and knowing what you want to cut first. 

Step 3: Understand your technology, tools, and resources

You need to understand how your overall testing automation solution will affect your overall environment. Do you have the proper accounts to run this? Do you have the proper environmental access? Do you have the right libraries and APIs and other pieces you may need to have your testing automation solution talk to your applications? You need to have a robust working solution that you can easily build into your overall framework without bogging anything down or creating broken or fragile tests. 

Recommended: How to select Automation Testing Tools?

Step 4: Make sure your data is good

A lot of test automation projects fail due to data failures. What if you could ensure the data is correct right at the start using another script in your automation framework or by running pre-scripts to validate or load the data, thereby saving you hours and hours of rewriting or redoing your tests? For each release and big framework iteration, you should deeply examine how you’re handling your data, how you’re storing your data, where your data is coming from, what your retry logic is, and if you have to worry about masking or de-identifying data. 

Step 5: Define your DevSecOps

A lot of testers can now work directly with Jenkins servers or other build-and-deploy tools, so you need to define that in your testing automation strategy. 

You need to ask:

• Where is the code stored? 
• How are you deploying it?
• What environments are you running it on and are they safe?
• Are the libraries and open source code you’re using secure?

You need to be doing some level of security scanning and have a process for how that scanning is done for your test automation framework. 

Step 6: Consider your testing environment

There are a lot of things to document around environmental conditions. Do you need certain tokens or VPNs? Do you need a launch box? How does that work and where does that live and who is responsible for it? How is the patching done on those systems? You need to document all of this, as it will also help greatly with onboarding new testers to your organization and getting the logins set up. In short, you need to understand where your code is running at all times and have it fully documented. 

 Step 7: Tag your tests

Having the ability to tag your tests and group them logically will allow you to say, “It doesn’t matter whether we have 20,000 or 50,000 test automation scripts because I know these are for my check-out, these are for my login, these are for smoke tests, etc…” If you don’t have those tags in place, you may be left doing a lot of groundwork trying to figure out the purpose of certain tests and figuring out which tests to run. Set a tagging agreement right up front to ensure consistent tagging and regular updates of the most commonly used tags. 

Step 8: Look for testing efficiencies

As you do more and more testing, you can start to apply the same testing logic in different areas to create efficiencies and shave off testing time and resources. For example, if you’re doing the same thing with the unit testing, manual testing, and automation testing, you probably don’t need three people running the exact same test. You can save a lot of time by fully understanding your overall testing automation strategy, all the way from the unit test down to the UI test. 

Step 9: Embrace agile tools 

Embrace your agile and DevOps tools and really work on your documentation. Your testing automation strategy should become a living document that’s updated and reviewed each sprint to make sure you’re sticking to your visions and goals. Embrace this process and use cloud-based tools such as GitHub to make it a success. That said, you don’t need to document everything. You should, however, have regular check-ins and micro-strategy sessions. 

[Video] Katalon x PNSQC: Test Automation Strategy: Insider Practices & Secrets  here

Tying it all Together

Remember: the main thing you want to accomplish with your test automation strategy is to focus on your goals and communication and not your format. How will you deliver value? What is your overall technology goal? How is it affecting the people in your organization and your environments?

Also—be sure to win the support of your business partners, product owners, and project managers, because they’re the ones who are going to support you when the testing squeeze comes. As long as they understand your strategy, they’ll trust you to make the decisions on what needs to be cut and what doesn’t. 

Finally, use your testing automation strategy as a basis for investing in the right technologies and growing your business through innovation—because, after all, that’s what this is all about, right? Growth, innovation, and not having to go in reverse.

Read more: Automation Testing 101