The Ultimate Guide To Continuous Testing


"Agility basically facilitates competitiveness. To compete in today's environment, you must act and react fast, otherwise your competition will simply beat you to it. Today, the barrier to compete is minimal, and the only way to defend one's stature is by innovating in short iterations and basically meaning adopting Agile."

Alon Girmonsky, CEO, BlazeMeter

When you shift testing left, among many other benefits, you can achieve continuous testing, executing automated checks as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate. Continuous testing also involves early and frequent testing, proper code management, and adjusting the various quality-related tasks over time to maximize the efficiency of the whole process.

Some of the key benefits of continuous testing include:

  • Lower the cost of development
  • Reduce waste
  • Reduce risk upon release
  • Increase system reliability
  • Release to production faster
  • Compete more fiercely in the marketplace
Shift left testing

Many teams today are trying to build or refine their continuous testing machine. We believe that in order to improve the results of software production, it’s necessary to consider three fundamental pillars that are closely linked together: processes, tools and people. Teams can improve by bettering their processes, but the tools and the team members must also adapt in order for the new and improved processes to stick.

Processes,tools and people

For example, within Agile, teams may find themselves in frameworks like Scrum that create a process, for which they will find themselves using various types of tools for everything from communication to task management and there will typically be great focus on the motivation and commitment of the team.

All aspects of Agile approaches are well designed to be adaptable to change, which is its main focus. The idea is to assume that what the customer needs is not fixed nor established in a contract. Thus, it is essential to work on constant adaptations in a way that does not cause the project costs to skyrocket or become unmanageable.

What is needed to adapt to change and yet maintain a high level of quality?

For any team, the most typical problem that arises when introducing changes in a system is fear: fear of breaking something that was already working before and not realizing it until after it has reached the customer's hands. When a user is given a new version of the application, there is nothing worse than finding that what always used to work no longer does.

To address this problem, teams must be able to detect any errors of any kind as soon as possible. When speaking of errors, we are referring to any kind of error or bug: a miscalculation, performance problems, security vulnerability, aspects of usability, maintainability problems (e.g., code quality factors) or anything that may inconvenience a user. And what we mean by “as soon as possible” is as soon after when the error was inserted into the system as possible.

Thus, important practices like continuous testing, which goes hand in hand with continuous integration (CI), emerge and gain importance for Agile teams.

Basically, continuous integration proposes building and testing the solution frequently, building a potentially customer-shippable product with a frequency that depends on each team, varying from one new version after each commit to one every day or week. Continuous integration tools allow for defining a sequence of tasks to be performed automatically, among which typically include:

  • Updating code to the latest version
  • Building the solution
  • Automated checks
  • Code quality checks
  • Security checks

And the list can go on with hundreds of integrations and tests that one may wish to add.

There are dozens of tools to support these automated systems, some as old as Ant, Maven and MSbuild, configuration management tools as Chef, Puppet or Ansible, to popular CI tools such as Jenkins, CruiseControl, Bamboo and TeamCity. And if that were not enough, there are also cloud solutions like Amazon Pipeline Code, TravisCI, CircleCI, Codeship and Microsoft’s Visual Studio Team Services (to name a few).

So, all one has to do to have continuous testing and continuous integration is simply implement one of these CI tools and then quality will be baked into everything. Right?


The problem is that many teams fall into this very fallacy and overlook many preconditions for continuous testing and CI that must be met beforehand, resulting in failure. Some of the preconditions include the proper handling of code version management, incident management, data and environment management, automated checks at different testing levels (unit, build tests, API, UI), performance tests, internal code quality and so on.

Fortunately, this guide will help you navigate through several of the hoops your test team has to jump through on your path to continuous testing (the highest level of testing maturity according to our maturity model) all the while igniting the shift from traditional quality assurance to quality engineering.

Download the Ultimate Guide to Continuous Testing as a PDF

Everything teams need to know to shift-left testing and reach the level of maturity in their quality engineering processes to enable CI/CD.