Global
At Ness Digital Engineering, we partner with a wide range of organizations to deliver software as a product over many releases, spanning a decade in some cases. As software eats the world, organizations realize it has to be treated as a product and leverage companies such as ours to improve release cadence, test coverage and generally adopt software engineering best practices.
The initial situation when the relationship debuts varies wildly. Some have a clear backlog of tasks to be addressed while their teams focus on advanced aspects of their product for example. In that case, we would leverage a thorough knowledge transfer to get our teams quickly up and running alongside their engineers. Other cases focus on the introduction of practices to improve release cadence. This includes test automation. One mental model that helps assess the situation is Mike Cohn’s test pyramid.
The pyramid model for test automation (Mike Cohn) recommends adoption of three layers of tests for the right automation mix. Unit tests at the base of the pyramid, take up most of the automation testing work, followed by the middle layer consisting of integration or service level tests. The uppermost layer is the thinnest comprising of GUI tests. All of these are targets for automation. Cohn’s model is ideal but still helpful as a shared model of understanding. The recent advances in container technology have significantly lowered the cost of setting up and running integration tests. This is reflected by alternative shapes proposed e.g. by Simon Brown or Spotify’s Honeycomb which are more generous in proportion on integration tests relative to the base.
What we see in practice discussing with a wide range of enterprises, is that these idealized proportions are not known or not clearly defined. Introduction of test automation is challenging and some organizations did not resist the temptation of introducing end-to-end GUI automation testing at the detriment of the other test types. While authors may disagree on the relative proportions of the bottom two groups, the upper category is preferably kept smallest.
Systemic view
When facing a situation of too many automated GUI tests, which become hard to manage, are brittle and ultimately do not contribute to fast release cycles, we can be reminded of a known pattern in systems thinking: Shifting the Burden (Peter Senghe).
We have one problem (too many GUI tests), two approaches (create unit and/or microservice integration tests OR create GUI tests at the bottom, respectively top of the pyramid), and one adverse side-effect leading to a tipped test pyramid. This archetype identified in Senghe’s book, “The Fifth Discipline” combines two balancing loops and one reinforcing loop as depicted below. These loops conspire to move the system in the wrong direction.
The mandate of introducing practices to straighten test type proportions for automation requires a structured and methodical approach. In addition, such endeavors only succeed with a management mandate and understanding that changes become visible after long periods for reasons shown next.
Assessment of the current state
In this first step, we consider various aspects such as the number of tests that exist in each layer, their stability and reliability, coverage for critical scenarios in the acceptance layer, and what tooling and organizational support exists for the system until sufficient tests are created in order to assess current state.
Definition of a test strategy
Here, our experts consider the technical and organizational needs, the different types of tests needed and their mapping to the continuous delivery pipeline stages. They assess the tools and framework to adopt, as well as ways to empower teams (incl. product owners) in ways to configure the test mix and frequency of test execution at various stages in the pipeline.
CI/CD strategy
The focus lies on identification of tools and infrastructure supporting continuous integration (CI) and continuous delivery (CD) pipelines. These will need to support a requisite frequency for the execution of different tests. For instance, unit tests can run with every build, integration and acceptance tests at different, regular intervals.
Systematic test base increase
The core of the work then consists in increasing the test base while taking changes to legacy code as well as new feature development into consideration. Teams need to discuss how and what they would cover with each new feature coming into their sprints. Coverage for legacy code will take longer as the test base builds up gradually, as and when the code is modified or refactored. Adoption of Test-Driven Development (TDD) will be worthwhile to consider here.
Test auditing
Regular assessments of aspects related to test duplication, code coverage and deprecation of unwanted tests will prevent test rot revert to the initial state of pyramid. The frequency and scope for the reviews is defined and adapted for every client context.
Culture shift
Scripting tests needs to be treated as coding functionality and requires a culture and mindset aligned to this approach. Having the right team that works alongside developers to automate tests is key to achieving success. Mentoring and thought leadership, with management buy-in, are key to distil this mindset and shape the culture of the organization.
Support from management is essential. The initial period likely produces some slowdown in the velocity of deliverables, possibly coupled with investment needs in tooling. Once the slump passed, we see consistent results. Automation and the right balance in test types yields higher test coverage, lower maintenance costs, shorter feedback loops, improved pipelines, faster release cycles and the entire teams overall commitment to continuous improvements.
