As a consultant, trainer, and developer who specializes in software quality and automated testing, I frequently get asked for my opinions and advice on how to approach designing a test automation stack.
So today I thought I would prioritize sharing some extracts from my forthcoming guide that states my opinions about what makes a good test automation stack, and how you can approach creating one for yourself.
What Makes a Good Test Automation Stack?
My opinion about individual products and vendors changes with time, just like the products themselves. But what will never change are my four golden rules about test automation, which are:
1. The purpose of test automation is to provide hyper-fast feedback to developers.
“Feedback” means both confirmation of success and notification of a problem. For example, a bug is a type of feedback that signals a problem, typically reported well after the developer thinks their work is complete. Similarly, a test failure is another form of feedback and this happens at coding time.
Why is this so important?
Because of my second golden rule that concerns feedback itself, which states:
2. The longer feedback lives in a system, the more costly it is to deal with it.
Think of a bug that is fixed quickly when the developer that wrote the code still has context versus when the bug is discovered weeks later and fixed by someone that has to relearn the code. Which would be quicker and therefore cheaper to fix?
In addition,
3. The more unresolved feedback that accumulates in a product, the lower the quality of the product itself.
For example, if customers are providing feedback that the product is not serving their needs, and this feedback is captured but not dealt with, then the product is not fulfilling the potential value to its users and is therefore of low quality.
All of the above amounts to my fourth rule, which is:
4. The faster you can detect and deal with feedback, the lower cost of development and the higher the quality of your product.
You can increase quality if you focus your efforts on detecting and dealing with feedback. It’s that simple. Therefore, every decision you make in the design and implementation of a test stack must be in service of optimizing the speed of detecting feedback as that’s the first step to dealing with it.
Who Should Design Your Test Automation Stack?
Any member of your team may contribute to the design and implementation of your test stack. Natural leaders of the effort would typically be the technical architects, a strong full-stack developer or perhaps the leader of the QA team. But most importantly of all,
The developers themselves must own the test stack.
To understand why, think back to the previously mentioned golden rule — “The first priority of test automation is to provide hyper-fast feedback to developers.”
This golden rule applies at every aspect of your delivery cycle, but especially in the local development environment. So for a test automation stack to succeed in practice, it has to complement the workflow of developers and deliver feedback in real time as code is written. Not after, but during.
Too often I see the QA team owning the test automation, but the QA team operates after development and that’s too late! In more advanced teams I see developers relying purely on their continuous integration servers to run their automated tests. While this is lightyears ahead of the QA team owning the test automation, it’s still suboptimal.
Think about it this way: if you’re driving somewhere using a GPS system and you take a wrong turn, would you prefer to be told immediately or would you rather wait until you arrive at the wrong destination?
Protect Quality at Every Step
So how can you build a hyper-fast feedback loop?
You increase feedback at every stage of construction.
And that means eliminating the possibility of defects entering the system at every stage of the software development lifecycle. Let’s walkthrough one example.
To use a real-world metaphor, let’s say we’re building an air-conditioner (AC).
- Clean
The first step is to remove any files generated from previous builds, such as compiled files, and start the build lifecycle from a fresh baseline.
This is like clearing your workbench of all the bits and pieces before you start building something new. - Compile Code
This step takes source code files and turns them into executable code. The product of this step is to create the code for a given component.
This step is the equivalent of producing the components needed to build the AC, such as the fan needed to propel air and the motor that spins the fan. - Run Unit Tests
Unit tests are just another name for component executable specifications that tell us if the components are working as specified. If any components are not, then there’s no reason to continue building the system. Remember that value is created using components, so a faulty component would result in faulty value.
In the AC example, one unit test would make sure the fan blades are able to propel air when turned. If the fan blades don’t work, it’s better to first fix the fan blades and then restart the building process. - Run Integration Tests
Once we know that the components are working individually, we can put them together and check if they are creating value in unison. This is the purpose of integration tests. These tests combine just enough units to create value and, like the previous step, if we find the components are not working together as specified, then there’s no reason to continue building the system since the value is faulty.
Back to the AC example, if we assemble the fan and motor and discover that air is not being propelled due to the spindle not fitting, then there’s no point putting the assembly with the rest of the AC. - Deploy App
If we get to this point and we have great test coverage, it means all the components work individually and in unison to create value. This gives us great confidence to assemble the full system. But we’re still not 100% confident it will work, since something else can go wrong in the end-to-end assembly process. This is why we need to do further testing after we deploy the app.
With the AC metaphor, this stage is about putting it all the components and assemblies together and connecting it to the mains power. - Run End-to-End Tests
With the system fully deployed, we can now make sure it provides the intended value to the consumer. These tests ensure the value specifications are being fulfilled.
Press the on button, turn the temperature control knob to the cold settings, and ensure that cold air is being propelled.
At this point, we have very high confidence that the new code provides the intended value, and we can ship the AC to the customer.
However…
If these steps are done in sequence at the end of each feature (as is usually the case), the developer does not get real feedback until this entire process is complete. And that takes way too long, and therefore it is too expensive.
But with a hyper-fast test stack, you get great confidence while you are actually typing code!
Yes! It is absolutely possible to give developers end-to-end feedback in real-time during development. Read on to find out how.
Match Your Test Stack To Your Tech Stack
Given that testing stacks should closely follow development capabilities, it makes sense to consider your technical stack as a starting point for choosing a test stack.
This article is focused on a Node.js tech stack. I love Node.js because it’s so damn fast to perform tasks like build, test and deploy, and because techniques like running tests when files change are the norm.
If you are not using Node.js
This article should still be useful since the concepts are the same. Just search Google for the equivalent tooling for your tech stack, and let me know what you find either in the comments below or on Twitter.
The Best Products for a Hyper-Fast Test Stack
With reference to the process defined above, here is a list of my recommendations for testing quality at each of those major steps.
For Real-Time Unit Tests Locally
I recommend Wallaby.js
Wallaby.js tells you if ANY test passes or fails in real-time, on every keypress!
With every single key I press, Wallaby lets me know if something broke or passed. It employs a very clever technique to know exactly which tests to run, and it runs them all in parallel. The result will make you say “wow!”
A video paints a million words (from the Wallaby.js site):
More info: https://wallabyjs.com
For Real-Time Integration Tests Locally
I recommend Chimp.js
Chimp validates your business domain logic every time you save a file.
I use Chimp’s domain testing mode. It’s a tool that we wrote at my company that watches the file system and reruns the service tests with every change. The result is fast feedback on the state of the domain logic for Node.js apps.
More info: http://chimpjs.com
For Real-Time End-To-End Tests Locally
I also recommend Chimp.js
Chimp validates the end-to-end feature you’re currently working on every time you save a file.
I use Chimp’s end-to-end testing mode. Instead of running all the end-to-end tests on every file save (which would take way too long), Chimp watches for a specific tag and reruns the tagged tests. This simple trick keeps you very focused on the task at hand while providing super fast end-to-end feedback.
Chimp also works for any web application — no matter the backend.
More info: http://chimpjs.com
For Orchestration
I recommend Gulp.js
Gulp is the conductor of the orchestra.
Getting all the pieces to seamlessly work together can be a challenge and you need a tool that can work as fast as you. This is where Gulp.js comes in.
Gulp.js is a streaming build system that’s great for orchestration. I use it to run complex tasks locally — for example: start the app server, wait for a message, and then start Chimp. On the continuous integration server, it runs unit tests using Mocha & Karma, collects code coverage and runs smoke tests.
You can also use Gulp with non Node.js technologies also. I have recently used Gulp to control Maven tasks for a Java project because of Gulp’s superior file-watching capability.
More info: http://gulpjs.com
In my next post
I’ll be showing you how you can expand the fast feedback philosophy created in the dev environment to the wider team, and I’ll also show you how to create a holistic quality process that is built for faster feedback at every step.
Agree or Disagree?
If you disagree with my recommendations of tools, or know of a different tools or techniques that you think is even better at hyper-fast feedback, I always want to hear about it! So please do let me know via the comments box here, or by connecting on twitter at @sam_hatoum
And if you like what you read, please help share this with others by clicking the recommend button.
Thanks, and I look forward to helping you deliver higher quality software, faster.
Sam.
Let me know if you have any questions or thoughts in the comments below.
