When to set up automated checks
In a previous assignment I worked with test in the form of creating unit test scripts for an automated regression suite. It provided me with lot of learning opportunities about the product as well as the automation process, but left me with a feeling of not doing very much testing even though I worked hard and produced several scripts that where used as intended.
I have worked more with automation since then and read about it in articles and books to get to grips with its benefits and how to do it well. While reading a couple of interviews at uTest the other day some pieces began to fall into place.
In answer to what the greatest threats and opportunities for the testing craft are Michael Bolton talks about the risks of conflating testing and checking. So what are the differences?
To sum up Michaels definition from a posting on his own blog Michael states that: "Checking is something that we do with the motivation of confirming existing beliefs. Checking is a process of confirmation, verification, and validation. When we already believe something to be true, we verify our belief by checking." ... "Checking is focused on making sure that the program doesn’t fail."
"Testing is something that we do with the motivation of finding new information. Testing is a process of exploration, discovery, investigation, and learning." ... "testing is focused on “learning sufficiently everything that matters about how the program works and about how it might not work.” "
Reading this definition the testing craft instantly became more exciting to me while it also helps to pin point the value of checking (often seen in automated regression suites). When you feel that you don't add value in the way you intended to, it's easy to miss the ways that you do. Using the context-driven approach to testing it is possible to break down the scripting I did into testing (learning about the product, the interfaces, expected results, analysing the actual result etc) and checking (running a static set of instructions over and over). The value of both now becomes more apparent and is more easily evaluated.
Kaner et. al. puts forth a rule of thumb in the introduction to chapter 5, Automating testing, of "Lessons learned in software testing":
"Use automation when it advances the mission of testing. Evaluate your success at automation in terms of the extent to which it has helped you achieve your mission"
I won't try to analyse my scripting assignment, since I never had much of an overview of the total testing at the company in question, but in upcoming assignments I will now have an additional insight as well as additional questions to ask in order to evaluate the work I and my collegues are performing.
James Bach has some additional thoughts on when to set up automated checks (often called automated tests) in an interview at uTest:
"- Is the product highly controllable and observable? A command line tool that provides its output solely to the console window is inexpensive to automate, compared to an iPod touchscreen app. I want to get under the GUI.
- How expensive is the tool I’m using? I urge you not to use expensive tools, even if they work. Never let your manager buy them. Because expensive tools become something you MUST use, even if they don’t work. A free tool may be freely abandoned. This gives you flexibility.
- How well can I automate the oracle? Will the bugs be able to elude my automation because it can’t tell if a complex graphic is rendered correctly?
- What is the learning and testing value I’m giving up by using automated checks? I find that doing a test multiple times also causes me to learn and see new things in the product. Furthermore, when I re-run tests, I often run them in a different way, and that allows me to find new bugs.
- Can the automated check be parameterized and randomized, so that I get lots of similar checks for very little additional investment? I like automation more for data intensive testing, because I get new tests just by changing the database.
- Is the technology “Pyramid shaped?” In some products lot of underlying code boils up to one simple output, by placing checks on that output, we may be able to find lots of bugs. In other products, there are many different pathways, and you need a lot more checks to get decent coverage.
- How critical are the checks to the business? Is this critical functionality? Is it a common usage scenario? There are candidates for smoke testing.
- Is this part of the product especially prone to breaking? If so, that may be good for automation, UNLESS, it breaks in a way that breaks the automation.
- When I automate, I do it incrementally, in small bits.
I want automated checks for high value, highly testable parts of the product, and I want to do them in such a way as they aren’t constantly breaking or giving me false readings. I want to augment those checks periodic sapient testing as a cross-check."
In retrospect I would say that the work I did was check-automation driven by testing and check-execution evaluated with testing techniques. From this point of view it's much easier to evaluate the value of both the testing and the checking efforts.
Definitions are mighty tools, just a few lines of text can sort out years of work and improve your skills. Cool...
