Connect the Dots: How Visual Testing Enhances Cross-team Communication
View image | gettyimages.com
Quality Assurance (QA) is one of the integral parts of software development process. Every component and piece of a software module has to pass through quality assurance before they are deployed. It is the duty of the QA team to ask right questions, and communicate issues and bugs to developers and product owners, so that they can be fixed quickly. However, there are diverse people with different skill sets and roles in a project team that present communication challenges to QA, most notably coding abilities. Having screenshots and screenshots analysis tools in place help to overcome the communication challenges and make the process effective and efficient.
Communication is crucial to reinforce effectiveness. As a picture tells a thousand words, explanatory screenshot tells a million. Screenshots help people learn, they grab attention and explain tough concepts in simpler manner. Embedding a screenshot analysis tool for quality assurance in a development process make it easier for QA team to communicate issues with the project team and stakeholders; developers, managers, and even support, marketing, and sales engineering. This helps to make the process seamless by reducing the communication friction and improving work efficiency.
One Screenshot is Worth a Thousand Assertions
No wonder that increasing number of companies have started to utilize visual testing tools, not only to enhance QA and CI-CD processes, but also to improve internal communication.
Instead using complex error logs, advanced visual testing tools, such as Applitools, offer stakeholders simple image comparisons that pinpoints them to the issues that require attention, helping to cut down on miscommunication as well as streamline operations for reporting bugs and other issues related to quality assurance.
Since explanatory screenshots are quite efficient in describing all sorts of issues in application during development process, visual testing tools can eliminate to and fro email chains that are time consuming and confusing.
In addition, another major benefit to using automated visual testing solutions, such as Applitools, over manual and traditional bug reporting system, is the additional background information provided by it. Along with annotated screenshots, one has ability to access operating system and browser specifics for the individual clients. Information like this is important for quality assurance team members to reproduce issues on their own systems as required.
Cloud-based solutions also enable any team member to access the visual testing dashboard - anywhere, any time - this instant collaboration capability substantially reduces response time across the board.
Bottom Line
Visual testing tools provide competitive advantage to companies, by improving internal and external communications and contributing to a smoother development process. With tools that assist QA team in communicating effectively with the rest of the R&D team, the company can increase team cohesion, motivate, improve confidence, and in the meanwhile scale up companies’ products, time-to-market and overall quality to the next level.
How To Handle Visual Testing False Positives, Part 2
Special guest post by Dave Haeffner - author of The Selenium Guidebook
The Problem
In the last write-up I covered common issues that can lead to false positives in your automated visual tests along with some workarounds for them.
While this was a good start, it was incomplete.
It provided enough information to add initial resiliency to your visual tests, but it glossed over other common scenarios that will cause false positives and create gaps in your test coverage. Not only that, the tactics I’ve demonstrated thus far won’t hold up when faced with these new scenarios.
A Solution
These are the sorts of challenges that give visual testing a bad rap. But by leveraging a more sophisticated image comparison engine we can save ourselves a lot of time and frustration by side-stepping these issues entirely.
But before we get too far ahead of ourselves, let’s step through these additional scenarios to make sure we know what we’re up against.
An Example
In the last write-up we stepped through how to work with small pixel shifts (e.g., an element moving 1 pixel left or right on subsequent page loads) by increasing the mismatch tolerance (the value that WebdriverCSS uses to determine if the page has changed significantly).
NOTE: For a full walk-through you can read the previous post here.
To briefly recap — this code loads an instance of Firefox, visits the example application, captures a baseline image, then refreshes the page, captures another image, and checks to see if there is a significant change between the baseline image and the second image.
For this example, our test code works well. It passes when the menu button shifts by a pixel (avoiding a false positive), and it fails when it shifts by 20 pixels (a legitimate failure). But when the same approach is applied to a slightly different example (e.g., an image shifting by a single pixel) the test will get tripped up.
Let’s change the URL to point to this new image example and see for ourselves.
If we save this file and run it (e.g., node false_positives.js from the command-line) it will fail when it should have passed. Here’s the diff image from the failure:
Compared to a text menu button, when an image moves, it accounts for a greater number of pixels on the page. So when the image in this example moves a single pixel from it’s original location, it’s enough to trigger the mismatch tolerance (which causes the test to fail). Simply put, the failure occurred because more pixels changed, even though it was just a single pixel to the left or right.
And it’s not that the image we’re testing is overly detailed. The same thing will occur with a simpler image.
Conventional wisdom posits that we could simply increase the mismatch tolerance even higher. This is a bad idea and should be avoided since it can quickly make your automated visual tests go off the rails by opening up holes in your coverage that you’re not aware of.
Another Example
And that’s not to say that we don’t already have holes in our coverage.
In addition to tricky false positives, we also run the risk of missing legitimate failures. For example, let’s point our test at an example that has an occasional typo and see if we can catch it.
And just to be certain, let’s also set the mismatch tolerance back to the it’s original default (reverting it from 0.20 to 0.05) — making the visual comparison stricter.
If we save the file and run it (e.g., node false_positives.js from the command-line) it will run and pass. But it should have failed. It missed the typo entirely.
Now, granted, if the typo occurred elsewhere (e.g., at the beginning of the sentence) the test may catch it. But not if the typo were composed of a character of similar pixel width to the original character.
Missing typos may seem like a contrived example, regardless it’s a hole in our visual test coverage. And if it can slip through the cracks, then something else could too.
A Better Solution
With a more sophisticated image comparison engine like Applitools Eyes we can easily avoid these issues since it doesn't rely on thresholds like our previous examples. It actually ignores changes that are not visible to the human eye, but still identifies valid changes (no matter how tiny) in pages of all sizes. And thankfully WebdriverCSS comes with support baked-in.
NOTE: For more info on Applitools Eyes, check out their Quick Start Guide and take a spin for yourself. They have SDKs for a bunch of different languages and are constantly adding to that list.
You just need to create a free Applitools Eyes account (no credit-card required) and grab your API key from your account dashboard. Once you have that, we just need to update the test setup for WebdriverCSS (removing the existing WebdriverCSS init configuration, replacing it with your key).
After that, there are a couple additional tweaks to make.
Previously we just used 'body' for all of the test parameters. But now we want the test to be more descriptive since the results will show up in the Applitools dashboard and we’ll want to discern which test goes to what since we’ll be doing multiple test runs.
So let’s specify the name of the site (e.g., .webdrivercss('the-internet')) and the page name (e.g., name: 'typos') — the focus element will stay the same (e.g., elem: 'body').
Just like with the assertions in previous examples, res is a collection of the results for the elements used in the test. The main difference here is that we’re performing an assert.equal (instead of assert.ok) to compare the results of the test steps against the baseline image stored in Applitools. We've also included custom output for the assertion (so a permalink for the Applitools job will get outputted for us on failure).
If we save the file and run it (e.g., node false_positives.js from the command-line) the test will run, fail, and provide us with a URL (to the job in Applitools Eyes) in the terminal output. This failure can be ignored since it’s not actually a failure. It’s a side-effect of the Eyes platform capturing a new baseline image. It’s the second test run that we’ll want to pay attention to.
NOTE: If you don’t want to save this initial test run as the baseline, open the test in Applitools Eyes (e.g., either through URL provided or through the dashboard), click Reject, Save, and then re-run your test.
Now let’s run the test again. This time it will be compared against the baseline, fail, and provide us with a URL to job in Eyes. When we view the result page, we can see that the failure was legitimate because it caught the typo.
Here is an image of failure result.
One Last Example
Now, for closure, let’s point our test back at our shifting image example and see how Applitools Eyes holds up.
Just like before, we’ll want to run the test twice. And on the second test run it will pass — clearing the previous hurdle of a false positive from the image shifting by a single pixel.
Outro
It’s worth noting (although not demonstrated in these last two examples) that Applitools Eyes handles other false positive scenarios (e.g., anti-aliasing, dynamic data, etc.) that would normally trip up your tests. And these examples merely scratch the surface of what the Eyes platform can do for your visual testing efforts — which I’ll cover in more depth in future posts.
For now, I’d say you’re armed with more than enough information to dig into automated visual testing on your own; regardless of whether you go with an open source or proprietary solution.
Happy Testing!
Read Dave's 2 previous posts in this series:
(1) How to Do Visual Testing with Selenium
(2) How to Handle Visual Testing False Positives - Part 1
Dave Haeffner is the writer of Elemental Selenium - a free, once weekly Selenium tip newsletter read by thousands of testing professionals. He’s also the creator and maintainer of ChemistryKit (an open-source Selenium framework), and author of The Selenium Guidebook. He’s helped numerous companies successfully implement automated acceptance testing; including The Motley Fool, ManTech International, Sittercity, Animoto, and Aquent. He’s also a founder/co-organizer of the Selenium Hangout, and has spoken at numerous conferences and meet-ups about automated acceptance testing.
