#dave haeffner

[applitools]

[Webinar Recording] State of Selenium and Test Automation: 4 Leading Experts Share Their Vision of Things to Come

Interested in learning what are the key skills expected from software testing professionals? Want to hear first-hand where is Selenium going? Then you must listen to this on-demand session!

Four experts in Test Automation and free and open-source software shared with us their opinions on the state of testing, the hottest trends in test automation, and their vision on the future of Selenium.

Dave Haeffner, Ashley Hunsberger, Karen Sandler and Joe Colantonio joined forces to discuss emerging trends that will shape your testing environment in the months and years to come, including key takeaways and highlights from the last Selenium Conference, that took place last month in Austin Texas.

Watch it now:

[applitools]

*EXCLUSIVE WEBINAR* The Future of Test Automation: Leading Experts Share Their Vision for 2017

Join four of the world’s top Test Automation experts, and hear their take on the hottest software testing trends coming your way in 2017.

Register Now!

Selenium gurus Dave Haeffner, Jim Evans, Simon Stewart and Brian Jordan, join forces to discuss emerging trends, required skills, and best practices that will shape your testing environment during 2017, including key takeaways and highlights from the last Selenium conference, that took place in London during November 2016.

Interested in learning what are the key aspects and skills expected from software testing professionals in 2017? Then you can’t miss this webinar!

Register Now, and join us live on December 15.

[applitools]

[Webinar Recording] Dave Haeffner’s Proven Method to Grading the Quality of Selenium Tests

Watch this advanced webinar with Test Automation expert Dave Haeffner, and learn how to accurately grade the quality of your Selenium tests.

So you've written your fair share of Selenium tests.

Perhaps you've dabbled with Page Objects, Wait Strategies (aka Implicit and/or Explicit Waits), and you feel confident about your locators. Your test code might be in pretty good shape -- able to work reliably as time marches on and the application under test continues to evolve and your testing needs continue grow with it.

But how do you know? It's not like there is a quantitative way to measure this.

Or is there?

Watch this advanced session with Selenium expert Dave Haeffner, as he steps through the core tenets of good test and page object design, locators, and a repeatable and quantitative approach for assessing your test code. When you're done, you'll be able to see how your tests and page objects stack up, and what changes are needed to help them stand the test of time.

Dave's slide-deck can be found HERE

[applitools]

Automating Your Test Runs with Continuous Integration - CI Series: Part 3/3

by Dave Haeffner - author of The selenium Guidebook

This is the final post in a 3-part series on getting started off with automated web testing on the right foot. You can find the first two posts here and here.

Automating Your Test Runs with Continuous Integration

You'll probably get a lot of mileage out of your automated tests if you run things from your computer, look at the results, and tell people when there are issues in the application. But that only helps you solve part of the problem.

The real goal in test automation is to find issues reliably, quickly, and automatically -- and ideally, in sync with the development workflow you're a part of.

To do that we need to use a Continuous Integration server.

A Continuous Integration Server Primer

A Continuous Integration server (a.k.a. CI) is responsible for merging code that is actively being developed into a central place (e.g., "trunk" or "master") frequently (e.g., several times a day, or on every code commit, etc.) to find issues early so they can be addressed quickly — all for the sake of releasing working software in a timely fashion.

With CI, we can automate our test runs so they can happen as part of the development workflow. The lion’s share of tests that are typically run on a CI Server are unit (and potentially integration) tests. But we can very easily add in our recently written Selenium tests.

There are numerous CI Servers available for use today, most notably:

Bamboo

Jenkins

Solano Labs

TravisCI

etc.

Let's step through an example of using Jenkins on CloudBees.

An Example

Jenkins is a fully functional, widely adopted, and open-source CI adn CD (Contibuous Delivery) server. Its a great candidate for us to step through. And DEV@cloud is an enterprise-grade hosted Jenkins service offered by CloudBees, the enterprise Jenkins company. It takes the infrastructure overhead out of the equation for us.

1. Quick Setup

We'll first need to create a free trial account, which we can do here.

Once logged in we can click on Get Started with Builds from the account page. This will take us to our Jenkins server. We can also get to the server by visiting http://your-username.ci.cloudbees.com. Give it a minute to provision, when it's done, you'll be presented with a welcome screen.

NOTE: Before moving on, click the ENABLE AUTO-REFRESH link at the top right-hand side of the page. Otherwise you'll need to manually refresh the page to see results (e.g., when running a job and waiting for results to appear).

2. Create A Job

Now that Jenkins is loaded, let's create a Job and configure it to run our tests.

Click New Item from the top-left of the Dashboard

Give it a descriptive name (e.g., Login Tests IE8)

Select Freestyle project

Click OK

This will load a configuration screen for the Jenkins job.

3. Pull In Your Test Code

Ideally your test will live in a version control system (like Git). There are many benefits to doing this, but the immediate one is that you can configure your job (under Source Code Management) to pull in the test code from the version control repository and run it.

Scroll down to the Source Code Management section

Select the Git option

Input the Repository URL (e.g., https://github.com/tourdedave/getting-started-blog-series.git)

Now we're ready to tell the Jenkins Job how to run our tests.

4. Add Build Execution Commands

Scroll down to the Build section

Click on Add Build Step and select Execute Shell

In the Command input box, add the following commands:

export SAUCE_USERNAME="your-sauce-username" export SAUCE_ACCESS_KEY="your-sauce-access-key" export APPLITOOLS_API_KEY="your-applitools-api-key" gem install bundler bundle install bundle exec rspec

Since our tests have never run on this server we need to include the installation and running of the bundler gem (gem install bundler and bundle install) to download and install the libraries (a.k.a. gems) used in our test suite. And we also need to specify our credentials for Sauce Labs and Applitools Eyes (unless you decided to hard-code these values in your test already - if so, then you don't need to specify them here).

5. Run Tests & View The Results

Now we're ready to save, run our tests, and view the job result.

Click Save

Click Build Now from the left-hand side of the screen

When the build completes, the result will be listed on the job's home screen under Build History.

You can drill into the job to see what was happening behind the scenes. To do that click on the build from Build History and select Console Output (from the left navigation). This output will be your best bet in tracking down an unexpected result.

In this case, we can see that there was a failure. If we follow the URLs provided, we can see a video replay of the test in Sauce Labs (link) and a diff image in Applitools Eyes.

The culprit for the failure here wasn't a failure of functionality, but a visual defect with the image on the Login button.

A Small Bit of Cleanup

Before we can call our setup complete, we'll want a better failure report for our test job. That way when there's a failure we won't have to sift through the console output for info. Instead we should get it all in a formatted report. For that, we'll turn to JUnit XML (a standard format that CI servers support).

This functionality doesn't come built into RSpec, but it's simple enough to add through the use of another gem. There are plenty to choose from with RSpec, but we'll go with rspec_junit_formatter.

After we install the gem we need to specify some extra command-line arguments when running our tests. A formatter type (e.g., --format RspecJunitFormatter) and an output file for the XML (e.g., --out results.xml). And since this type of output is really only useful when running on our CI server, we'll want an easy way to turn it on and off.

# filename: .rspec <% if ENV['ci'] == 'on' %> --format RspecJunitFormatter --out tmp/result.xml <% end %>

Within RSpec comes the ability to specify command line arguments that are used frequently in a file (e.g., .rspec) that lives in the root of the test directory. In it we specify the new commands we want to use and wrap them in a conditional that checks an environment variable that denotes whether or not the tests are being run on a CI server (e.g., if ENV['ci'] == 'on').

Now it's a small matter of updating our Jenkins job to consume this new JUnit XML output file by adding a post-build action to publish it as a report.

Then we need to tell the Jenkins job where the XML file is. Since it ends up in the root of the test directory, we can just specify the file extension with a wildcard.

Lastly, we need to update the shell commands for the build to set the ci environment variable to on.

Now when we run our test, we'll get a test report which states which test failed. And when we drill into it, we get the URLs for the jobs in Sauce Labs and Applitools Eyes.

One More Thing: Notifications

In order to maximize your CI effectiveness, you'll want to send out notifications to alert your team members when there's a failure.

There are numerous ways to go about this (e.g., e-mail, chat, text, co-located visual cues, etc). And thankfully there are numerous, freely available plugins that can help facilitate whichever method you want. You can find out more about Jenkins' plugins here.

For instance, if you wanted to use chat notifications and you use a service like HipChat or Slack, you would do a plugin search and find one of the following plugins:

After installing the plugin for your chat service, you will need to provide the necessary information to configure it (e.g., an authorization token, the channel/chat room where you want notifications to go, what kinds of notifications you want sent, etc.) and then add it as a Post-build Action to your job (or jobs).

Now when your CI job runs and fails, a notification will be sent to the chat room you configured.

Outro

If you've been following along through this whole series, then you should now have a test that leverages Selenium fundamentals, that performs visual checks (thanks to Applitools Eyes), which is running on whatever browser/operating system combinations you care about (thanks to Sauce Labs), and running on a CI server with notifications being sent to you and your team (thanks to CloudBees).

This is a powerful combination that will help you find unexpected bugs (thanks to the automated visual checks) and act as a means of collaboration for you and your team.

And by using a CI Server you're able to put your tests to work by using computers for what they're good at -- automation. This frees you up to focus on more important things. But keep in mind that there are numerous ways to configure your CI server. Be sure to tune it to what works best for you and your team. It's well worth the effort.

Links to the previous two posts in Dave Haeffner's CI series:

Post 1: How To Get Started with Automated Web Testing

Post 2: How To Run Your Automated Web Tests on Any Browser

Happy Testing!

Watch this post come to live! Learn how to build powerful automated cross-browser tests, covering visual testing, configured to run automatically on a continuous integration (CI) server - presented by Dave Haeffner. Click here to watch it now, on-demand.

[applitools]

Selenium Tips & Tricks - Talk by Dave Haeffner

Must-see talk for test automation professionals!

Learn how to use Selenium like a Pro! Join Dave Haeffner, Selenium expert and author of The Selenium Guidebook, as he steps through the best and most useful tips & tricks from his weekly Selenium tip newsletter.

In this talk, Dave covers the following topics:

Headless test execution

Testing HTTP status codes

Blacklisting third-party content

Load testing

Broken image checking

Testing forgot password

Working with A/B testing

File downloads

and more...

Watch this 2-part in-depth Selenium talk right here:

Selenium Tips & Tricks - Part 1:

Selenium Tips & Tricks - Part 2:

This talk was recorded at the Selenium IL Meetup on March 2, 2015

.

[applitools]

How To Do Cross-browser Visual Testing with Selenium

by Dave Haeffner - author of the The Selenium Guidebook

The Problem

It's easy enough to get started doing visual testing with a single browser, and with Selenium you can quickly expand your efforts into different browsers. But its difficult to verify that your web application looks right on all of them (e.g., making sure the elements are not missing/mis-aligned/hidden, etc.).

Especially when there are small rendering inconsistencies between browsers that can easily cause your visual tests to fail. Things like how an image gets rendered. This poses a real challenge since you can't use traditional visual testing techniques like pixel comparison. So standard workarounds like modifying the match tolerance won't work.

A Solution

By leveraging a solution like Applitools Eyes we can reliably run the same visual tests against multiple browsers (which we can gain access to through the use of Sauce Labs).

A Visual Matching Primer

Normally, you would use a strict visual match for visual regression testing. This is good for verifying the layout of an application in the same execution environment (e.g., same browser, screen size, device, etc.), and with exactly the same (or very similar) content. And if you want to cover multiple execution environments (e.g., multiple browsers, various screen sizes, etc.) you need to maintain several baseline images.

But with layout matching (a feature that Applitools Eyes offers) we can use a single baseline for validating multiple execution environments and/or sites with extremely dynamic content.

Let's dig in with an example.

An Example

NOTE: This example builds on the test code from this previous write-up which covers how to add visual testing to existing Selenium tests using Applitools Eyes and Sauce Labs. You can see the full code example from it here. In order to follow along with this example, you'll want to familiarize yourself with the sample code. Also, if you want to play along at home, you'll need to have an account for both Applitools Eyes and SauceLabs (they have free trial options).

To prepare ourselves for cross-browser testing, we'll need to modify our setup() method. This is where we'll focus all of our efforts for this post.

Here is where we left off with the setup() method from the previous post.

// filename: Login.java // ... @Before public void setup() throws Exception { DesiredCapabilities capabilities = DesiredCapabilities.internetExplorer(); capabilities.setCapability("platform", Platform.XP); capabilities.setCapability("version", "8"); capabilities.setCapability("name", testName); String sauceUrl = String.format( "http://%s:%[email protected]:80/wd/hub", "YOUR_SAUCE_USERNAME", "YOUR_SAUCE_ACCESS_KEY"); WebDriver browser = new RemoteWebDriver(new URL(sauceUrl), capabilities); sessionId = ((RemoteWebDriver) browser).getSessionId().toString(); eyes = new Eyes(); eyes.setApiKey("YOUR_APPLITOOLS_API_KEY"); driver = eyes.open(browser, "the-internet", testName); } // ...

First, let's modify the DesiredCapabilities instantiation so that we can more flexibly specify the browser name.

@Before public void setup() throws Exception { DesiredCapabilities capabilities = new DesiredCapabilities(); capabilities.setCapability("browserName", "firefox"); capabilities.setCapability("platform", Platform.XP); capabilities.setCapability("version", "36"); capabilities.setCapability("name", testName);

Next, we'll want to specify the name of the baseline so we can reuse it. If one doesn't exist, it will be created. And if one exists, it will be used for comparison. This enables us to run our test in one browser, capture a baseline, and then rerun the same test against another browser, and compare the results between the browsers. And if we don't do it, a separate baseline will automatically be created and used for each browser or screen size.

eyes.setBaselineName(testName); driver = eyes.open(browser, "the-internet", testName); }

Now let's save the file and run the test to capture the baseline (e.g., mvn clean test -Dtest=Login.java from the command line).

Once the test completes, Applitools will provide the URL to the job in the test output. You can review it to make sure it is what you expect. If it is Accept and Save it. Alternatively, you can assume the baseline image is correct and proceed without checking it. It will automatically be used as the baseline on future test runs.

Now that we have a baseline, let's change our browserName and version capabilities so our test will run against a different browser.

@Before public void setup() throws Exception { DesiredCapabilities capabilities = new DesiredCapabilities(); capabilities.setCapability("browserName", "internet explorer"); capabilities.setCapability("platform", Platform.XP); capabilities.setCapability("version", "8"); // ...

When we save and run it (e.g., mvn clean test -Dtest=Login.java from the command-line) the test fails.

When we view the results we can see that there are no visual bugs between Firefox and Internet Explorer. Instead, there are differences in how the page was rendered (e.g., different screen sizes, different placement of text, etc.).

The test was considered a failure because the Applitools Eyes match level defaults to Strict mode, which performs a match that is very exact. For effective cross-browser visual testing, we'll need to use a different match level called Layout2.

Let's update our test code to use this match level instead.

import com.applitools.eyes.MatchLevel; // ... eyes.setBaselineName(testName); eyes.setMatchLevel(MatchLevel.LAYOUT2); driver = eyes.open(browser, "the-internet", testName); }

Now when we save our test and run it again (e.g., mvn clean test -Dtest=Login.java from the command-line) it will pass.

For Consistent Results

The test worked this time, but for consistent results we'll want to specify the viewport size in our Applitools setup. This will help ensure that the viewport size of each browser is consistent regardless of the browser used and the system's screen resolution.

If you don't know what size to specify, go with a generic value like 1000x600.

import com.applitools.eyes.RectangleSize; // ... eyes.setBaselineName(testName); eyes.setMatchLevel(MatchLevel.LAYOUT2); driver = eyes.open(browser, "the-internet", testName, new RectangleSize(1000, 600)); }

If the application you're testing is responsive you can verify it's layout by changing the viewport size to force the page layout to change. But regardless of the viewport size specified, Applitools Eyes will scroll through the page and stitch together a full page screenshot for validation on each test run.

A Small Bit of Cleanup

Rather than constantly modifying our capabilities by hand to change the browser, version, and platform let's update the test setup to retrieve runtime properties specified on the command-line instead.

@Before public void setup() throws Exception { DesiredCapabilities capabilities = new DesiredCapabilities(); capabilities.setCapability("browserName", System.getProperty("browser", "firefox")); capabilities.setCapability("platform", System.getProperty("platform", "Windows XP")); capabilities.setCapability("version", System.getProperty("browserVersion", "36")); // ...

With this approach we're also able to set sensible defaults, which we've done. So now if we don't specify anything, Firefox 36 will run on Windows XP.

To specify values we need to use the -D flag when running our tests on the command line. Here are some examples of it in use. Notice the use of double-quotes for values with spaces.

mvn clean test -Dtest=Login.java -Dbrowser="internet explorer" -DbrowserVersion=8 mvn clean test -Dtest=Login.java -Dbrowser="internet explorer" -DbrowserVersion=10 -Dplatform="Windows 8" mvn clean test -Dtest=Login.java -Dbrowser=firefox -DbrowserVersion=26 -Dplatform="Windows 7" mvn clean test -Dtest=Login.java -Dbrowser=safari -DbrowserVersion=8 -Dplatform="OS X 10.10" mvn clean test -Dtest=Login.java -Dbrowser=chrome -DbrowserVersion=40 -Dplatform="OS X 10.8"

For a full list of available browser and operating system combinations, check out Sauce Labs' platform list.

Outro

Now you're ready to start cross-browser visual testing on your own. Stay tuned for future posts where I'll dig into testing responsive apps, checking for visual regressions across multiple environments, and a whole lot more.

Until then, Happy Testing!

[applitools]

How To Add Visual Testing To Your Existing Selenium Tests - by Dave Haeffner

In previous write-ups by Selenium expert Dave Haeffner (published here, on the Applitools blog), he covered the basics of automated visual testing and how to execute it. Following those posts, many of our readers requested more in-depth information on how automated visual testing fits into existing automated testing practices.

Some of the questions received were: Do you need to write and maintain a separate set of tests? What about your existing Selenium tests? What do you do if there isn’t a sufficient library for the programming language you’re currently using?

In response, Dave Haeffner wrote a post that will put your mind at ease: you can build automated visual testing checks into your existing Selenium tests.

In is in-depth post: "Adding Automated Visual Testing to Existing Selenium Tests", Dave demonstrates how, by leveraging a third-party platform such as Applitools Eyes, this is a simple feat. And when coupled with a cross-browser test framework, such as Sauce Labs, you can quickly add coverage for those hard to reach browser, device, and platform combinations.

You can read Dave's full post: Adding Automated Visual Testing to Existing Selenium Tests, on the Sauce Labs blog.

In addition, if you're interested in learning more on how Automated Visual Testing can boost your cross-browser coverage and reduce maintenance, we invite you to watch this hands-on webinar [free, on-demannd], we hosted with our friends at Sauce Labs.

[applitools]

How To Add Visual Testing To Your BDD Tests

Post by Dave Haeffner - author of The Selenium Guidebook

The Problem

If you're new to Behavior Driven Development (BDD) tooling (e.g., Cucumber) it may not be obvious how scenarios you've specified (using the Gherkin syntax) translates into test automation. If that's the case, then adding visual testing to the mix might be a challenge as well.

A Solution

By using the built-in functionality of our BDD tool of choice, we can easily generate step definitions for our scenarios and add in test execution with a third-party provider like Applitools (for visual testing) and Sauce Labs (for access to additional browsers/devices).

NOTE: In order to do BDD well don't start with automation. You should focus on communication instead. This way everyone on the team (both business and tech alike) have a shared understanding of what needs to be done and what is actually being done. There are loads of great write-ups to help guide you on this. Pieces like "Step Away From The Tools" by Liz Keogh (link) and Specification Workshops from "Bridging The Communication Gap" by Gojko Adzic (link).

An Example

For this example, let's use Cucumber to step through automating the login of a website. Scenarios for valid and invalid users would look something like this:

# filename: features/login.feature Feature: Login Scenario: Valid User Given a user with valid credentials When they log in Then they will have access to secure portions of the site Scenario: Invalid User Given a user with invalid credentials When they log in Then they will not gain access to secure portions of the site

Gherkin scenarios are plain text files that end in .feature. And they live in a directory called features.

Inside of the features directory, there are some additional folders we'll want to use (e.g., step_definitions and support).

├── features │ ├── login.feature │ ├── step_definitions │ └── support

When we save the feature file and run it (e.g., cucumber from the command-line), Cucumber will see if there are any step definitions that match. If there aren't, it will provide us with some code to get us started.

You can implement step definitions for undefined steps with these snippets: Given(/^a user with valid credentials$/) do pending # express the regexp above with the code you wish you had end When(/^they log in$/) do pending # express the regexp above with the code you wish you had end Then(/^they will have access to secure portions of the site$/) do pending # express the regexp above with the code you wish you had end Given(/^a user with invalid credentials$/) do pending # express the regexp above with the code you wish you had end Then(/^they will not gain access to secure portions of the site$/) do pending # express the regexp above with the code you wish you had end

We can copy this outputted code and paste it into a new file -- login.rb in the step_definitions directory.

This is where we'll place our test actions (e.g., Selenium commands, assertions, etc.). But before we do that, we'll need to take care of setting up and tearing down our Selenium session.

That gets handled in the support directory. All files in this directory get executed before the tests. But there's one file in particular that will get executed before anything else -- and that's env.rb. Let's create this file and add in our Selenium configuration with access to Applitools and Sauce Labs.

# filename: support/env.rb require 'selenium-webdriver' require 'rspec/expectations' include RSpec::Matchers require 'eyes_selenium' Before do |scenario| @eyes = Applitools::Eyes.new @eyes.api_key = 'your Applitools API key' caps = Selenium::WebDriver::Remote::Capabilities.internet_explorer caps.version = '8' caps.platform = 'Windows XP' caps['name'] = scenario.title browser = Selenium::WebDriver.for( :remote, url: "http://your-sauce-username:[email protected]:80/wd/hub", desired_capabilities: caps) @driver = @eyes.open(app_name: 'the-internet', test_name: scenario.title, driver: browser) end After do @eyes.abort_if_not_closed @driver.quit end

At the top of the file we pull in our requisite libraries (e.g., selenium-webdriver to load and drive the browser, rspec/expecations and RSpec::Matchers to perform an assertion, and eyes_selenium for visual testing with Applitools Eyes).

Next we specify our setup and teardown in Before and After blocks. Things specified here will occur before and after each scenario specified in our feature files.

In Before we create an instance of Applitools Eyes, configure the browser/operating system we want on Sauce Labs, and join the two together -- storing the final outcome (which is a Selenium WebDriver instance that is now connected to both Applitools and Sauce Labs) in a @driver variable. This variable will automatically be made available for use in the step definitions.

In After we ensure that our Applitools connection closes (in addition to quitting the instance of Selenium).

Now let's wire everything up in our step definition.

# filename: step_definitions/login.rb Given(/^a user with valid credentials$/) do @user = { username: 'tomsmith', password: 'SuperSecretPassword!' } end Given(/^a user with invalid credentials$/) do @user = { username: 'tomsmith', password: 'badpassword' } end When(/^they log in$/) do @driver.get 'http://the-internet.herokuapp.com/login' @driver.find_element(id: 'username').send_keys(@user[:username]) @driver.find_element(id: 'password').send_keys(@user[:password]) @driver.find_element(id: 'login').submit end Then(/^they will have access to secure portions of the site$/) do @eyes.check_window('Logged In') expect(@driver.find_element(css: '.flash.success').displayed?).to eql true @eyes.close end Then(/^they will not gain access to secure portions of the site$/) do @eyes.check_window('Not Logged In') expect(@driver.find_element(id: 'login').displayed?).to eql true expect(@driver.find_element(css: '.flash.error').displayed?).to eql true @eyes.close end

Our Selenium actions are simple -- we're finding the form elements and inputting text (e.g., username and password), submitting the form, and checking to make sure the correct notification message appeared in an assertion.

With our Applitools commands (e.g., @eyes.check_window() and @eyes.close) we are capturing snapshots of the page and comparing them against the baseline image.

Expected Outcome

If we save this file and then run cucumber again (e.g., cucumber from the command-line), here is what will happen:

Create an instance of Selenium on Sauce Labs

Connect the Selenium instance to Applitools Eyes

Test actions run

Assertions (e.g., expect()) and image checks (e.g., @eyes.check_window()) occur

Applitools and Sauce Labs sessions close

Test results get outputted

> cucumber Feature: Login Scenario: Login Succeeded # features/login.feature:3 Given a user with valid credentials # features/step_definitions/login.rb:1 When they log in # features/step_definitions/login.rb:8 Then they will have access to secure portions of the site # features/step_definitions/login.rb:16 Scenario: Login Failed # features/login.feature:8 Given a user with invalid credentials # features/step_definitions/login.rb:22 When they log in # features/step_definitions/login.rb:8 Then they will not gain access to secure portions of the site # features/step_definitions/login.rb:29 2 scenarios (2 passed) 6 steps (6 passed) 1m39.410s

If Applitools found a visual bug, it would fail the test by raising an exception and output the URL to the job -- which would look like this:

> cucumber Feature: Login Scenario: Login Succeeded # features/login.feature:3 Given a user with valid credentials # features/step_definitions/login.rb:1 When they log in # features/step_definitions/login.rb:8 Then they will have access to secure portions of the site # features/step_definitions/login.rb:16 'Login Succeeded' of 'the-internet'. see details at https://eyes.applitools.com/app/sessions/251976656790606 (Applitools::TestFailedError) ./features/step_definitions/login.rb:19:in `/^they will have access to secure portions of the site$/' features/login.feature:6:in `Then they will have access to secure portions of the site' Scenario: Login Failed # features/login.feature:8 Given a user with invalid credentials # features/step_definitions/login.rb:22 When they log in # features/step_definitions/login.rb:8 Then they will not gain access to secure portions of the site # features/step_definitions/login.rb:29 'Login Failed' of 'the-internet'. see details at https://eyes.applitools.com/app/sessions/251976656732861 (Applitools::TestFailedError) ./features/step_definitions/login.rb:33:in `/^they will not gain access to secure portions of the site$/' features/login.feature:11:in `Then they will not gain access to secure portions of the site' Failing Scenarios: cucumber features/login.feature:3 # Scenario: Login Succeeded cucumber features/login.feature:8 # Scenario: Login Failed 2 scenarios (2 failed) 6 steps (2 failed, 4 passed) 2m9.743s

A Small Bit of Cleanup

Right now we're explicitly calling eyes.close() to perform a comparison of our tests against their baseline images, but we don't have to. We can easily make it so this automatically gets called at the end of every test run.

To do that, we'll need to eyes.close() to our After block in support/env.rb.

# filename: support/env.rb ... After do @driver.quit @eyes.close end

By placing @eyes.close here we can remove @eyes.abort_if_not_closed since it's no longer necessary.

We can also remove @eyes.close from our Then step definitions. Additionally, we can remove our RSpec assertions since they are redundant -- and our visual validation is effectively performing hundreds of validations.

Then(/^they will have access to secure portions of the site$/) do @eyes.check_window('Logged In') end Then(/^they will not gain access to secure portions of the site$/) do @eyes.check_window('Not Logged In') end

If we save these changes and run the tests one more time (e.g., cucumber from the command-line) they will execute just like before. And when there's a failure, the test output will be correctly attributed to the failing scenario.

Outro

Hopefully this tip has helped save you some time when it comes to automating your BDD scripts and connecting them to services like Sauce Labs and Applitools.

Happy Testing!

Dave Haeffner is the writer of Elemental Selenium - a free, once weekly Selenium tip newsletter read by thousands of testing professionals. He’s also the creator and maintainer of ChemistryKit (an open-source Selenium framework), and author of The Selenium Guidebook. He’s helped numerous companies successfully implement automated acceptance testing; including The Motley Fool, ManTech International, Sittercity, Animoto, and Aquent. He’s also a founder/co-organizer of the Selenium Hangout, and has spoken at numerous conferences and meet-ups about automated acceptance testing.

Image taken from here