#ui tests

[script-ease]

Speeding up Automated Tests in iOS

If you have been writing UI Tests for your iOS applications and have been frustrated by the amount of time it takes to run the full test sui

[pchensoftware]

Xcode UI Tests

Note that wait(for:timeout:) will wait for a set of expectations, while waitForExpectations() is a syncing point because it will wait for all existing expectations before continuing or failing.

http://masilotti.com/xctest-waiting/

[qualitytesting]

Selection: A UIKit to XCUIElement Translation

I forget or wonder how the attributes of XCUIElements in UI tests translate into the world of UIKit for app development a lot. Other people probably do at some point or another too, so here's my translation of the concept of "selection" between the two.

Note: XCUIElement conforms to the XCUIElementAttributes protocol, and so any property of XCUIElementAttributes will be present on an XCUIElement.

Selection

XCTest: XCUIElementAttributes.isSelected UIKit: UIView.accessibilityTraits.contains(UIAccessibilityTraitSelected)

Key phrases: Switch, toggle, on or off state, stable state

An XCUIElement can either be in a 'selected' state, or a 'not selected' state. This is a trait that is usually applied to UI elements like switches, which have an 'on' and 'off' state.

In XCTest, there is a simple property, isSelected, on all XCUIElements.

In UIKit, selection is denoted using Accessibility Traits. UIViews and their descendants can have one or more "accessibility traits", which help to explain their state. Traits are applied as a bitmask.

Selection does not only apply to switches. Selection states can be added to any kind of view. Some UIKit views have selection built in, like buttons, because they inherit from UIControl.

Buttons can behave in an on/off (toggle) manner, but may not. Before implementing selection as a solution, you should check that your UI element has two distinct, stable states. For example, 'being selected' should not apply to a button which submits a form, since that is not a toggle behaviour, and should also not be used in place of [enabled/disabled] behaviour.

Use Case

A shopping app has many products. Each product has an associated heart image which indicates whether the product is on the user's wishlist or not. When the product is on the user's wishlist, the image is a coloured-in heart. When the product is not on the user's wishlist, the image is an outline of a heart, with no colour fill. When the user taps the heart image, its state is toggled, and the product is added/removed from the user's wishlist as appropriate.

UIKit:

let wishlistIndicator = UIImageView! let unselectedWishlistImage = UIImage! let selectedWishlistImage = UIImage! ... func toggleWishlist() { if wishlistImage.accessibilityTraits.contains(UIAccessibilityTraitSelected) { // Switch to unselected state wishlistIndicator.accessibilityTraits = UIAccessibilityTraitButton wishlistIndicator.image = unselectedWishlistImage } else { // Switch to selected state wishlistIndicator.accessibilityTraits = (UIAccessibilityTraitButton | UIAccessibilityTraitSelected) wishlistIndicator.image = selectedWishlistImage } }

XCTest:

func testWishlistIndicatorCanBeSelected() { let app = XCUIApplication() app.launch() ... let wishlistIndicator = app.images["wishlistIndicator"] wishlistIndicator.tap() XCTAssertTrue(wishlistIndicator.selected) }

[qualitytesting]

Hi, I've started using BDD and Cucumber few months back. Also reading , listening a lot about good practices in BDD. I would like to learn from your experience of Test Pyramid. Earlier I was advocate of doing all automation at UI level. But now I feel Test Pyramid is best way to give faster feedback to developer and product manager. Thanks & Regards, Vikram

Hi Vikram,It’s true, the test pyramid is the way to go in an ideal world. The theory being that the earlier you find bugs, the better.If you find them during the planning phase, before any code gets written, that’s great - you just change (or abort) your approach during planning - you’ve prevented the bug from ever existing. Finding them during development is not *as* great, but at least the developer can solve it before they think they’re finished - all in one go with the rest of feature development. If a bug is found during testing, after the developer thinks they’re done and has moved on to another task, more expense is added to the process of fixing the bug, because the developer will need to get their head back into ‘the zone’ around that feature, having moved to another ‘zone’ to develop the next feature. The worst stage to find a bug is after you’ve deployed whatever you’re working on to your users, as you then have to add your reputation to the list of expenses.The test pyramid helps us to find bugs as early on in the process as possible. Doing BDD is the best way that I know to find bugs during the planning stage - the bugs found during planning don’t cost any development time. Since you’re defining the behaviour of a feature before you start developing it, you and the other members of your team have time allotted to really think about the feature, whether it’s possible, and whether the behaviour you’re expecting makes sense, and because you’re keeping the discussion about *behaviour*, which is very high-level, everyone on the team can join in and ensure they’re on the same page.Please remember that ‘doing BDD’ is not the same as having Gherkin or “Given When Then” scenarios somewhere in your workflow. It is **only** **B**ehaviour-**D**riven **D**evelopment if you are defining the behaviour as a team **before development** starts.That said, BDD doesn’t really have anything to do with the test pyramid - they just compliment each other.The power of the test pyramid comes in at the unit test level. Unit tests could be a product of TDD, but they don’t have to be. What’s important for the test pyramid to work is that the code **is** unit tested. As much as possible. Unit tests form the base of the test pyramid, and if your code coverage is poor at this level, you’re going to have to do more of other kinds of testing to get the same kind of confidence in the stability of your app.Unit tests are fantastic because even with hundreds of them, they complete within seconds. Get as much unit test coverage as possible because they are **really cheap to run**. UI tests - automated or not - can take minutes, hours or days, but unit tests can be done in the time it takes you to take a sip of water or check your phone for messages.Why is this valuable? Apart from not wanting to end up with a test ice cream cone instead of a test pyramid, having a bunch of unit tests allows your developers to run a quick, mini-regression test before they say that development is “done”. If they do find any failures when they run the unit tests, they can fix them right there and then, without having to switch contexts, or waste any of their testers’ time. Faster feedback means that bugs get found and fixed earlier.Additionally, anything that is covered by a unit test doesn’t need to be covered by any other kind of test, and even if you can test something really quickly, you won’t be as fast as a unit test to do the test and report your result.The problem with unit tests is getting them written. In my experience, the developer writing the feature will be responsible for writing the unit tests. I know that there are teams out there who have processes where one developer writes the unit tests for a feature and the other developer will write the feature to get all the unit tests to pass, and others where testers will write unit tests separately from feature development, but I’ve never worked on any of these teams.The biggest challenge you will face is getting them written, **not because the developer doesn’t want to**, but because of time pressures induced by the business side of the team. Deadlines and being first-to-market and showing high productivity levels with tangible outputs are all **the enemies of unit tests**. To business stakeholders, unit tests are pretty much invisible, and while you can try to sell the process to them by insisting that increasing unit testing will improve stability and speed up output in the long run, it’s difficult for business people to allocate time to something that’s not visible feature output, especially at first, or if anyone on the team has never worked with unit tests before.If you’re not doing test-driven development, where writing unit tests is a pre-requisite to writing code for production, it’s easy and common for unit testing time to be pulled away from unit testing and onto something else that visibly provides value. Creating a unit testing culture requires buy-in from everyone on your team. Especially the person/people who will be writing them.With a strong base of unit tests, your testing pyramid may contain some integration tests and/or end-to-end tests. This is where automated UI testing sits. If your product has multiple layers (e.g. a remote API is involved) then you may want to have UI tests with and without stubbing for integration and end-to-end tests respectively.Stubbed UI tests can be relatively fast compared to end-to-end UI tests, and should be more stable. Have as many stubbed UI tests as is necessary to test the integration of your features with each other. Again, at the end-to-end level, have as many UI tests as necessary to test the product working with the full stack. If there any cases which couldn’t be covered by unit tests, you can substitute a UI test for that case at one of these levels.If you’ve done everything right up until this point, the UI tests that you have are generally covering cases around the UI implementation. It’s difficult and impractical to test UI code with unit tests, so this should be the main are of deficit covered by your UI tests. If you’re testing logic within the product you’re making, think about whether it could be tested by a unit test rather than a UI test.Anything that can’t become an automated UI test or a unit test has to become a manual test. Manual tests are necessary in pretty much every project with a UI, but they are expensive to repeat. Manual test cases may be the cheapest to write - unit tests and UI tests involve someone with programming knowledge writing code - but once unit tests and UI tests are written, they can be repeated with the push of a button. Manual tests require a human to be paid to repeat them, and that’s slow and expensive. Slow and expensive **but** necessary and valuable.A human doing manual tests will be able to give you usability feedback, and is able to notice when something looks weird or not quite right. This is a very valuable part of testing and is not one to be overlooked. Yes, automate as much as is practical, but remember that there are a lot of things that are not practical or possible to automate, and for that, you will need a human. Even though they are slow and expensive.The gist of it is that computers are incredibly fast and good at repetition. The test pyramid is a model for using this to your advantage. The test pyramid makes a lot of logical sense, but the biggest obstacle to achieving it is finding and agreeing on a balance between spending your time on achieving flawless logic and investing in your code, or spending your time making something that you can sell for money so that you still have a job tomorrow.

[qualitytesting]

Chaos Testing on iOS

Today, I wrote a couple of chaos tests using XCTest in Swift, which you can find on GitHub.

XCTest is Apple's (first-party!) test framework, which has recently expanded to include support for UI testing, as opposed to just unit testing.

XCTest made it easy to create this small suite of chaos tests.

It's not bug-free, but most of the time it is pretty slick. The APIs are simple and solid, resulting in code that's easy to understand, and execution is speedy in comparison to Appium, which seems to be the community's first choice of third-party mobile UI automation framework at the moment.

Using a first-party tool has unrivalled advantages; not having to wait for compatibility updates to your test framework before you can run your tests on the latest OS with the latest version of Xcode is incredibly valuable. It allows you to stay current with the latest technologies, not just for testing, but for your entire team. Teams will often co-ordinate when they update their tools to ensure compatibility, and having day one support for your testing framework means that you won't be holding up that transition.

What is Chaos Testing?

Chaos testing, in a front-end UI automation context, is where you throw a bunch of nonsensical actions at your UI and see how it deals with it.

You're hoping there won't be any crashes, weird glitches or general failures while random taps, swipes and rotations are being hurled in the direction of your app.

Chaos testing is a great way to find manifestations of memory leaks, unhandled scenarios or situations a person with the curse of context would never have come across.

It's not a perfect test strategy, though. You'll need to capture any crash reports it produces, or record the test run in case there's a failure, so that when something does go wrong, you can look back and find out what, or at least find somewhere to start investigating.

ChaosUITesting

You will need:

Xcode >7.2

iOS >9.0 simulator

A working iOS app project

Setup Instructions

TL;DR: Add a UI testing target to your Xcode project and import the ChaosUITests.swift file from this repository.

Open your Xcode project

Go to File > New > Target...

Select iOS > Test

Select iOS UI Testing Bundle

Click Next

Name your UI tests something like ChaosUITests

Click Finish

Replace the automatically generated tests file with the ChaosUITests.swift file from this repository.

Initiating Chaos

You can run the tests by viewing the ChaosUITests.swift file and clicking the little play button next to the class declaration or test method declarations:

Or you can run the tests by opening the Test navigator in the Navigator pane on the left of Xcode and clicking the little play button next to the ChaosUITests target:

There are two tests; one which tests for a set amount of time and one which keeps going until a certain number of gestures has been executed.

To customise the tests, you can change some of the constants declared at the top of the ChaosUITests class.

Duration

let duration: Double = 60 * 3 // Execution time limit in seconds

Duration is set to three minutes by default. You can customise the duration with any Double value.

Gesture Limit

let gestureLimit = 100 // Number of gestures to be executed

The gesture limit is set to 100 by default. You can set this number equal to any positive valid unsigned integer value.

Minimum Gesture Frequency

let minimumGestureFrequency: UInt32 = 1 // Minimum amount of time to pass between gestures in seconds

The minimum frequency of gestures has been set to 1 second by default. Shorter times were tested using usleep(UInt32), but these proved to be unstable.

You can make this integer higher if desired, but things are going to happen pretty slowly if it's increased.