Calabash-Script - iOS Test Automation using ClojureScript
h1, h2, h3, h4, h5, h6, p, blockquote { margin: 0; padding: 0; } body { font-family: "Helvetica Neue", Helvetica, "Hiragino Sans GB", Arial, sans-serif; font-size: 13px; line-height: 18px; color: #737373; margin: 10px 13px 10px 13px; } a { color: #0069d6; } a:hover { color: #0050a3; text-decoration: none; } a img { border: none; } p { margin-bottom: 9px; } h1, h2, h3, h4, h5, h6 { color: #404040; line-height: 36px; } h1 { margin-bottom: 18px; font-size: 30px; } h2 { font-size: 24px; } h3 { font-size: 18px; } h4 { font-size: 16px; } h5 { font-size: 14px; } h6 { font-size: 13px; } hr { margin: 0 0 19px; border: 0; border-bottom: 1px solid #ccc; } blockquote { padding: 13px 13px 21px 15px; margin-bottom: 18px; font-family:georgia,serif; font-style: italic; } blockquote:before { content:"\201C"; font-size:40px; margin-left:-10px; font-family:georgia,serif; color:#eee; } blockquote p { font-size: 14px; font-weight: 300; line-height: 18px; margin-bottom: 0; font-style: italic; } code, pre { font-family: Monaco, Andale Mono, Courier New, monospace; background-color: #222; } code { color: white; padding: 1px 3px; font-size: 12px; text-shadow: none; -webkit-border-radius: 3px; -moz-border-radius: 3px; border-radius: 3px; } pre { display: block; padding: 14px; color: #D8FA3C; margin: 0 0 17px; text-shadow: none; line-height: 16px; font-size: 11px; border: 1px solid #d9d9d9; white-space: pre-wrap; word-wrap: break-word; } code pre { margin-bottom: -20px; } pre code { background-color: #fff; color:#737373; font-size: 11px; padding: 0; } @media screen and (min-width: 768px) { body { width: 748px; margin:10px auto; } } body { color: #EDEDED; background-color: #000000; } .builtin { /* font-lock-builtin-face */ color: #FF6400; } .constant { /* font-lock-constant-face */ color: #4c83ff; } .keyword { /* font-lock-keyword-face */ color: #FBDE2D; } .string { /* font-lock-string-face */ color: #61CE3C; } .variable-name { /* font-lock-variable-name-face */ color: #D8FA3C; } a { color: inherit; background-color: inherit; font: inherit; text-decoration: inherit; } a:hover { text-decoration: underline; } By Karl Krukow, co-instructor at Conj Labs - professional Clojure training
This post serves as an introduction to CalabashScript which lets you test iOS applications using ClojureScript.
But first (as required by the Clojure community), The Rationale:
Rationale
Apple provides a technology called UIAutomation which allows one to use JavaScript to write automated tests for iOS apps.
The good thing about UIA is that it is officially supported by Apple, you can test any app built for debugging, and it is quite featureful: You can do things like locking the screen, turning volume up/down or even simulating a 'Shake' on the device.
Unfortunately, from a developer's perspective, UIAutomation (UIA) has several problems: tests are written in JavaScript using a verbose and rather awkward API. For example, this line of code finds the text "Chocolate Cake" in a table:
UIATarget.localTarget().frontMostApp().mainWindow().tableViews()[0].cells()[0].elements()["Chocolate Cake"];
Of course this could be shortened by introducing local variables, but the problems of UIA remain: one must explicitly and imperatively navigate through the view hierarchy. The explicit view structure, and imperative nature of the code leads to verbose tests that tend to be phrase too specific assertions.
UIA elements are represented using a proprietary API containing some awkward types that get us into an expression-like problem. For example, UIAElementArray which is not an array, and UIAElementNil which has unclear semantics.
Finally, the development experience is no less than horrible, IMHO. Ignoring for a moment bad and missing documentation, and the flawed instruments tool, my top problems are: No debugger, no stack traces, horrible error messages. Here is one:
2013-01-10 20:18:47.823 instruments[20240:1603] -[__NSCFBoolean makeNewlinesNotSpecial]: unrecognized selector sent to instance 0x7fff7709c020 Instruments Trace Error : -[__NSCFBoolean makeNewlinesNotSpecial]: unrecognized selector sent to instance 0x7fff7709c020
An Objective-C unrecognized selector from the underlying tool, no line number or anything that can point me back to the source. What did I do to deserve this error? I tried to log a boolean:
var elements = UIATarget.localTarget().frontMostApp().mainWindow().elements(); UIALogger.logMessage(elements instanceof Array); //the answer is false in case you're wondering
As you can imagine, I could go on, but let's instead focus on CalabashScript.
How does CalabashScript help?
Some of the mentioned problems are inherent to the UIAutomation technology while others can be solved to some degree. CalabashScript tries to solve some of the problems by replacing JavaScript with ClojureScript and providing
a data-centric model of the view hiearchy
a query language to declaratively find views (avoiding explicit hierarchy traversal)
providing a REPL for interactive exploratory development
providing high-level functions for interacting with views based on this query language (tap, pan, swipe etc).
Getting Started
You can read installation instruction on the CalabashScript github repo. Once you've setup a dependency on CalabashScript, you can define simple tests in ClojureScript. Key to writing these tests is understanding the query language. The query language is a language inspired by that in our testing framework: Calabash iOS. The idea is to declaratively find UI objects using queries which are like css selectors but phrased in terms of ClojureScript data structures. For example
[:button {:marked "Edit"}]
UIAButtons with "name" or "label" equal to "Edit". You can also (declaratively!) describe partial hiearchy:
[:navigationBar :button {:marked "Edit"}]
This query finds the same buttons, but only if they are descendants of a UIANavigationBar object.
For many tests, you rely on iOS accessibility labels, which are usually unique on a screen. For locating by accessibility label you can use [:view marked "Label"].
The result of the query function is a regular lazy ClojureScript Seq of maps, so you can use all the Clojure data functions to define your test scripts:
(when-not (seq (query [:view {:marked "Olympic Water Polo, past and present"}])) (fail "Olympic Water Polo, past and present"))
There is also a built-in helper function for this: (check-element-exists [:view {:marked "Edit"}]).
The query language is really useful in giving you the power of ClojureScript to write your tests in a declarative manner. The CalabashScript API exposes functions for tapping, typing and panning (drag-drop) - all based on queries. For example, a drag gesture from one table cell to another can be quite hard to express in pure UIAutomation - with CalabashScript
(pan [:tableCell {:marked "Cell 0"} :button] [:tableCell {:marked "Cell 2"} :button])
Finally you can group tests using define-uia-test. Here is a simple example:
(define-uia-test "Archery details should be accessible via Events" (fn [] (tap-mark "Events") (utils/screenshot "Events") (sleep 3) (scroll-to [:view {:marked "water polo"}]) (utils/screenshot "Events") (tap-mark "water polo") (sleep 2) (when-not (seq (query [:view {:marked "Olympic Water Polo, past and present"}])) (fail "Olympic Water Polo, past and present")) (utils/screenshot "Water polo details")))
Notice you can generate screenshots using utils/screenshot.
There are many more functions that what can be described in this introductory post. You can study the main API here: https://github.com/krukow/calabash-script/blob/master/src/cljs/calabash_script/core.cljs.
What else? A REPL!
One of my favorite things about Clojure and ClojureScript is the development experience, and particularly the REPL. The REPL gives us a way to explore APIs, try out small snipplets of code, work in an ad-hoc and exploratory way with data and functions, read documentation etc.
ClojureScript runs where JavaScript runs, but each JavaScript environment usually requires a separate implementation of a REPL. The classic and awesome example is the Browser-connected REPL (by Brenton Ashworth). Another cool example by Bodil Stokke (@bodil) is the REPL for Node.JS.
A great thing about ClojureScript is that to create a REPL for a new environment, you only need to implement a 4-function protocol.
UIAutomation is a separate JavaScript execution environment, so none of the existing REPLs work with it. However, with inspiration from Bodil's implementation, I've created a simple REPL for UIAutomation, which means that CalabashScript has a REPL!
The UIAutomation REPL can be used to play with the CalabashScript API, explore your iOS app, shape your automated tests and play with the UIAutomation APIs. To use the REPL you add a dependency on calabash/uia-repl (leiningen [calabash/uia-repl "0.0.9"]) in your CalabashScript project. With this in place, start a regular Clojure REPL. Here is an example session from the sample project (use instructions there to follow along):
user> (require '[cljs.repl.uia :as uia]) nil user> (def app-path "/Users/krukow/github/2012-Olympics-iOS--iPad-and-iPhone--source-code/2012 Olympics/build/Applications/2012 Olympics.app") #'user/app-path user> (uia/run-uia-repl :app app-path) [instruments -t /Applications/Xcode.app/Contents/Applications/Instruments.app/Contents/PlugIns/AutomationInstrument.bundle/Contents/Resources/Automation.tracetemplate /Users/krukow/github/2012-Olympics-iOS--iPad-and-iPhone--source-code/2012 Olympics/build/Applications/2012 Olympics.app -D run/trace -e UIARESULTSPATH run -e UIASCRIPT /var/folders/63/lq05__ys7xjfcw7fn6vpz6fc0000gn/T/uia-repl4390725227287728864.js] ...
The REPL is a bit slow to start up. This is an artefact of UIAutomation's JavaScript environment. We are using a bit of a hack to get dynamic JavaScript evaluation going in this environment (UIAutomation wasn't designed with this use case in mind!). First of all, you can only evaluate one chunk of JS code per second, and secondly chunks need to be split because there is a size limitation. Fortunately ClojureScript comes with the Google Clojure Compiler which makes it easy to parse up JavaScript and split it on a semantically valid position.
From here we can play with CalabashScript and explore the app (the open source 2012 olympics app in this case):
ClojureScript:user> (ns example (:require [calabash-script.core :as c])) ... ("calabash-script.core") Loading files for (calabash-script.core) ClojureScript:example> (c/names :button) ;; explore names/labes of all buttons on screen ("Home" "Events" "Count Down" "Schedule" "Anthem") ClojureScript:example> (c/tap [:button {:marked "Events"}]) ;; go to events ClojureScript:example> (c/scroll-to [:view {:marked "water polo"}]) ;; scroll to water polo ClojureScript:example> (c/query [:button]) ({:name "archery", :rect {:x 20, :y -1595, :height 101, :width 101}, :hit-point {}, :el #<[object UIAButton]>, :label "archery"} {:name "athletics", :rect {:x 199, :y -1595, :height 101, :width 101}, :hit-point {}, :el #<[object UIAButton]>, :label "athletics"} {:name "badminton", :rect {:x 20, :y -1486, :height 101, :width 101}, :hit-point {}, :el #<[object UIAButton]>, :label "badminton"} ;... )
I'd like a screencast...
OK! I've dumped to short screencasts on youtube showing CalabashScript running, and showing the UIAutomation-based REPL. Enjoy!
This link demos running a plain CalabashScript test.
This link demos CalabashScript in a UIAutomation-based REPL.
What is the status on CalabashScript
I find CalabashScript useful, and would use it in cases where a client would require that I use UIAutomation instead of Calabash. I've developed CalabashScript as a proof of concept and as an exercise in ClojureScript. I will maintain it on a hobby basis, but encourage contributions and would love to see a community form.
Where can we meet?
Well if you don't want to keep meeting like this... :) Please join Lau and I on our professional Clojure training: ConjLabs