iPhone testing, the lay of the land
18 Aug 2009Today, someone asked about acceptance testing of iPhone applications. I answered that there are unit testing frameworks for objective-c but no solutions similar to what Selenium is for web applications.
An hour later during a boxing bag session, I do this to clear my head and as a replacement for cigarette breaks, I figured that my answer was probably wrong. It was based on information I have gathered a few months ago. So I decided to do my homework and brush up my knowledge.
Apple has written a great introduction about unit testing for objective-c. That's a good start, but since an iPhone application is rather GUI heavy, I want to drive the user interface and access its components. In addition to that, I want to assert conditions, otherwise it would be a simplified macro recorder.
Recently I have read Scripted GUI Testing with Ruby. I really liked how the author used RSpec and its story runner to test applications. So I figured it would be great to drive an iPhone application in a similar way, preferably with something that works like Cucumber.
Matt Gallagher has found a way to run automated, scripted tests on an iPhone app's user interface. Basically, he creates a category on UIView with a method that can be called on any UIView object to get a description of it and all its children. To make selecting a specific element easier, he converts the description to XML so that one can use XPATH queries. He also dug deep into undocumented cocoa code to be able to simulate touch events. Those are needed to manipulated the user interface.
There is a project called bromine which uses Matt's approach to do automated user interface testing. That's a good start but I don't really like to write my tests in XML. XML is meant for machines to read, not humans.
Ian Dees, the guy who write Scripted GUI Testing with Ruby, seemed to have the same feeling. He took Matt Gallagher's code, made it work with Cucumber and called it brominet. He also gave a great presentation on that topic. This is definitely an interesting approach, especially because it uses Cucumber.
Last but not least, I stumbled upon a project called uispec. It's a BDD framework which is modeled after RSpec and drives the iPhone UI. The tests can either be written in objective-c or a DSL called UIScript. I gave their sample project a try and recored a video showing it in action.
I must say, uispec looks very promising. Especially the control selectors look simple and straightforward, even though the console output could be a little bit nicer. One thing I don't like are all the warnings the compiler produced. For the sample project, I got 132 warnings. For example:
warning: 'UIQuery' may not respond to '-text'In my opinion, compiler warnings should be treated as errors and therefore should be resolved. I don't like to work on a project, that generates over 100 warnings just because a test framework is in use.
For now, I think a combination of uispec and Cucumber would be my preferred solution to do automated iPhone testing.