In the dynamic world of iOS development, ensuring the quality and reliability of your applications is paramount. iOS tests, encompassing techniques like Test-Driven Development (TDD) and Behavior-Driven Development (BDD), along with robust integration and acceptance testing strategies, are not merely optional add-ons but essential components of a successful development lifecycle. By embracing these practices, developers can proactively identify and address potential issues, leading to more stable, maintainable, and ultimately, more successful applications. This article delves into the core concepts of iOS testing, exploring the methodologies, tools, and best practices that empower developers to build high-quality iOS apps with confidence. We’ll examine how these different types of tests work together to create a comprehensive safety net for your codebase, reducing the risk of bugs and improving overall application performance and user experience. This comprehensive guide will serve as a roadmap for developers of all skill levels looking to improve their testing practices and build robust iOS applications.
Understanding iOS Testing Methodologies: TDD and BDD
Test-Driven Development (TDD) and Behavior-Driven Development (BDD) are two popular methodologies that emphasize writing tests before writing the actual code. TDD follows a “red-green-refactor” cycle, where you first write a failing test (“red”), then write the minimum amount of code to pass the test (“green”), and finally refactor the code to improve its structure and readability while ensuring the test still passes. This iterative approach forces you to think about the desired behavior of your code before you implement it, leading to more focused and well-defined code.
BDD, on the other hand, focuses on describing the expected behavior of the application from a user’s perspective. Using a human-readable language like Gherkin (Given-When-Then), BDD tests define scenarios that describe how the application should behave in different situations. This approach promotes collaboration between developers, testers, and stakeholders, ensuring that everyone has a clear understanding of the application’s requirements. Frameworks like Cucumberish can be used to integrate BDD principles into your iOS projects. Cucumber is a tool that supports BDD.
While TDD and BDD have distinct approaches, they share a common goal: to improve the quality and reliability of software. By writing tests upfront, developers can catch errors early in the development process, reducing the cost and effort required to fix them later. They also create a living documentation of how the code should behave. According to a study by IBM, “fixing a defect during the maintenance phase can cost up to 100 times more than fixing it during the design phase” IBM Developerworks. This statistic highlights the significant cost savings that can be achieved through effective testing practices like TDD and BDD.
Integration Testing: Ensuring Components Work Together
Integration testing focuses on verifying that different components or modules of an application work correctly together. Unlike unit tests, which test individual units of code in isolation, integration tests examine the interactions between multiple units to ensure that they communicate and collaborate as expected. This is crucial because even if individual components pass their unit tests, they may still fail to work correctly when integrated with other parts of the system. This is especially important in complex iOS applications that rely on external APIs or data sources.
One common approach to integration testing in iOS is to test the interaction between view controllers and data models. For example, you might write an integration test to verify that a view controller correctly displays data fetched from a remote API. Another important aspect of integration testing is verifying the behavior of network requests and responses. You can use tools like OHHTTPStubs to mock network requests and simulate different scenarios, such as successful responses, error responses, and timeouts. This allows you to test how your application handles different network conditions without relying on a real API.
Effective integration testing requires careful planning and execution. It’s important to identify the key components and interactions that need to be tested and to design tests that thoroughly exercise those interactions. You should also consider using a test-driven approach to integration testing, writing the tests before implementing the integration logic. This helps to ensure that the integration is designed with testability in mind. Remember that thorough integration testing is paramount for a stable and reliable iOS application. Make sure to use robust mocking frameworks to simulate external dependencies.
Acceptance Testing: Validating End-to-End Functionality
Acceptance testing, also known as user acceptance testing (UAT), is a type of testing that verifies that the application meets the requirements of the end-users or stakeholders. It’s performed after integration testing and before the application is released to production. Acceptance tests are typically written from a user’s perspective and focus on validating the end-to-end functionality of the application. These tests ensure that the application behaves as expected in real-world scenarios and that it meets the business requirements.
In iOS development, acceptance tests often involve testing the user interface (UI) and verifying that users can perform common tasks successfully. For example, you might write an acceptance test to verify that a user can successfully log in to the application, browse a list of products, add a product to their shopping cart, and complete the checkout process. Tools like XCUITest, Apple’s UI testing framework, allow you to automate these types of tests and run them on real devices or simulators. XCUITest provides APIs for interacting with UI elements, such as buttons, text fields, and tables, and for verifying the state of the UI.
The key to effective acceptance testing is to involve the end-users or stakeholders in the testing process. They can provide valuable feedback on the usability and functionality of the application and help to identify any issues that may have been missed during earlier testing phases. Acceptance testing is not just about finding bugs; it’s also about ensuring that the application meets the needs of the users and that it provides a positive user experience. According to research by the Standish Group, “user involvement is the single most important factor for project success” The Standish Group. This highlights the importance of involving end-users in the acceptance testing process.
Tools and Frameworks for iOS Testing
The iOS ecosystem offers a wide range of tools and frameworks to support different types of testing. For unit testing, XCTest is the standard framework provided by Apple. It allows you to write and run unit tests directly within Xcode. For mocking dependencies, frameworks like OCMock and Mockito offer powerful features for creating mock objects that simulate the behavior of real objects. These frameworks make it easier to isolate units of code and test them in isolation.
For UI testing, XCUITest is the recommended framework. It provides APIs for interacting with UI elements and verifying the state of the UI. For BDD, frameworks like Cucumberish allow you to write tests in a human-readable format and execute them against your iOS application. These frameworks provide a bridge between the BDD methodology and the iOS development environment. Furthermore, continuous integration (CI) tools like Jenkins, Travis CI, and CircleCI can automate the process of running tests whenever code is committed to a repository. This helps to ensure that tests are run frequently and that any regressions are detected early.
Choosing the right tools and frameworks is essential for effective iOS testing. Consider the specific needs of your project and the skills of your development team when making your selection. It’s also important to stay up-to-date with the latest tools and techniques in the iOS testing ecosystem. Here’s a summary of key tools:
- XCTest: Apple’s built-in testing framework
- OCMock/Mockito: Mocking frameworks for isolating dependencies
- XCUITest: UI testing framework for automating user interactions
To streamline your testing workflow, consider these steps:
- Set up a continuous integration (CI) environment.
- Integrate your testing frameworks into your CI pipeline.
- Write tests for new features and bug fixes.
- Run tests automatically on every commit.
This setup promotes a culture of continuous testing and helps to prevent regressions.
- What is the difference between unit testing and integration testing?
- Unit testing tests individual units of code in isolation, while integration testing tests the interaction between multiple units.
- What is the purpose of UI testing?
- UI testing verifies that the user interface behaves as expected and that users can perform common tasks successfully.
- How can I improve the performance of my iOS tests?
- Optimize your tests by using mock objects, minimizing network requests, and running tests in parallel.
iOS tests are a critical component of the software development process, especially for mobile applications. They ensure the reliability and robustness of applications by validating code functionality and preventing bugs. Different types of tests like unit tests, integration tests, and UI tests each play a vital role in creating a comprehensive testing strategy. Utilizing frameworks like XCTest and XCUITest, developers can automate these tests and integrate them into a continuous integration pipeline for efficient and reliable software delivery.
The journey to building robust and reliable iOS applications hinges on embracing a culture of thorough testing. We’ve explored various testing methodologies, from the fine-grained precision of unit tests to the holistic validation of acceptance tests, emphasizing the importance of each in ensuring a high-quality user experience. By leveraging the tools and frameworks available within the iOS ecosystem, developers can proactively identify and address potential issues, leading to more stable and maintainable codebases. Don’t wait to implement these strategies. Start by integrating unit tests into your existing projects and gradually expand your testing coverage to include integration and acceptance tests. Explore the wealth of resources and frameworks available to streamline your testing workflow and create a comprehensive testing strategy that aligns with your project’s needs. The investment you make in testing today will pay dividends in the form of reduced bugs, improved user satisfaction, and a more confident development process. Consider exploring advanced topics like property-based testing and mutation testing to further enhance your testing capabilities. Question & Answer :
Unit Testing
Test::Unit Style
- OCUnit/SenTestingKit as explained in iOS Development Guide: Unit Testing Applications & other OCUnit references.
- Examples: iPhoneUnitTests, Three20
- CATCH
- GHUnit
- Google Toolbox for Mac: iPhone Unit Testing
RSpec Style
- Kiwi (which also comes with mocking & expectations)
- Cedar
- Jasmine with UI Automation as shown in dexterous’ iOS-Acceptance-Testing specs
Acceptance Testing
Selenium Style
-
UI Automation (works on device)
-
Tuneup js - cool library for using with UIAutomation.
-
Capturing User Interface Actions into Automation Scripts
It’s possible to use Cucumber (written in JavaScript) to drive UI Automation. This would be a great open-source project. Then, we could write Gherkin to run UI Automation testing. For now, I’ll just write Gherkin as comments.
UPDATE: Zucchini Framework seems to blend Cucumber & UI Automation! :)
Old Blog Posts:
-
UISpec with UISpecRunner
Cucumber Style
-
Frank and iCuke (based on the Cucumber meets iPhone talk)
- The Frank Google Group has much more activity than the iCuke Google Group.
- Frank runs on both device and simulator, while iCuke only runs in simulator.
- Frank seems to have a more comprehensive set of step definitions than iCuke’s step definitions. And, Frank also has a step definition compendium on their wiki.
- I proposed that we merge iCuke & Frank (similar to how Merb & Rails merged) since they have the same common goal: Cucumber for iOS.
-
Zucchini Framework uses Cucumber syntax for writing tests and uses CoffeeScript for step definitions.
Additions
- OCMock for mocking
- OCHamcrest and/or Expecta for expectations
Conclusion
Well, obviously, there’s no right answer to this question, but here’s what I’m choosing to go with currently:
For unit testing, I used to use OCUnit/SenTestingKit in XCode 4. It’s simple & solid. But, I prefer the language of BDD over TDD (Why is RSpec better than Test::Unit?) because our words create our world. So now, I use Kiwi with ARC & Kiwi code completion/autocompletion. I prefer Kiwi over Cedar because it’s built on top of OCUnit and comes with RSpec-style matchers & mocks/stubs. UPDATE: I’m now looking into OCMock because, currently, Kiwi doesn’t support stubbing toll-free bridged objects.
For acceptance testing, I use UI Automation because it’s awesome. It lets you record each test case, making writing tests automatic. Also, Apple develops it, and so it has a promising future. It also works on the device and from Instruments, which allows for other cool features, like showing memory leaks. Unfortunately, with UI Automation, I don’t know how to run Objective-C code, but with Frank & iCuke you can. So, I’ll just test the lower-level Objective-C stuff with unit tests, or create UIButtons only for the TEST build configuration, which when clicked, will run Objective-C code.
Which solutions do you use?
Related Questions
- Is there a BDD solution that presently works well with iOS4 and Xcode4?
- SenTestingKit (integrated with XCode) versus GHUnit on XCode 4 for Unit Testing?
- Testing asynchronous code on iOS with OCunit
- SenTestingKit in Xcode 4: Asynchronous testing?
- How does unit testing on the iPhone work?
tl;dr
At Pivotal we wrote Cedar because we use and love Rspec on our Ruby projects. Cedar isn’t meant to replace or compete with OCUnit; it’s meant to bring the possibility of BDD-style testing to Objective C, just as Rspec pioneered BDD-style testing in Ruby, but hasn’t eliminated Test::Unit. Choosing one or the other is largely a matter of style preferences.
In some cases we designed Cedar to overcome some shortcomings in the way OCUnit works for us. Specifically, we wanted to be able to use the debugger in tests, to run tests from the command line and in CI builds, and get useful text output of test results. These things may be more or less useful to you.
Long answer
Deciding between two testing frameworks like Cedar and OCUnit (for example) comes down to two things: preferred style, and ease of use. I’ll start with the style, because that’s simply a matter of opinion and preference; ease of use tends to be a set of tradeoffs.
Style considerations transcend what technology or language you use. xUnit-style unit testing has been around for far longer than BDD-style testing, but the latter has rapidly gained in popularity, largely due to Rspec.
The primary advantage of xUnit-style testing is its simplicity, and wide adoption (amongst developers who write unit tests); nearly any language you could consider writing code in has an xUnit-style framework available.
BDD-style frameworks tend to have two main differences when compared to xUnit-style: how you structure the test (or specs), and the syntax for writing your assertions. For me, the structural difference is the main differentiator. xUnit tests are one-dimensional, with one setUp method for all tests in a given test class. The classes that we test, however, aren’t one-dimensional; we often need to test actions in several different, potentially conflicting, contexts. For example, consider a simple ShoppingCart class, with an addItem: method (for the purposes of this answer I’ll use Objective C syntax). The behavior of this method may differ when the cart is empty compared to when the cart contains other items; it may differ if the user has entered a discount code; it may differ if the specified item can’t be shipped by the selected shipping method; etc. As these possible conditions intersect with one another you end up with a geometrically increasing number of possible contexts; in xUnit-style testing this often leads to a lot of methods with names like testAddItemWhenCartIsEmptyAndNoDiscountCodeAndShippingMethodApplies. The structure of BDD-style frameworks allows you to organize these conditions individually, which I find makes it easier to make sure I cover all cases, as well as easier to find, change, or add individual conditions. As an example, using Cedar syntax, the method above would look like this:
describe(@"ShoppingCart", ^{ describe(@"addItem:", ^{ describe(@"when the cart is empty", ^{ describe(@"with no discount code", ^{ describe(@"when the shipping method applies to the item", ^{ it(@"should add the item to the cart", ^{ ... }); it(@"should add the full price of the item to the overall price", ^{ ... }); }); describe(@"when the shipping method does not apply to the item", ^{ ... }); }); describe(@"with a discount code", ^{ ... }); }); describe(@"when the cart contains other items, ^{ ... }); }); });
In some cases you’ll find contexts in that contain the same sets of assertions, which you can DRY up using shared example contexts.
The second main difference between BDD-style frameworks and xUnit-style frameworks, assertion (or “matcher”) syntax, simply makes the style of the specs somewhat nicer; some people really like it, others don’t.
That leads to the question of ease of use. In this case, each framework has its pros and cons:
-
OCUnit has been around much longer than Cedar, and is integrated directly into Xcode. This means it’s simple to make a new test target, and, most of the time, getting tests up and running “just works.” On the other hand, we found that in some cases, such as running on an iOS device, getting OCUnit tests to work was nigh impossible. Setting up Cedar specs takes some more work than OCUnit tests, since you have get the library and link against it yourself (never a trivial task in Xcode). We’re working on making setup easier, and any suggestions are more than welcome.
-
OCUnit runs tests as part of the build. This means you don’t need to run an executable to make your tests run; if any tests fail, your build fails. This makes the process of running tests one step simpler, and test output goes directly into your build output window which makes it easy to see. We chose to have Cedar specs build into an executable which you run separately for a few reasons:
- We wanted to be able to use the debugger. You run Cedar specs just like you would run any other executable, so you can use the debugger in the same way.
- We wanted easy console logging in tests. You can use NSLog() in OCUnit tests, but the output goes into the build window where you have to unfold the build step in order to read it.
- We wanted easy to read test reporting, both on the command line and in Xcode. OCUnit results appear nicely in the build window in Xcode, but building from the command line (or as part of a CI process) results in test output intermingled with lots and lots of other build output. With separate build and run phases Cedar separates the output so the test output is easy to find. The default Cedar test runner copies the standard style of printing “.” for each passing spec, “F” for failing specs, etc. Cedar also has the ability to use custom reporter objects, so you can have it output results any way you like, with a little effort.
-
OCUnit is the official unit testing framework for Objective C, and is supported by Apple. Apple has basically limitless resources, so if they want something done it will get done. And, after all, this is Apple’s sandbox we’re playing in. The flip side of that coin, however, is that Apple receives on the order of a bajillion support requests and bug reports each day. They’re remarkably good about handling them all, but they may not be able to handle issues you report immediately, or at all. Cedar is much newer and less baked than OCUnit, but if you have questions or problems or suggestions send a message to the Cedar mailing list ([email protected]) and we’ll do what we can to help you out. Also, feel free to fork the code from Github (github.com/pivotal/cedar) and add whatever you think is missing. We make our testing frameworks open source for a reason.
-
Running OCUnit tests on iOS devices can be difficult. Honestly, I haven’t tried this for quite some time, so it may have gotten easier, but the last time I tried I simply couldn’t get OCUnit tests for any UIKit functionality to work. When we wrote Cedar we made sure that we could test UIKit-dependent code both on the simulator and on devices.
Finally, we wrote Cedar for unit testing, which means it’s not really comparable with projects like UISpec. It’s been quite a while since I tried using UISpec, but I understood it to be focused primarily on programmatically driving the UI on an iOS device. We specifically decided not to try to have Cedar support these types of specs, since Apple was (at the time) about to announce UIAutomation.
