Never Too Many Devices

Mobile software testing involves many variables such as operating system versions, types of connectivity, networks, devices, and environments. To account for so many variables, the approach to testing apps poses an interesting dilemma: Should testing be performed on physical devices, simulators, emulators, or cloud-based devices? Is there a unique value in sticking to physical devices? At Bottle Rocket, we use a combination of these with an emphasis on physical devices—and we do mean emphasis. We recently surpassed 400 test devices in our library.

Device Alternatives?

Users expect apps, including those with the ability to sync between devices with shared accounts, to work flawlessly on their devices. So, a comprehensive library of devices to mimic the user experience is essential in testing. However, the number of combinations of devices and operating systems that users have is huge. For this reason, many companies opt to rely mostly on simulators, emulators, and/or cloud-based devices for their software testing.

These device alternatives can be very helpful in the beginning of the application development lifecycle, but they are not ideal as the sole testing resource. Many important pieces of mobile interaction are lost when physical devices are left out.

Real-world users are equipped with real devices in different environments, meaning the effects of the following items may vary:

• Battery life
• Network conditions
• Memory limitations
• Location services
• Bluetooth
• NFC
• Device orientations
• Accelerometer
• Operating systems
• Peripherals

Recreating the user experience as closely as possible is what Bottle Rocket strives for. In our efforts to do this, we recently surpassed four hundred test devices in our QA arsenal! This is a milestone in itself since one of the biggest challenges that exists in application testing is the cost of obtaining and maintaining such a large volume of physical devices for testing purposes.

Bottle Rocket Setup

Our device lab contains Griffin cabinets with a capacity to hold and charge up to eighty devices at a time. Rocketeers pick out the devices they need, check them out library-style with our device checkout software, and begin their testing. All Rocketeers can access the checkout software remotely, so devices can be easily located and accounted for conveniently.

This system allows testers to plan ahead and dedicate certain devices to their project for as long as they are needed. Once a project starts, the lead tester maps out which devices will be needed during the testing process. They are checked out from the device cabinets and used throughout the project, only being returned when all testing is completed. When finished, checking the devices back in is as easy as pushing a button.

Maintainability

Solely relying on physical devices to test requires a great deal of time and money. Maintaining such a library involves frequent upgrades in order to remain relevant and effective. If our system doesn’t contain the latest and greatest mobile devices, a big chunk of tech-savvy users and their experiences will be unaccounted for. We need to remain ahead when it comes to testing on the latest hardware. We pay close attention to mobile announcements such as Google I/O and WWDC every year to gather intel on what’s coming and plan ahead by preordering any new devices or scoping out the latest operating system and its release date. So why all the focus on what’s coming? It’s important to know the adoption trends, new hardware features, such as aspect ratios, or if any new peripherals are announced.

New operating systems are especially impactful as they can touch any part of the device and installed applications. New features and tweaks to existing operating systems can affect corresponding areas of our applications, so we must look into the patch notes of any release and test our applications accordingly. A recent example of this is the release of iOS 11 – for every new beta released, we update our device operating systems and run tests on all of our applications. It’s a quick sanity check to verify all features cooperate with the updates. Developers have taken advantage of these betas as well in order to research and experiment with the new ARKit.

It is beneficial to have varying device types - high end and low end, new and old. Keeping old operating systems on devices is part of this method. Android users especially are not pushed to upgrade operating systems as aggressively as other platforms. Android OS 6 is still the most widely used according to Android’s developer portal, despite OS 7 having been out for more than a year. Covering OS 4, 5, 6, and 7 is necessary to ensure quality for all users.

In summary, using physical devices will assure the most accurate testing and analysis since that is the only way to duplicate the diversity of users’ environments, network settings, operating system versions, and hardware. Keeping up with current and future devices and operating system versions will also ensure quality results. Bottle Rocket will continue to build its device library – we’re beyond 400 now, and have no plans of stopping!

Interested in testing your app in our device library? Let us know at [email protected]