Scratching the Surface – Why Testing on Mobile is Different

Written by: on March 14, 2017

Every design task reaps benefits from doing user testing. No matter which design space you practice in, user testing provides valuable feedback you cannot get from inside the walls of your design team.

However, when it comes to mobile, user testing is a bit more complicated, and the primary reason is in the name: mobile. Mobile design exists everywhere, on many device sizes and platforms. Even when we are targeting a niche market and know our users well, we will never be able to predict when and where our designs will come to life.

To understand why testing on mobile is different, you have to understand the platform. Designers who live and breathe mobile know that their work is influenced by the platform, and should be. Apple and Google share their user interface guidelines to give us a strong base to build on. We are given well-researched documentation crafted by the leaders of UX design. Throwing this out would be comparable to throwing out the rules of Gestalt. We design for users that know their platform and have learned how to navigate it. For instance, navigating a segmented controller in iOS or tapping a floating action button in Android becomes second nature. Designing on mobile requires you to begin with a good foundation of these guidelines. You want your app to feel familiar the first time users open it. When creating an application that strays away from these guidelines, designers should ask themselves:

“What value am I adding by making this different?”

This will help you determine if your design execution benefits or hinders the user’s experience. There are times when straying from the guidelines makes sense. Consider a situation where the loading time will be longer than normal and a standard progress indicator fails to keep the user engaged. Using an unorthodox transition to hold a user’s attention is better in this case, but should be recognized as a custom transition that needs to be tested. By doing this, you are able to address these discrepancies in user testing in order to check their effectiveness. Although there might be other goals when your design enters the user testing phase, deviations from platform guidelines should always be tested.

Once you determine what the focus of your user testing will be, it’s time to decide what type of testing will be the most beneficial. Services available online have made user testing more attainable for designers. These services work with the prototyping software you build in (InVision, Proto.io, etc.) and distribute your prototype to users remotely or provide you with a link to distribute to users yourself. This option allows you to circulate your prototype with little effort, giving you more quantitative results. Using services to perform testing tends to be more effective in the final stretches of a design launch and soon thereafter. Getting numerical results rather than an in-depth analysis of user flows provides a design team with the validation they need to make small tweaks and changes before and after a release.    

If you intend to use a testing service to carry out your testing needs, it’s important to research the service you’ve chosen. Some software will only test a functional prototype while others are able to test basic wireframes. For the sake of simplicity, let’s focus on a scenario where you want to test a prototype in the early wireframe stage. Here are a couple of services that support mobile testing in the early prototyping phase:

Lookback – 

Pros:

  • Records user and screen while they interact with the prototype
  • Available on both iOS and Android devices
  • Works with InVison, Proto.io and Marvel prototypes
  • Low monthly fee of $29/month for unlimited testing

Cons:

  • Designers have to find their own testers

Lookback is a barebones testing service that gives you the most bang for your buck. With a low risk monthly fee that doesn’t charge you per test, you are able to distribute your prototype to as many or as few users as necessary. All this sounds great, but if your team doesn’t use a recruiting agency or have an experienced team member to handle the task of finding users, I would consider another option.  

Validately – 

Pros:

  • Hear and see users interact with your prototype
  • Supports Invision, Marvel and Axure prototypes
  • Offers moderated and unmoderated options
  • Offers a range of recruiting options
  • Provides analytics and reporting

Cons:

  • This service can get pricey, with options ranging from $199-$399/month

Validately give you a variety of choices when it comes to test moderation and recruiting. They provide you with 36 studies (group of recordings) per year, but give you the option to add another 36  studies for an extra $99 dollars. This is definitely a long term and more expensive solution, but gives you more wiggle room to customize different testing scenarios you encounter.

Loop11 – 

Pros:

  • Provides extras like heat maps and path analyses
  • Once tests have been completed, reporting is done for you
  • Fair price of $149-$199/month

Cons:

  • You have to find your own users, although Loop11 provides some resources to help with that
  • Geared towards website testing

This program provides a middle of the road option as far as pricing goes. Although this platform is geared towards website testing and consulting, it recently added support for mobile prototyping using any interface that can be built in InVision, Axure or JustInMind.

UserTesting – 

Pros:

  • Works with most prototyping tools including Invision, Marvel, Flinto, Axure, Justinmind and proto.io
  • Results are available soon after testing has been completed
  • Gives you the option to build your own test, or have them assist you with planning, moderating and creating presentations for your testing results

Cons:

  • Expensive – the individual plan will run you $49 per test, while the enterprise plan varies

The most common service when it comes to user testing. A great option if your company uses more than one prototyping tool to carry out user testing. Both plans offer access to basic testing capabilities, but you have to opt for the enterprise plan in order to unlock more extensive research options.

If you want more qualitative feedback on your prototype, consider taking user testing into your own hands. For the right project, it makes sense to have users on-site so you can moderate their testing. This allows you to have more control over testing variables, giving you the ability to shift the focus of the test if needed. However, this does place the burden on the internal team of having to find users. Recruiting participants to test your app should not be taken lightly, the type of users you recruit will have the most influence on your test results.

A large reason why you may want to consider running your own user testing when designing for mobile is the ability to control the lab environment. Desktop and mobile are used in different situations and circumstances. Desktops are stationary, and while laptops are more mobile, they move around in similar environments. This may sound minuscule, but consider the function of a fitness app. Environmental conditions will be different for each person, and each time they use the app.

Let’s take a simple list-building app into consideration. Suppose that I have prototyped an app that enables users to add items/tasks to a list and complete them. A user can create a grocery list, add homework items, workout routines, etc.  A mobile application will most likely have more than one primary environment, much like our list building app. To improve testing results on mobile, designers should have testers perform tasks in primary environments in order to see how users interact with their app on the go. In a perfect world, extending the testing timeline to include multiple environments would not be an issue, but if scope doesn’t allow for that, making a few adjustments to a typical testing process could help. One option could be recruiting less users to allow more time for each user to test the application in multiple environments. Another option could be trimming the tasks that users complete. Consider focusing on one or two specific tasks for the user to run through, allowing for more time to switch environments. Whatever your solution is to accommodate multiple environments, when testing a mobile application, environments should be your first consideration since they have the most influence on how your user interacts with your design.

No matter how you decide to test your application, remember that there are challenges unique to mobile that require creative problem solving. User testing is just one step in the process to creating a user-centric design. Platform guidelines, device sizes and environments are only a sample of variables that need to be heavily weighed in each step of the design process.    

If you’re interested in hiring POSSIBLE Mobile for UX/UI mobile design, please fill out the form below.

Contact Us

We would love to hear from you! Please fill out this form and we will get in touch with you shortly.
Meghan Dever

Meghan Dever

Meghan is a UX/UI designer at POSSIBLE Mobile. She has been creating new mobile experiences at POSSIBLE Mobile since 2013, and has worked on several major brands that have been featured on the iOS App Store and Google Play Store.

Article


Add your voice to the discussion: