The Simple Guide to Product Testing

I want to start by saying that we’re testing [the product], and not you. You can’t do or say anything wrong, and nothing is going to hurt our feelings.

…at least that’s what we say to every single user during our in-person testing sessions. And we mean it. We’re not actually testing the user, right? We’re testing the product, regardless of the stage it’s in — from a concept to a healthy, released product.

When we’re at the very beginning of a new project and validating a product idea through testing, we are miles away from proper usability testing (which we do later in the project). This, for instance, is just one of the reasons we don’t refer to these sessions as usability testing nor user testing, but rather — product testing.

These guidelines are not only for teams already building products but also for anyone that just started to think about new product ideas and ways to validate them.

While there are multiple things to keep in mind when planning and conducting a product testing, here’s a simple list that can help you focus:

1. Start testing the very first week of the project
2. Do more than “just” usability testing
3. Five users per target group
4. Do it more than once
5. Take care of the details

1. Start testing the very first week of the project

As an agency, we start new projects with a short discovery / research phase followed by a hands-on workshop called Design Sprints. Sprints are great when kicking off new initiatives, testing new product ideas, features, etc. We end the workshop by creating a medium fidelity prototype (a rapid process for us with tools like Sketch and InVision) that we then test with users. We want to validate the direction we’re heading at the very start.

Can you imagine how rewarding that is? There was almost nothing a couple a days ago — now there is a concept, a business case, and actual feedback from real customers.

What a great way to start a project.

2. Do more than “just” usability testing

The priorities in testing sessions change as the project evolves. We might start out really heavy on validating the product idea and prioritizing features, but as we progress, we focus less on that and more of how things come together.

Let’s discover how the testing focus shifts throughout the project:

1. Concept
   Validate the product idea with the desired target audience
2. Features
   Prioritize features (especially when building an MVP)
3. Information architecture
   Test out different approaches to information architecture
4. User Flow
   How proposed flows affect the user experience and behaviour
5. Visual Design
   A series of tests related to visual design, tone & voice, readability, branding, naming, micro-copy, etc…

3. Five users per target group

You’ve heard this a million times — 5 participants in a testing session is enough to get the majority (circa 85%) of the feedback you need at that point.

What’s important to note is that’s actually five users per a target group, meaning if you have a primary and secondary type of users, you need to multiply that number.

Identify people who match your demographics (or even better: match your personas) and then screen for behavioral traits, attitudes, and goals that match those of your users. —NN

We tend to always have six humans recruited (plus floaters available for morning and afternoon sessions). The tendency to do more than five comes from experience — sometimes users that show up aren’t a good match, sometimes the A/V has issues or there is a fire drill (happened more than once!).

4. Do it more than once

A very important thing for the entire team to realize is that early concept testing is by no means enough. Even though a project is underway, there should still be regular testing sessions as the designs get more fidelity, longer flows, richer features… as the final UX prototype has been created, as we jump into development.

The most important thing is to guide those tests towards the desired outcome. For instance — in testing the product in the visual design phase, it’s important for the tests, assumptions and goals to focus on items directly related to this phase. While I’m positive you will encounter feedback related to other things that what you intended to test (everything from product fit to the actual user experience), capture that feedback and pencil it into the product roadmap.

5. Take care of the details

As always in life, the details are not the details. We’ve seen completely different testing results when one or more of details were not executed with care. It’s important to keep the story we’re telling to users alive and not to break the flow.

Our user needs to stay in the shoes we’ve asked her to put on.

1. Use real content in your prototype
   These users were sourced because they are the target audience — for instance, they frequently order food on mobile, they have previously learned Spanish, etc. They will definitely look at content in your prototype and sometimes be blocked by an easily solvable gap or error.
2. Know thy flow
   You are effectively guiding the user through a path without them knowing where they are in the flow — e.g. how many steps or screens there are left. You need to guide them, and more importantly, you need to have a mental map of the flow in your head if something goes unplanned (as it often does). It’s always important to know what’s the primary focus on each screen so that the study goals can be properly validated.
3. Know your hardware
   Set yourself up for success and don’t let “typical” hardware behaviors get in the way. I’m talking about screensavers, brightness levels, notifications, other apps, etc.
4. Know your software
   Prototyping tools and behind the scenes interactions can sometimes conflict with intended actions for our users. For instance — Swiping on mobile is (obviously) intuitive for users, even though a prototyping software might interpret this as [navigate-to-next-screen] rather than interacting with the screen the desired way. Although you envisioned the study user flow in one way, this one simple “feature” can disrupt the organic navigation through the screens. Bare that in mind when creating the ideal setup. If you’re using InVision, you can work around this by disabling the “Explore all screens” feature.

At the end of the day

The most striking truth of the curve is that zero users give zero insights.

If you’re doing product design, you should do product testing as well. We hope this guide helps you in this process.

Originally published on InVision blog.