What is Usability Testing?<\/h2>\n\n\n\n
Usability testing is a structured way to evaluate how real users interact with websites, mobile apps, software, or other digital products. <\/p>\n\n\n\n
The goal is to uncover issues in the design that frustrate, confuse, or trip up users, as well as potential opportunities to improve user satisfaction, efficiency, and confidence.<\/p>\n\n\n\n
At the center of every type of usability testing there are three core components:<\/p>\n\n\n\n
- \n
- Test<\/strong>: A carefully designed and controlled evaluation where participants attempt predefined tasks to assess usability.<\/li>\n\n\n\n
- Facilitator<\/strong>: The person who guides the usability test, observes user behavior, and asks follow-up questions to uncover qualitative data.<\/li>\n\n\n\n
- Participant<\/strong>: A real user who interacts with the product during testing, attempts tasks, and provides direct feedback (e.g. verbal comments, survey responses, or ratings) and indirect feedback (e.g. hesitation, errors, time on task, or click patterns).<\/li>\n<\/ul>\n\n\n\n
There may also be a defined role for observers who watch the test (live or recorded) to gain perspective, but don\u2019t interact with the participant.<\/p>\n\n\n\n
To provide a simple example, a usability test might focus on the checkout experience, and a facilitator would observe a participant as they tried to make a purchase. If they struggle entering information, figuring out where to select, and make errors that slow down the process, the facilitator is going to have tons of ideas about specific areas of the design that need attention.<\/p>\n\n\n\n
The capabilities and limits of usability testing<\/h3>\n\n\n\n
By systematically observing how real users interact with your site, usability testing can help you<\/strong>:<\/p>\n\n\n\n
- \n
- Identify pain points and flaws in your design<\/strong><\/li>\n\n\n\n
- Improve user flow and task efficiency<\/strong><\/li>\n\n\n\n
- Validate whether users understand your design<\/strong><\/li>\n\n\n\n
- Discover unexpected user behaviors and unmet needs<\/strong><\/li>\n<\/ul>\n\n\n\n
In spite of what people may ask you to discover, on its own usability testing can\u2019t help you<\/strong>:<\/p>\n\n\n\n
- \n
- Measure overall user satisfaction<\/strong><\/li>\n\n\n\n
- Test for demand or product market fit<\/strong><\/li>\n\n\n\n
- Find better performing designs<\/strong><\/li>\n\n\n\n
- Evaluate user preferences<\/strong><\/li>\n<\/ul>\n\n\n\n
Now usability testing can play a role in helping you increase user satisfaction, building a more usable design, and much more. But you can\u2019t go into usability testing expecting to get data on user preferences or expecting a more perfect design to emerge \u2014 usability highlights issues for fixing, it can\u2019t tell you how to fit them.<\/p>\n\n\n\n
The heart of the problem addressed by usability testing is that business owners and UX designers<\/a> have to make assumptions about how users will interact. They can make educated guesses, based on the best practices for website navigation<\/a>, web design, typography elements<\/a>, user psychology, and their own personal experience from designing in other contexts.<\/p>\n\n\n\n
No matter their level of talent or experience, designers have to rely on assumptions, and every assumption builds risk into the design. <\/p>\n\n\n\n
What usability testing does is provide real data on actual people using the product. Instead of working from assumptions and guesses, designers can base their decisions on objective, observable results. <\/p>\n\n\n\n
Is user testing the same thing?<\/strong><\/p>\n\n\n\n
It depends. You may see the terms \u201cusability testing\u201d and \u201cuser testing\u201d used interchangeably. You might also see people who use discrete definitions for both terms.<\/p>\n\n\n\n
In this post, I am only going to refer to usability testing because that\u2019s what it is. It\u2019s not a test of users, it\u2019s a test of the design\u2019s usability. The participants in the test who play the role of users are not being tested \u2014 they are the ones testing how efficient, intuitive, and easy to learn your design is, and where it falls short.<\/p>\n\n\n\n
My take: You can use whatever term you want, just be consistent about it.<\/p>\n\n\n\n
When is usability testing valuable?<\/h3>\n\n\n\n
Usability testing usually starts with a clear business concern: revenue is dropping, users are annoyed, churn is increasing, or key tasks take too long. <\/p>\n\n\n\n
In these cases, the goal is straightforward \u2014 identify why the design isn\u2019t working and what needs to be changed in order to fix the problem. <\/p>\n\n\n\n
But you don\u2019t have to wait for a problem to get started. Regular usability testing is valuable, and many companies use it proactively at key moments, such as:<\/p>\n\n\n\n
- \n
- During prototyping and design<\/strong>: Testing early helps catch usability issues before they become costly to fix. You can validate design concepts and ensure user expectations align with functionality.<\/li>\n\n\n\n
- Pre-launch<\/strong>: Running usability tests before rolling out a new product, feature, or update is a good way to reduce the risk of shipping something confusing or broken<\/li>\n\n\n\n
- Before and after a redesign<\/strong>: Testing before a redesign quantifies existing usability pain points. Testing after ensures your changes actually make things better\u2014not worse.<\/li>\n\n\n\n
- Annual benchmarking<\/strong>: Companies run usability testing annually (or more often for complex sites and apps) to track whether design changes improve or degrade the user experience.<\/li>\n\n\n\n
- Comparing different designs<\/strong>: You can test different versions of your own site, variations of a product, or even compare against competitor apps to identify usability gaps and opportunities.<\/li>\n<\/ul>\n\n\n\n
Another way to think about when to employ usability testing is to think about it as part of the development process. In the words of Garry Tan, usability testing helps you \u201cfigure out what\u2019s broken before you build<\/a>,\u201d and forms an essential part of the product development lifecycle.<\/p>\n\n\n\n
![\"Post](\"https:\/\/ceblog.s3.amazonaws.com\/wp-content\/uploads\/2020\/05\/10133937\/Usability-Testing-Garry-Tan-Post.png\")
<\/figure>\n\n\n\nAs the CEO of Y Combinator<\/a>, the longstanding silicon valley incubator that has funded some of the world’s most successful startups, Tan understands better than most the risks of neglecting usability testing.<\/p>\n\n\n\n
Usability testing formats<\/h3>\n\n\n\n
This is a quick overview of your options for structuring a usability test. Each of the three formats comes with clear benefits and costs.<\/p>\n\n\n\n
In-Person Monitored<\/strong>: This is live, \u201cin-lab\u201d usability testing where a facilitator sits near the participant, observing their interactions and potentially asking follow-up questions. Participants may be encouraged to think aloud while engaging with the product. Post-task questions, surveys, or post-test interviews may be included \u2014 or the test may be designed to be more hands\u2019 off, with fewer interruptions.<\/p>\n\n\n\n
- \n
- Benefit<\/strong>: Provides the richest, highest-quality data. The facilitator can observe body language, catch subtle hesitations, and dig into the \u201cwhy\u201d behind user behaviors in real-time.<\/li>\n\n\n\n
- Cost<\/strong>: Most expensive, even as a DIY effort. Recruiting the right participants, setting up a controlled testing environment, hiring trained facilitators, and using testing software all add up. Many companies outsource in-person usability testing to vendors with established recruiting pipelines, controlled locations, and trained facilitators.<\/li>\n<\/ul>\n\n\n\n
Remote Monitored<\/strong>: This is a synchronous (real-time) usability test conducted remotely via screen-sharing and video conferencing software<\/a>. A facilitator observes the participant\u2019s interactions, asks follow-up questions, and may request them to think aloud while completing tasks.<\/p>\n\n\n\n
- \n
- Benefit<\/strong>: It offers many of the same benefits as in-person testing, such as real-time probing into usability issues, but without the challenges of setting up a physical testing space or the geographic limitations of recruiting participants.<\/li>\n\n\n\n
- Cost<\/strong>: More affordable than in-person testing but still requires scheduling, trained facilitators, and reliable remote testing tools. Companies may outsource the entire testing process, or some of the responsibilities such as recruiting participants.<\/li>\n<\/ul>\n\n\n\n
Remote Unmonitored<\/strong>: This is an asynchronous usability test where participants complete assigned tasks on their own, typically on a specialized usability testing platform that records their interactions. Along with calculating metrics like task completion rate, platforms collect feedback through automated surveys, session recordings<\/a>, and heatmaps.<\/p>\n\n\n\n
- \n
- Benefit<\/strong>: Compared to other formats, this is easier and faster to scale, which is important if you are interested in statistically significant results. It enables quick feedback from a broad audience without the need for real-time facilitation.<\/li>\n\n\n\n
- Cost<\/strong>: Significantly less expensive than other formats, but the real cost is the lack of real-time follow-up questions, which puts a low ceiling on the potential for qualitative data. Without a facilitator, it\u2019s harder to interpret why users struggle with certain tasks.<\/li>\n<\/ul>\n\n\n\n
Usability testing terminology<\/h3>\n\n\n\n
You should understand and be able to explain all of the terminology involved with usability testing. What is a user task? What is a user need? And how \u2014 precisely \u2014 are these concepts different?<\/p>\n\n\n\n
If you have any sort of mushy conception about usability testing terminology, it will cause tons of problems for your team (who won\u2019t be speaking the same language) and during meetings where you will struggle to explain your rationale to stakeholders.<\/p>\n\n\n\n
Below, I\u2019ve laid definitions for key usability testing terminology. I\u2019m not going to claim these definitions are perfect, but they are logically consistent and aligned with accepted UX best practices.<\/p>\n\n\n\n
Let’s start with terminology specific to usability testing:<\/p>\n\n\n\n
- \n
- Goal<\/strong>: This is what the user wants to accomplish, the outcome they want by the end of the process.<\/li>\n\n\n\n
- Intended path<\/strong>: The designed course of action necessary to reach the goal.<\/li>\n\n\n\n
- Needs<\/strong>: These are requirements necessary to get to the goal.<\/li>\n\n\n\n
- Tasks<\/strong>: These <\/strong>are behaviors necessary to get to the goal.<\/li>\n\n\n\n
- Pain points<\/strong>: These are obstacles and points of friction that block a user from completing tasks, need to be overcome, or stop the user from completing a task.<\/li>\n\n\n\n
- Errors<\/strong>: These are mistakes or deviations from the intended path of completing the task.<\/li>\n<\/ul>\n\n\n\n
Let\u2019s take a look at an example that uses this terminology:<\/p>\n\n\n\n
A facilitator observes a participant try to accomplish the goal <\/strong>of paying a bill online. The participant needs<\/strong> include their credit card info and login credentials. They must complete several tasks<\/strong>, such as entering their account number, authenticating their identity, creating a new payment, and entering their credit card information. Pain points<\/strong>, such as a long payment form with unnecessary fields result in errors <\/strong>that require the participant to re-enter the same information multiple times.<\/p>\n\n\n\n
Now let’s look at terminology specific to the analysis of usability testing: <\/p>\n\n\n\n
- \n
- Observations<\/strong>: User behaviors you can see but don\u2019t yet understand. You know what users did, but not why they did it, what they were trying to accomplish, or how they felt.<\/li>\n\n\n\n
- Findings<\/strong>: Patterns or facts uncovered during testing that have some explanation. For example, 5 of 7 users couldn\u2019t find the bill pay button, or users explicitly said they were frustrated searching for it.<\/li>\n\n\n\n
- Insights<\/strong>: Findings with clear causes and implications. For example, users can\u2019t find the bill pay button because it blends in with other elements, leading to frustration and abandoned payments.<\/li>\n\n\n\n
- Motivations<\/strong>: The underlying reasons driving user actions. For example, avoiding late fees may motivate someone to pay bills online. Motivations aren\u2019t always conscious.<\/li>\n<\/ul>\n\n\n\n
If you are running remote, unmonitored usability testing, you are not likely to get insights. How could you? You simply aren\u2019t getting the kind of qualitative usability testing data you need to understand the causes of user errors that you observed.<\/p>\n\n\n\n
The reason I bring this up is that many people (especially those that are unfamiliar with UX) want to treat observations and findings as if they are insights. They are not, and if you are the person who owns usability testing, you are going to have to police this.<\/p>\n\n\n\n
For example, someone comes in and says \u201cWe saw users do X, we think it\u2019s because of Y, and so we\u2019re going to do Z.\u201d<\/p>\n\n\n\n
Okay \u2014 what are the assumptions here? Why are you sure that it\u2019s because of Y and not P or Q or anything else? How confident are you that X is because of Y?<\/p>\n\n\n\n
Really push people to show their work if they are proposing fixes for the usability issues uncovered during testing. Don\u2019t let them pass off findings as insights and sneak assumptions into your design that add to the risk<\/em> of frustrating users instead of mitigating it.<\/p>\n\n\n\n
There are ways to supplement that data you get during usability testing that can help you turn observations into the insights you need to be confident making changes to your design. We\u2019ll turn to those next.<\/p>\n\n\n\n
Ways to augment data during usability testing<\/h3>\n\n\n\n
If you’re going through the effort of recruiting and compensating participants, adding these supplemental methods can surface valuable insights, sometimes with minimal extra effort.<\/p>\n\n\n\n
Here are some of the popular methods that researchers gather richer data during usability testing:<\/p>\n\n\n\n
- \n
- System Usability Scale (SUS)<\/strong>: A standardized 10-question survey that provides a quick, reliable measure of perceived usability. There’s tons of SUS data out there, making the survey useful for benchmarking and comparing designs.<\/li>\n\n\n\n
- Single Ease Question (SEQ)<\/strong>: A simple, one-question survey that asks users to rate task difficulty immediately after completing a task. This is good for identifying frustration points without interrupting the test flow.<\/li>\n\n\n\n
- NASA Task Load Index (TLI)<\/strong>: This is a time-consuming survey that assesses the perceived workload of each task across six dimensions, such as mental demand, effort, and frustration level. It is widely used for evaluating complex tasks.<\/li>\n\n\n\n
- Eye tracking<\/strong>:Specialized software or hardware that tracks exactly where users look on a screen. It helps researchers understand visual attention patterns, how users scan, whether they notice key elements, and so on.<\/li>\n\n\n\n
- Post-test interviews<\/strong>: Typically short, structured conversations after usability tests to gain additional feedback on user decisions, frustrations, expectations, and motivations. Long user interviews can also be extremely useful, though they are not the norm.<\/li>\n\n\n\n
- A\/B testing<\/strong>: This is not a usability testing method per se, but running A\/B testing<\/a> can complement usability test findings by figuring out which design variations lead to better performance in real-world usage.<\/li>\n\n\n\n
- Heatmaps and click tracking<\/strong>: Tools in this category create visualizations from analytics data that show where users engaged, clicked, and scrolled. Heatmaps<\/a> are helpful for validating findings from usability testing and comparing performance between versions of a product or site.<\/li>\n<\/ul>\n\n\n\n
Not all of these supplemental methods will be valuable. It depends on your goals for testing. That said, it\u2019s hard to see how implementing surveys to gather qualitative data would be unhelpful in making decisions about how to improve the design.<\/p>\n\n\n\n
How To Run Usability Testing<\/h2>\n\n\n\n
Usability testing is not glamorous, but this process gets results \u2014 better user experiences and fewer frustrations. Here are the essential steps:<\/p>\n\n\n\n
- \n
- Define the goal: <\/strong>Start by clearly outlining the specific objective of the usability test. Focus on identifying areas of friction, confusion, or inefficiency within the user experience. Tie this goal to measurable outcomes, such as increasing sign-ups or improving task efficiency.<\/li>\n\n\n\n
- Determine what to test<\/strong>: Identify the specific user journey or interaction to evaluate, such as site navigation or the checkout process. Ensure the test\u2019s scope aligns with the business goal. Prioritize the elements to test, focusing on areas that have the most significant impact.<\/li>\n\n\n\n
- Select the testing format<\/strong>: Choose the appropriate format based on your goals and resources. Decide between in-person, remote, moderated, or unmoderated testing, considering budget and participant availability.<\/li>\n\n\n\n
- Build an accurate test<\/strong>: Create a functional prototype or flow that mimics real-world usage. Develop clear, actionable tasks that align with actual user behavior to ensure meaningful results.<\/li>\n\n\n\n
- Recruit participants<\/strong>: Select participants who match demographics and psychographics<\/a> of your target audience. For smaller tests, recruit 5-10 users; for larger tests, consider 20-50 participants to ensure statistical significance.<\/li>\n\n\n\n
- Conduct the usability test<\/strong>: Guide participants through the tasks, observing their behavior and collecting feedback. Use screen-recording software to track actions, and, if possible, ask participants to verbalize their thoughts during the test for richer insights.<\/li>\n\n\n\n
- Collect supplemental data (optional)<\/strong>: Enhance your findings with additional data, such as surveys, eye tracking, or post-test interviews. These tools can provide deeper context for the usability issues identified.<\/li>\n\n\n\n
- Analyze the results and take action<\/strong>: <\/strong>Review the test results, identifying key usability issues and patterns. Prioritize fixes based on their impact and feasibility, ensuring the insights directly inform product or design updates.<\/li>\n<\/ol>\n\n\n\n
Even in its most simple and streamlined forms, usability testing is a lot of work to do properly. And there is zero, absolutely zero value in shortcutting any of these steps. It will skew the data in a direction that cannot possibly be accounted for. <\/p>\n\n\n\n
- Determine what to test<\/strong>: Identify the specific user journey or interaction to evaluate, such as site navigation or the checkout process. Ensure the test\u2019s scope aligns with the business goal. Prioritize the elements to test, focusing on areas that have the most significant impact.<\/li>\n\n\n\n
- Single Ease Question (SEQ)<\/strong>: A simple, one-question survey that asks users to rate task difficulty immediately after completing a task. This is good for identifying frustration points without interrupting the test flow.<\/li>\n\n\n\n
- Findings<\/strong>: Patterns or facts uncovered during testing that have some explanation. For example, 5 of 7 users couldn\u2019t find the bill pay button, or users explicitly said they were frustrated searching for it.<\/li>\n\n\n\n
- Intended path<\/strong>: The designed course of action necessary to reach the goal.<\/li>\n\n\n\n
- Cost<\/strong>: Significantly less expensive than other formats, but the real cost is the lack of real-time follow-up questions, which puts a low ceiling on the potential for qualitative data. Without a facilitator, it\u2019s harder to interpret why users struggle with certain tasks.<\/li>\n<\/ul>\n\n\n\n
- Cost<\/strong>: More affordable than in-person testing but still requires scheduling, trained facilitators, and reliable remote testing tools. Companies may outsource the entire testing process, or some of the responsibilities such as recruiting participants.<\/li>\n<\/ul>\n\n\n\n
- Cost<\/strong>: Most expensive, even as a DIY effort. Recruiting the right participants, setting up a controlled testing environment, hiring trained facilitators, and using testing software all add up. Many companies outsource in-person usability testing to vendors with established recruiting pipelines, controlled locations, and trained facilitators.<\/li>\n<\/ul>\n\n\n\n
- Pre-launch<\/strong>: Running usability tests before rolling out a new product, feature, or update is a good way to reduce the risk of shipping something confusing or broken<\/li>\n\n\n\n
- Test for demand or product market fit<\/strong><\/li>\n\n\n\n
- Improve user flow and task efficiency<\/strong><\/li>\n\n\n\n
- Facilitator<\/strong>: The person who guides the usability test, observes user behavior, and asks follow-up questions to uncover qualitative data.<\/li>\n\n\n\n