Usability testing is missing from the traditional instructional design process. But not having a way to find problems is a problem. We need to make sure that eLearning courses, learning portals, online performance support, chatbots and other digital products aren’t obstacles to learning. The think aloud method is perhaps the best way to discover problems with the user interface, navigation, layout, and instructions.
What is the Think Aloud Method?
The think aloud method provides a window into the minds of learners and users. This method has uses in education, cognitive psychology, product design, engineering, and other fields. In this article, I’m focusing on how learning designers can employ the think aloud technique for usability testing. This can help us ensure our digital products are user friendly.
Usability testing refers to identifying the problems with a product or service by testing it with sample users from the target audience. The think aloud protocol requires a facilitator or researcher and a participant. The facilitator asks the participant to narrate everything that goes through their mind concurrent with completing a task. It’s is one of those obvious yet brilliant techniques for understanding how another person thinks.
For example, during the last usability test I recorded, the user identified a problem with the navigation. She reported what she was thinking. “Where is the next button? How do I get to the next slide? Oh, I guess I click this over here.”
It is not always easy for a participant to remember to think aloud. The facilitator my need to prompt the participant with a reminder. Some participants may prefer to interpret or analyze their thoughts rather than narrating them. In those instances, the facilitator will remind the participant to simply report what they are thinking.
When To Run Usability Tests for Learning Design
The best time to run a test is early in the design process before you get too far along. This could be when you have a solid prototype or when a short but representative portion of your product is complete. Beware that identifying problems may mean reworking the structure, user interface or other time consuming revisions. So if you are planning on usability testing, do it sooner rather than later. If you’re unfamiliar with prototyping, see What should I put in my eLearning prototype?
Types of Think Aloud Protocols
The think aloud protocol is popular with user experience designers to identify design deficiencies. In a recent usability study, thinking aloud was the usability testing technique of choice with 86% of respondents (Fan, Shi & Truong, 2020). The other two top methods were interviews at 60% and heuristic evaluations (reviews by experts) at 41%.
There are two common types of think aloud usability tests: concurrent verbalization and retrospective. Participants complete a task and narrate what is going through their mind. In the retrospective think aloud method, participants report on their experience after task completion. Concurrent verbalization is our focus here.
How to Run a Think Aloud Usability Test
- Goals. Identify the goals of the usability test.
- Recruitment. Recruit diverse participants from your target audience. Ideally, they will have varied job roles, backgrounds, educations, etc.
- Tasks. Design realistic tasks for the participant to complete. In learning products, this might be activities such as asking participants to:
-
- Sign in to the LMS, find course XYZ and launch the course.
- Go through Lesson 2 and complete the activities.
- Complete the self-check and email the results to yourself.
- Find the resource on graphic design and download it.
- Script. Write a script to read at the start of the test. Start with a friendly introduction to help the participant feel comfortable. You may want to start with a statement like, “We are testing the product, not you.” Introduce yourself and ask about the person’s job or role. Explain how the test will work. You may also want to show a demonstration of the technique. Jakob Neilson has one here: Demonstrate Thinking Aloud by Showing Users a Video.
- Run. Run the test in person or using a video conference tool. Be prepared to prompt the participant to only report what they are thinking.
- Insights. Gather your data in one place and look for patterns.
- Present. Present your findings to help everyone have a common understanding. This is typically your team and stakeholders.
- Revise. Revise your learning product to alleviate the problems that participants identified.
How Many People to Test
The number of people to test for problem discovery is debated among UX professionals. UX expert Jakob Nielsen states that after five participants, the same problems begin to show up. Contrary to that opinion, Six and Mayfield (2016) state that there is no one-size fits all number and it depends on the complexity of the product. See How to Determine the Right Number of Participants for Usability Studies. You will probably need to try out the technique to determine when problem identification becomes repetitive. I’m guessing that in many cases, five participants will make a world of difference in finding problems with your design.
Making the Think Aloud Protocol Accessible
Visually Impaired Users. If you are running a think aloud test with visually impaired users who rely on screen readers, you will need to run the test differently. That’s because the user will first need to listen to the screen reader device before speaking. In these cases then, it is possible to use a technique called partial concurrent thinking aloud (PCTA). The technique uses a modified set of concurrent verbalization plus retrospective analysis. See Web Usability with Screen Readers.
Deaf and Hard of Hearing Users. One viable method to use with deaf and hard of hearing users is the gestural thinking aloud protocol (GTAP). Using this approach, there is a facilitator and an interpreter on hand with the participant. Similar to the approach with hearing individuals, participants are asked to sign the thoughts they are having while completing a task. In one study, this produced mostly similar results to testing with users who reported their thoughts orally (Roberts & Fels 2005).
Pros and Cons of the Think Aloud Method
In 1998, Jakob Neilson wrote, “Thinking aloud may be the single most valuable usability engineering method.” And he stands by that assessment today. That’s because there are many advantages to the think aloud protocol:
- It is inexpensive to run.
- You can conduct tests in-person or virtually.
- It’s easy to learn how to facilitate using the think aloud method.
- With a script in hand, most people can manage this technique.
What are the problems with the think aloud method?
- Narrating their thoughts may make participants uncomfortable.
- Thinking aloud may put more demand on a person’s cognitive resources than completing a task in silence.
- If a facilitator asks a qualifying question, it could bias the person’s actions.
Conclusion
Although the think aloud technique is not perfect, it is probably the best approach for discovering problems with a learning design. Using this method with just a few participants is likely to make your digital learning products easier to use and understand. Hopefully, this will improve learning.
References:
- Blummer, B. & Kenton, J. (2014). Methodology: the think-aloud problem-solving activity and post-activity interview in Improving Information Search: A Metacognitive Approach, 113-123.
- Fan, M., Sh, S.i & Truong, K. (2020). Practices and Challenges of Using Think-Aloud Protocols in Industry: An International Survey. Journal of Usability Studies, Vol. 15, Issue 2.
- Güss, D.C. (2018). What Is Going Through Your Mind? Thinking Aloud as a Method in Cross-Cultural Psychology. Frontiers in Psychology.
- Roberts, V. & Fels, D. (2005). Methods for inclusion: Employing think aloud protocols in software usability studies with individuals who are deaf. International Journal of Human-Computer Studies, 64 489–501.
- Steffano, F., Borsci S., & Stamerra, G. (2010). Web usability evaluation with screen reader users: implementation of the partial concurrent thinking aloud technique. Cognitive Processing, 11:263–272.
Wonderful! Let me know how it goes.
Connie
Thanks Connie.
I’m going to try this with the training videos we are creating.
Thank you Yvonne!
Dr. Andrea Gregg and colleagues wrote a chapter on applying think-alouds to course design. Definitely worth the read. https://edtechbooks.org/ux/15_think_aloud_obser