One of the biggest criticisms of multiple choice questions is that they only test factual knowledge. But it doesn’t have to be that way. We can also use multiple choice questions to assess higher-order thinking.
Higher Order Thinking in a Nutshell
Higher order thinking goes beyond memorizing and recalling facts and data. It even goes beyond comprehension. Higher-order thinking refers to cognitive processes that involve analytical, critical and creative thinking. The concept is based on various learning taxonomies, such as application, analysis, evaluation, creation, problem-solving, connecting ideas and making decisions. Keep in mind that many of these cognitive tasks, including the recall of information, appear to occur simultaneously. See Alternatives to Bloom’s Taxonomy for criticisms of the hierarchical classifications.
Because test items must be aligned with performance objectives, you’ll need to include higher-order thinking skills from the start. And yes, these may be better measured through open ended questions, essays and discussions. But if you find yourself needing to use multiple choice tests, you can make the best of this situation with these three approaches.
1. Real-world Scenarios
One of the most effective ways to assess higher-order thinking is to use scenario-based questions, particularly ones that simulate real work experiences. The sample question pairs below demonstrate how to transform a question from the simple recall to a higher-order thinking skill using the scenarios approach. Note: Only the question stems are shown. Assume there are multiple choice options beneath each question.
Example 1:
- Before: What symbol does a formula always start with in Excel?
- After: To total the first ten data cells in column B, which one of the following formulas should you use?
Example 2:
- Before: What is the first concern of an emergency worker?
- After: An emergency technician (ET) arrived at the scene of an accident. Three people were injured. One person had scrapes and bruises, a second was moaning in pain and a third was quickly losing blood from a skull injury. Why did the ET triage the skull injury as the highest priority?
Example 3:
- Before: When writing a CSS style sheet, what property is used to position an element to the left or right?
- After: How will the text and photo be aligned using the code below?
CSS: .photo {float: right;} HTML: My Cat <img src=”cat.jpg” alt=”photo of stuff” class=”photo” />
2. Analysis of Visuals
You can also assess critical thinking skills by asking learners to analyze or interpret information from visuals, which are provided as part of the question stem or the answer choices. In many cases, visuals such as job aids, diagrams and graphs simulate workplace tasks. See a few examples below. Note: Assume there are multiple choice options beneath each question and that visuals are displayed.
Example 1:
- Before: What is/are the function(s) of a computer’s projecting mechanism? (Select all that apply.)
- After: Using the repair flowchart shown here, what should you check if the monitor stops working?
Example 2:
- Before: Choose the best definition of rate card.
- After: Using the rate card, what is the best time for a customer in the United States to make an international call to Brazil?
Example 3:
- Before: Which country will have the largest population in the year 2030?
- After: What does this graph predict about world population in the year 2030?
Example 4:
- Before: Select the best definition of active listening.
- After: Which video best demonstrates active listening during a call with an irate customer?
3. The Answer Plus The Reason Why
A third approach to measuring critical or creative thinking is to ask learners to synthesize what they’ve learned into an explanation. Although these test items are more difficult to write, they can be quite satisfactory for assessments. The possible responses include the answer and then a variety of reasons that support the answer. Of course, only one reason is logical and correct, based on the knowledge and skills being assessed.
Example 1:
- Before: What are three signs of edema?
- After: A patient entered the hospital with edema of both lower extremities. What action should the nurse take and why?
Example 2:
- Before: Select the most effective tone for writing technical documentation.
- After: Select the paragraph and reasoning that best demonstrate how to start a technical document.
Example 3:
- Before: Name three qualities of effective entrepreneurs.
- After: Review the financials of these three startup companies. Which one would you invest in and why?
Also see 10 Rules for Writing Multiple Choice Questions for ways to make your tests more accurate and error-free. Plus see Patti Shank’s book: Write Better Multiple-Choice Questions to Assess Learning for more great tips.
Hi Connie,
Thank you so much for your article. I’m currently a MSc student doing developmental and educational psychology research and have been suggested a dissertation topic on visual learning strategies and higher-order thinking. The scope of study was narrowed to using the argumentation vie-diagram (AVD) (a diagram which is similar to mind-map but focuses on helping one to creative critical solutions for both arguments and counterargument as well as to create compromises/solutions to solve a case scenario/problem based question/statement etc. by integrating the ideas from both sides) as the learning strategy and the HOT skill would be creative thinking — the highest level of HOT skills according to the Bloom’s Taxonomy. My supervisor suggested me to use MCQs to test creative thinking/creative solutions using MCQs as the test format with the use of AVD as the learning strategy. However, since the nature of MCQ consists of close-ended questions so it provides little to no space for creativity. I’ve been struggling to find related rating scale and MCQs for assessing creativity based on this matter. May I ask if you have any ideas in creating MCQs to suit this case scenario?
Glad it was helpful, Carlton.
Greate article Connie. I’m not always a fan of quizzes etc as the way they are written often means they are just testing short term memory. However, multiple-choice questions written in the way you have described can be brilliant for testing so much more. Thanks for sharing.
Hi Amy,
I do have three examples there, such as “A patient entered the hospital with edema of both lower extremities. What action should the nurse take and why?”
Maybe you’d like to see an answer choice? Even though I’m not a medical person, I can imagine an answer like this:
a. The nurse calls a resident heart specialist to examine the patient because she wants to rule out congestive heart failure, which can be an underlying condition of edema.
Is that what you were looking for?
Can you give an example of an “answer plus the reason why” question?
Hi Dick,
I think your advice is sound as long as we avoid the appearance of “tricking” the student and that the test questions are tied to the original learning objectives. Thanks for your thoughtful comment.
Connie
The questions should be written very generally (“Which alternative(s) is/are correct regarding [subject]-“) Then all the logic should be in the alternatives. Yes, it’s a pain then to come up with 4-5 alternatives that not all are correct, BUT all make sense to a student – IF the student hasn’t studied the subject well enough.
And of course one should always use “multiple answers” of the quiz making tool has this implemented in a smart way.
Writing a good multiple choice question is a difficult task starting at grade K and continuing all the way through adult learning. Just for fun, I goggled “How to cheat at Multiple Choice Questions” and found 3,590,000 hits. The top ones included: “when in doubt choose c” and “all of the above” is usually the correct answer.
I found your post very helpful when thinking about writing multiple choice questions. The scenario option is perfect for the instructional designer. It seems as if it will almost write itself.
I took a stab at a before and after.
Before: What is the best app to use to find the definition of a word?
a) dictionary.com
b) webster word of the day
c) color dictionary
After: Your students are reading “The Whirligig” by Paul Fleischman and come across the word “inadvertently”. What is the best way for them to find the definition on their tablet?
a) look at dictionary.com
b) open the app “Webster word of the day”
c) use the “color dictionary” app
I believe that as an Instructional Designer assessment will be the most difficult part of our job. It will be easy to design a course and sit back and marvel at our work. The real “test” will be in the “tests” and the more authentic we can design them the better we will be.