The eLearning Coach

10 Rules For Writing Multiple Choice Questions

Connie Malamed — Fri, 19 Mar 2010 02:05:21 +0000

This is a back-to-basics article about the undervalued and little-discussed multiple choice question. It’s not as exciting as discussing 3D virtual learning environments, but it might be just as important. If you need to use tests, then you want to reduce the errors that occur from poorly written items.

The rules covered here make tests more accurate, so the questions are interpreted as intended and the answer options are clear and without hints. Just in case you’re not familiar with multiple choice terminology, it’s explained in the visual below.

Here are the ten rules. If you have any others, please add them through the Comments form below.

Rule #1: Test knowledge comprehension, not just recall

Multiple choice questions are criticized for testing the superficial recall of knowledge. You can go beyond this by asking learners to interpret facts, evaluate situations, explain cause and effect, make inferences, and predict results.

Rule #2: Use simple sentence structure and precise wording

Write test questions in a simple structure that is easy to understand. And try to be as accurate as possible in your word choices. Words can have many meanings depending on colloquial usage and context.

Rule #3: Place most of the words in the question stem

If you’re using a question stem, rather than an entire question, ensure that most of the words are in the stem. This way, the answer options can be short, making them less confusing and more legible.

Rule #4: Make all distractors plausible

All of the wrong answer choices should be completely reasonable. This can be very hard to accomplish, but avoid throwing in those give-away distractors as it detracts from the test’s validity. If you’re really stuck, get help from your friendly SME. (BTW, this word can also be spelled as “distracter.”)

Rule #5: Keep all answer choices the same length

This can be difficult to achieve, but expert test-takers can use answer length as a hint to the correct answer. Often the longest answer is the correct one. When I can’t get all four answers to the same length, I use two short and two long.

Rule #6: Avoid double negatives

No big news here, right? Don’t use combinations of these words in the same question: not, no, nor, the -un prefix, etc. For example, this type of question could confuse test-takers: ‘Which of the following comments would NOT be unwelcome in a work situation?’ Flip it around and write it in the positive form: ‘Which of the following comments are acceptable in a work situation?’

Rule #7: Mix up the order of the correct answers

Make sure that most of your correct answers aren’t in the “b” and “c” positions, which can often happen. Keep correct answers in random positions and don’t let them fall into a pattern that can be detected. When your test is written, go through and reorder where the correct answers are placed, if necessary.

Rule #8: Keep the number of options consistent

Did you ever have to convince a SME that he or she can’t have answer choices that go to ‘h’ in one question and ‘c’ in the next? It’s something of a user interface issue. Making the number of options consistent from question to question helps learners know what to expect. Research doesn’t seem to agree on whether 3 or 4 or 5 options is best. Personally, I like to use 4 options. It feels fair.

Rule #9: Avoid tricking test-takers

As faulty as they are, tests exist to measure knowledge. Never use questions or answer options that could trick a learner. If a question or its options can be interpreted in two ways or if the difference between options is too subtle, then find a way to rewrite it.

Rule #10: Use ‘All of the Above’ and ‘None of the Above’ with caution

I hate this rule because when you run out of distractors, All of the Above and None of the Above can come in handy. But they may not promote good instruction. Here’s why. All of the Above can be an obvious give-away answer when it’s not used consistently. Also, the All of the Above option can encourage guessing if the learner thinks one or two answers are correct. In addition, the downside to None of the Above is that you can’t tell if the learner really knew the correct answer.

Do you have any more rules to add to this list? Please use the Comments form.
Share this article with a friend using the social tools below.

Post from: The eLearning Coach

10 Rules For Writing Multiple Choice Questions

Learning From Visual Data

Connie Malamed — Tue, 16 Mar 2010 11:05:19 +0000

In the not so distant past, the visualization of data was reserved for the scientific community. It helped researchers grasp the structure and relationships in large sets of data. But the explosion of complex information in most knowledge domains has changed all that.

It seems that now, the appreciation for visual thinking is growing and you can find many data visualizations in the popular media. This makes me wonder if we’re doing as much as we can to visualize information when we design learning experiences.

Visualizing information makes it tangible so that people are more likely to comprehend it. If you’re looking for new ways to visually explain complex and abstract concepts, I’ve found that it helps to play around with data visualizations. Here are four compelling examples where you might find inspiration. The last three are highly interactive.

Explaining 1 Trillion Dollars

Although it might be difficult to understand 1 trillion dollars (an approximation of the US deficit), the BBC found a simple way to explain the number by starting with a door mat as one unit. This type of analogy could be transferred to the explanation of other complex numerical concepts in online learning. To see the animations, click on the But how big is that? button and then on each number category. Wait a minute for the whole thing to load. See 1 Trillion Explained.

How Different Groups Spend Their Day

The New York Times excels in interactive data visualizations. They have the staff and the vision to create interactive graphics that help you learn and draw your own conclusions. This particular example reveals a lot about the culture and society of the US. In terms of online learning, you get a lot here just by playing around. That alone is a powerful design lesson for all of us. See How Different Groups Spend Their Day.

Causalties of War

In this data set of war casualties from 2004 to 2009, users can drag a slider to see the information change by year (when you select the Since Troop Buildup Began link). Or you can click on individual links to show data related to specific events. Although programming large data sets might not be within our reach or domain, this is a good example of learning through interactivity. See Casualties of War.

Cost of Getting Sick

In this visualization, you can play with the the associated health care costs of various chronic conditions by selecting each colored slice of the chart and dragging the slider to a specific age. Of particular interest is the way the chart itself changes size depending on the costs associated with age. It’s the different combinations of data that bring about new ways of seeing things. In online learning, we need to visualize information to help people see things in new ways. See The Cost of Getting Sick.

Related Article:
Book Review: The Back of the Napkin (about visual thinking)

Do you get ideas from these examples? Can we use something similar (but less complex) in online learning programs? Let’s hear what you think.

Post from: The eLearning Coach

Learning From Visual Data

Lessons From The Guggenheim, Spain

Connie Malamed — Mon, 15 Mar 2010 15:04:38 +0000

Recently I was traveling to give a presentation and had the chance to visit the Guggenheim Museum in Bilbao, Spain. The overwhelming characteristic of the museum is its spectacular architecture—playful, innovative and designed to look random. It’s a structure that seems to dance rather than stand still.

Like many visitors, I paid a few Euros to rent a portable audio player for the self-guided tour to learn about the museum’s architecture and its architect, Frank Gehry. This experience of simultaneously touring and listening at the Guggenheim was one of the most compelling learning experiences I’ve had in a long time.

It was an ideal model for integrating a structured yet informal learning paradigm. The audio guide was structured in the sense that the content was organized into segments that created a meaningful presentation. It was informal in that it was a voluntary endeavor with no prerequisites or requirements. Users could take or leave what they wanted.

The experience was powerful and I wondered what I could transfer to the world of online learning. I don’t have all the answers, but here’s what I’ve figured out so far.

It engaged all of my senses. While touring this architectural wonder, it felt as though my brain couldn’t process all the sensory input I was perceiving. This state of heightened arousal is highly motivating and is one reason people seek new experiences. Possibly the only way to transfer this type of potent sensory experience online is through a virtual environment.

It heightened my awareness of physical space. Wandering through the Guggenheim’s atrium, you can see through the large glass windows how the architect tied together the interior and exterior space. It increased my understanding of how the physical environment affects our consciousness. Again, a 3D immersive environment would be ideal for this type of learning. See Your Brain On 3D Learning for more on this topic.

It gave me the controls to stop, start and repeat the flow of information. You may take it for granted that asynchronous online learning allows users to move at their own pace. We shouldn’t underestimate the importance of allowing learners to stop if they want to ponder, to start when they feel ready, and to repeat if they need to hear or see something more than once to comprehend it. This type of control often simulates how we learn naturally.

The content was well organized. The meaningful organization of information is what transforms facts and data into usable content. We can achieve this in all types of online learning with good principles of instructional design. Check out Content Organization Cheat Sheet for suggestions.

The audio content was rich. The guided audio tour integrated varied content and sound bytes. It included the architect’s dreams and plans for the structure, characteristics of the building materials, interesting anecdotes about the architect’s creative approach, quotes from others, and comments from an interview with the architect himself. The typical online learning program uses straight narration. Perhaps we can enhance the richness of the audio channel by including sound effects, interviews, music and different perspectives.

The experience was immediate. It’s hard to replicate the immediacy of walking through a museum that feels like a living sculpture but it’s something to strive for in our online learning. Can we create experiences that have so much impact, the learner is compelled to pay attention?

It allowed for discussion as well as solitary thought. As I wandered the building with my touring partner, we would pause the audio to discuss the points that seemed particularly fascinating. Then we would continue on, wandering around a room on our own paths.

We now know that discussion and collaboration contribute a great deal to the learning process. Synchronous learning events and social media technologies allow us to integrate this dimension of learning into the programs we design.

We have a great opportunity to learn about design from the experiences that surround us, so that we can make online learning as natural as possible.

Post from: The eLearning Coach

Lessons From The Guggenheim, Spain