You can’t make it through a year of designing eLearning without having to write test or assessment questions. If you don’t have formal training in this area you may not be familiar with test validity. Or perhaps you slept through that Educational Statistics class. (Personally, I found my Statistics textbook to be an excellent sleeping pill.)
It’s A Measuring Instrument
When you develop a test, you’re supposed to be measuring the knowledge and skills that the learner constructed, retained and can apply to new situations. However, the assessment may fail to result in a meaningful measurement for a few reasons.
- First, your test can’t represent all the content in the course.
- Second, perhaps you tested some of the least significant content.
- And third, the most important aspect of workplace learning is usually the application of knowledge in the real world. Testing knowledge alone in a vacuum has its limitations. Listen to How to Plan, Design and Write Tests or download the transcript.
Test Validity
In an effort to make your tests better measures of meaningful learning, consider the two criteria of effective tests: validity and reliability. This article explains test validity. Next, I recommend reading Are your online tests reliable? Test validity answers the question:
“Are you measuring what you think you’re measuring?”
In the world of workplace learning, focus on whether your test best represents the performance that needs improvement. Because you want the test to accurately assess knowledge and skills, the test questions must correspond to the most important performance objectives.
How To Weight Learning Objectives/Actions
One way to achieve greater validity is to weight the objectives. This helps ensure you are testing the most important content. Here is a reasonably quick way to go about it.
- Review the course performance goal so you can keep it in mind.
- Create a matrix in order to determine the value of each learning objective or action.
- Score each objective in terms of its importance to the learner’s job, its difficulty and the frequency in which it occurs in the course.
- Then total the scores.
- Objectives with a low score are low priority and will not be tested; objectives with the highest scores will be tested. (Note: if the training has not yet been developed, consider eliminating low scoring items from the course altogether.) See the matrix below for an example of learning objectives for a course on Executive Communication Skills.
Example Of Weighting Objectives
| PERFORMANCE OBJECTIVE/ACTION | IMPORTANCE TO JOB |
TASK DIFFICULTY | FREQUENCY | TOTAL |
|---|---|---|---|---|
| Use effective techniques to overcome barriers that prevent accurate communication between supervisors and frontline workers. | 5 | 4 | 4 | 13 |
| Determine the factors that keep executives isolated from staff. | 3 | 3 | 1 | 7 |
| Apply safety practices to the tasks that your warehouse employees perform. | 5 | 4 | 2 | 11 |
Other Ways To Improve Test Validity
- Keep the test items aligned with the high priority learning objectives.
- Develop test items that measure the application of knowledge, as in scenario-based items. See Writing Multiple Choice Test Questions for Higher Order Thinking.
- Have test items reviewed by an expert who understands the skills that the workplace requires.
- Ask a few target members of the audience to read through the test questions to ensure they are clearly worded.
- Allow a test to evolve by monitoring learner performance. Revise items that cause difficulty or revise that aspect of the course.
Resources:
- Are Your Online Tests Reliable?
- Writing Better Tests for Job Training
- Learn more about making tests valid in Wikipedia.
Leave a Reply