After going through all the effort of developing an online test, you want it to be an accurate measure. That’s why it’s so important to plan for online test reliability.
In Are Your Online Tests Valid?, we examined test validity or how you can be sure a test measures what it claims to measure. Test validity is required before reliability can be considered in any meaningful way. You may want to read the previous article first.
In this article, we’ll look at test reliability. A test with a high degree of reliability will be a more accurate measure of the learner’s knowledge and skills than one with low reliability. If you have trouble keeping all of these terms straight, think of it this way: reliability = consistency.
Test Reliability Is Consistency
Test reliability is an attempt to reduce the random errors that occur in all tests to a minimum. The way to reduce random errors is to make a test consistent. A test that is reliable or consistent has few variations within itself and produces similar results over time. This is often compared to a scale. If you weigh yourself every day and your weight is reasonably consistent, you consider the scale reliable. If the scale displays wildly different weights from day to day (even during the holidays), you would not consider it a reliable measure.
Test reliability answers the question:
TO WHAT DEGREE IS A TEST CONSISTENT IN WHAT IT MEASURES?
What Makes A Test Consistent?
A test that is reliable will have a degree of consistency evidenced by these characteristics:
- The test items seem similar or highly related. The test comes together as one whole.
- There are no great leaps in difficulty, wording and tone. It might seem like one person wrote the entire test.
- If the test were administered to similar groups, you would see similarities in the scores across the groups.
- The test is long enough to assess the learner’s knowledge. Very short tests are more affected by the “luck factor.”
How To Improve Online Test Reliability
- Ensure that the test measures related content. Avoid creating one test for several different courses.
- Ensure that testing conditions are similar for each learner. For example, if your testing software displays well in a particular browser, then make using the best browser a requirement.
- Add more questions to the test. A longer test is going to be more reliable.
- Word test questions very clearly so that no other interpretations are possible.
- Write test instructions so that they are easily understood.
- Make sure the answer choices are clearly different from each other and that distractors (wrong answers) are 100% wrong.
- Create test items of similar difficulty, when possible.
- Test members of the same audience group twice, ideally a month apart. If the distribution of scores are similar, the test is likely to be reliable. If the scores are very different, improve the questions that had a discrepancy. Take into account that scores on the second test may be a a bit higher. (Because of deadlines and budgets, administering two tests is probably unrealistic. Still, we can dream, can’t we?)
Relationship Of Reliability To Validity
A reliable test is not necessarily a valid test. A test can be internally consistent (reliable) but not be an accurate measure of what you claim to be measuring (validity).
RESOURCES:
I think you are ultimately right, Ken. And it is a positive sign that people are continuing to question the idea of testing altogether. This article shows ways to minimize the errors and to be as consistent as possible. You can never fully achieve 100% reliability.
Also, I think in academia, the stakes are often higher. In the workplace, there are many reasons for testing and organizations will continue to ask eLearning designers to develop tests. So from a practical perspective, improving the validity and reliability of the tests is the best we can do for now. Thanks for your opinion!
Kia ora e Connie!
To assume that any test can be consistent and/or reliable is to be too presumptive about human nature.
The results of a test are really what’s being referred to in this post, for the whole reason for creating a test is to gather results from those participants who sit it.
There is a fallacy in believing that any test, however well designed, can possibly be truly consistent. Even the best tests vary in their ‘consistency’ because the way people interpret tests will (also) vary considerably. This variation is not necessarily directly connected with the actual knowledge or skill abilities of participants.
For this reason, academic assessment methods, in countries throughout the world, are forever being reviewed. It is for this reason also that honours degree candidates, having sat their final examinations, may still be required to undergo an oral examination. Even the consistency of the results of these tests can be put in dispute simply because of the test environment itself, and how it is viewed and accepted by the participants.
Many studies have been done, especially on survey questionnaires and the like, which indicate clearly that interpretation of even an apparently unequivocal question can vary considerably, and among participants of similar background, ability and intelligence.
My own personal feeling is that to be confident about the consistency of any test is folly. Such an assumption will ultimately lead to inconsistent gathered information, as has been found with qualifying examination results collected and analysed throughout the world.
Catchya later
from Middle-earth