• Home
  • Free Resources
    • Storyboard Depot
    • Instructional Design Programs
    • Best Stock Photo Sites
    • PowerPoint Grids
    • eLearning Examples
    • Glossary of Online Learning Terms
    • Graphic Editing Tools
    • Icon Collections
  • My Books
  • About
  • Book Store
  • ON-DEMAND COURSES

The eLearning Coach

For designing effective learning experiences

  • Podcasts
  • LXD
  • eLearning Design
    • Audience
    • Instructional Design
    • Learning Objectives
    • Storyboarding
    • Tests
    • User Interface
  • eLearning 2.0
  • Cognition
  • Multimedia
    • Audio
    • Graphics
    • Video
  • Reviews
    • Book Reviews
    • Software Reviews
  • At Work
You are here: Home / eLearning Design / Are Your Online Tests Reliable?

Are Your Online Tests Reliable?

How to Improve Online Assessment

by Connie Malamed

Tweet27
Pin7
Share381
415 Shares

Are your online tests reliable?

After going through all the effort of developing an online test, you want it to be an accurate measure. That’s why it’s so important to plan for online test reliability.

In Are Your Online Tests Valid?, we examined test validity or how you can be sure a test measures what it claims to measure. Test validity is required before reliability can be considered in any meaningful way. You may want to read the previous article first.

In this article, we’ll look at test reliability. A test with a high degree of reliability will be a more accurate measure of the learner’s knowledge and skills than one with low reliability. If you have trouble keeping all of these terms straight, think of it this way: reliability = consistency.

Test Reliability Is Consistency

Test reliability is an attempt to reduce the random errors that occur in all tests to a minimum. The way to reduce random errors is to make a test consistent. A test that is reliable or consistent has few variations within itself and produces similar results over time. This is often compared to a scale. If you weigh yourself every day and your weight is reasonably consistent, you consider the scale reliable. If the scale displays wildly different weights from day to day (even during the holidays), you would not consider it a reliable measure.

Test reliability answers the question:

TO WHAT DEGREE IS A TEST CONSISTENT IN WHAT IT MEASURES?

What Makes A Test Consistent?

A test that is reliable will have a degree of consistency evidenced by these characteristics:

  • The test items seem similar or highly related. The test comes together as one whole.
  • There are no great leaps in difficulty, wording and tone. It might seem like one person wrote the entire test.
  • If the test were administered to similar groups, you would see similarities in the scores across the groups.
  • The test is long enough to assess the learner’s knowledge. Very short tests are more affected by the “luck factor.”

How To Improve Online Test Reliability

  • Ensure that the test measures related content. Avoid creating one test for several different courses.
  • Ensure that testing conditions are similar for each learner. For example, if your testing software displays well in a particular browser, then make using the best browser a requirement.
  • Add more questions to the test. A longer test is going to be more reliable.
  • Word test questions very clearly so that no other interpretations are possible.
  • Write test instructions so that they are easily understood.
  • Make sure the answer choices are clearly different from each other and that distractors (wrong answers) are 100% wrong.
  • Create test items of similar difficulty, when possible.
  • Test members of the same audience group twice, ideally a month apart. If the distribution of scores are similar, the test is likely to be reliable. If the scores are very different, improve the questions that had a discrepancy. Take into account that scores on the second test may be a a bit higher. (Because of deadlines and budgets, administering two tests is probably unrealistic. Still, we can dream, can’t we?)

Relationship Of Reliability To Validity

A reliable test is not necessarily a valid test. A test can be internally consistent (reliable) but not be an accurate measure of what you claim to be measuring (validity).

RESOURCES:

  • Are Your Online Tests Valid?
  • How to Plan, Design and Write Tests
  • Improving Test Quality

Get the latest articles, resources and freebies once a month plus my free eBook, Writing for Instructional Design.

Comments

  1. Connie Malamed says

    November 10, 2009 at 10:23 pm

    I think you are ultimately right, Ken. And it is a positive sign that people are continuing to question the idea of testing altogether. This article shows ways to minimize the errors and to be as consistent as possible. You can never fully achieve 100% reliability.

    Also, I think in academia, the stakes are often higher. In the workplace, there are many reasons for testing and organizations will continue to ask eLearning designers to develop tests. So from a practical perspective, improving the validity and reliability of the tests is the best we can do for now. Thanks for your opinion!

  2. Ken Allan says

    November 10, 2009 at 10:14 pm

    Kia ora e Connie!

    To assume that any test can be consistent and/or reliable is to be too presumptive about human nature.

    The results of a test are really what’s being referred to in this post, for the whole reason for creating a test is to gather results from those participants who sit it.

    There is a fallacy in believing that any test, however well designed, can possibly be truly consistent. Even the best tests vary in their ‘consistency’ because the way people interpret tests will (also) vary considerably. This variation is not necessarily directly connected with the actual knowledge or skill abilities of participants.

    For this reason, academic assessment methods, in countries throughout the world, are forever being reviewed. It is for this reason also that honours degree candidates, having sat their final examinations, may still be required to undergo an oral examination. Even the consistency of the results of these tests can be put in dispute simply because of the test environment itself, and how it is viewed and accepted by the participants.

    Many studies have been done, especially on survey questionnaires and the like, which indicate clearly that interpretation of even an apparently unequivocal question can vary considerably, and among participants of similar background, ability and intelligence.

    My own personal feeling is that to be confident about the consistency of any test is folly. Such an assumption will ultimately lead to inconsistent gathered information, as has been found with qualifying examination results collected and analysed throughout the world.

    Catchya later
    from Middle-earth

Trackbacks

  1. Evaluate 1 – Summative Assessments | Virtual CLASSROOM Reality says:
    September 29, 2017 at 8:22 am

    […] The infographic on the right gives some important information for tests reliability according  Malamed. […]

  2. Development of Classroom Assessment; Matching Type | rachelbesangre says:
    September 28, 2015 at 11:14 am

    […] B should be plausible answers to the premises in Column A. Otherwise, the test loses some of its reliability because some answers will be […]

  3. Getting Testy Part 3 — ID Musings says:
    December 9, 2012 at 7:36 pm

    […]  https://theelearningcoach.com/elearning_design/are-your-online-tests-reliable/ […]

  4. gen2oh.net » The Development Phase: ADDIE Serves Up Humble Pie says:
    September 24, 2012 at 11:30 pm

    […] If you haven’t, it’s gonna get bumpy. The materials won’t match your objectives. The questions you ask of learners won’t actually assess their mastery of the content. The materials will be patchy, or confusing, or worse – […]

  5. Reading 11/16/2009 « Hueihsien's Blog says:
    November 16, 2009 at 11:42 am

    […] Are Your Online Tests Reliable? […]

  6. Are Your Online Tests Reliable? : The eLearning Coach | OnLearn says:
    November 10, 2009 at 12:34 pm

    […] original post here:  Are Your Online Tests Reliable? : The eLearning Coach Comments [0]Digg […]

  7. uberVU - social comments says:
    November 10, 2009 at 9:18 am

    Social comments and analytics for this post…

    This post was mentioned on Twitter by elearningPosts: Are Your Online Tests Reliable? http://bit.ly/1qFeAj…

  8. Tweets that mention Are Your Online Tests Reliable? : The eLearning Coach -- Topsy.com says:
    November 10, 2009 at 6:45 am

    […] This post was mentioned on Twitter by David Hopkins, eLearning Learning. eLearning Learning said: Are Your Online Tests Reliable? http://bit.ly/1qFeAj […]

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Writing for Instructional Design

Subscribe to my newsletter AND get my eBook, Writing for Instructional Design.

Get the latest articles, resources and freebies once a month plus my free eBook, Writing for Instructional Design.

The Learning Guild

CSUN Instructional Design Program

CSUN Instructional Design Program

Fullerton Instructional Design Program

Fullerton Instructional Design Program

Learn ID at University of San Diego

u of san diego id program

ANSWERS TO QUESTIONS ABOUT A CAREER IN ID

answers to 5 key questions about a career in instructional design

NOW AVAILABLE AS AN EBOOK

You can now buy the email course as a PDF to get all the lessons at once. Price: $15.00 US.

eBook Cover
Click the image to buy the book.

SITE MENU

Home
Resources
My Books
About
Book Store
Membership Community

TOPIC MENU

Podcasts
LXD
eLearning Design
eLearning 2.0
Cognition
Multimedia (Graphics & Audio)
Reviews
At Work

RESOURCES

Storyboard Depot
Instructional Design Programs
Best Stock Images
PowerPoint Grids
eLearning Examples
Glossary of Online Learning
Graphic Editing Tools
Icon Collections

© Copyright 2021 Connie Malamed · All Rights Reserved