Evaluating eLearning courses creates the feedback loop that designers need to continually improve learning solutions. How else can we know what that audience needs and whether they improve their performance? You can conduct evaluations during the design phase, after creating prototypes, when draft versions of storyboards are ready, after participants have completed a training program, and by following trends in data over time.
You can use both formal and informal methods to evaluate eLearning courses. Either way, a straightforward equation underlies the process.
COLLECT EVALUATION DATA + MAKE REVISIONS = COURSE IMPROVEMENTS
Formative Evaluations of eLearning Courses
Conduct formative evaluations of eLearning while a course is in the early stages of development, such as while prototyping or storyboarding. The goal of formative evaluation is to get data and feedback before you go into full production, so that you can revise accordingly. Revisions are easier to make in a storyboard than in finished media.
There are numerous approaches to formative evaluations of eLearning, I’ll describe one that is easy to implement here.
It’s Just You and the Learner
One-on-one evaluations are powerful ways to gain insight into an eLearning course regarding its content and usability. With this approach, you sit with the evaluator as he or she reviews the storyboards, scripts and draft graphics in either paper-based or electronic form. During this time, have the evaluator point out instances when the content is difficult to understand.
To find usability issues, have the evaluator review a prototype of the course or review the interactions that are specified in the storyboard. Find out whether the screen instructions are clear, whether it is easy to perform the next required action and whether the screen design is well-organized. The user interface should never interfere with the learning process.
Take meticulous notes from this evaluation and consider recording the feedback. When evaluations from all sample audience members are complete, collate the data and revise your work accordingly. Formative evaluation can catch many errors, awkward sentences, unclear explanations, and graphics before you even get started. For more on quality processes, see Set Yourself Up to Create Quality eLearning.
Summative Evaluations of eLearning Courses
Conduct summative evaluations of eLearning courses after learners complete a fully operational course. Well-written questions that test higher order thinking may assess how much a participant learned and remembers immediately at the end of a course. But it does not measure the impact of training on an organization, which is the ultimate goal. Therefore, it’s important to measure other metrics over time as well.
Evaluating Test Performance
During the planning stages of the course, you set goals and wrote performance objectives to meet these goals. If the course was designed systematically, then the test questions will be aligned with the priority objectives. Collecting test performance data, therefore, is an effective way to evaluate immediate understanding and retention. But in workplace learning, this is often not a sufficient evaluation.
At the lowest level, you can evaluate whether learners are acquiring the skills taught in a course by analyzing the collective test scores. If the results of the assessment are not good, look for patterns in the data. When answers to a specific question or two are consistently incorrect, either the test question itself is unclear or the instruction for that content needs improvement. Modify, adapt and revise as needed.
During the analysis and design phases of a project, identify the metrics that can measure the effectiveness of a learning solution over time. This will be part of your evaluation plan. All organizations collect metrics of some kind. Examples include:
- New annual memberships to a professional association
- Annual turnover rate at a company
- Number of customer complaints in a month
- Quarterly sales figures
- Number of graduates from low income households
Access to this type of data will allow you to measure the effectiveness of training in the long run. You can also determine whether the cost of training is worth the time, effort and funds. See Will Thalheimer’s LTEM model for a sound research-based approach.
Organizations and clients are often hesitant to spend money on evaluating eLearning courses, not realizing that this is the only way to fully close the course design and development loop. If you can find a way to integrate evaluation procedures into your training process, you will see that the benefits are huge.