title: Evaluation of educational software.
abstract: Evaluation is an important component of developing educational software. Ideally, such evaluation quantifies and qualifies the effects of a new educational intervention on the learning process and outcomes. Conducting meaningful and rigorous educational evaluation is difficult, however. Challenges include defining and measuring educational outcomes, accounting for media effects, coping with practical problems in designing studies, and asking the right research questions. Practical considerations that make the design of evaluation studies difficult include confounding, potentially small effect sizes, contamination effects, and ethics. Two distinct approaches to evaluation are objectivist and subjectivist. These two complement each other in describing the whole range of effects a new educational program can have. Objectivist demonstration studies should be preceded by measurement studies that assess the reliability and validity of the evaluation instrument(s) used. Many evaluation studies compare the performance of learners who are exposed to either the new program or a more traditional approach. However, this method is problematic because test or exam performance is often a weak indicator of competence and may fail to capture important nuances in outcomes. Subjectivist studies are more qualitative in nature and may provide insights complementary to those gained with objectivist studies. Several published examples are used in this article to illustrate different evaluation methods. Readers are encouraged to contemplate a wide range of evaluation study designs and explore increasingly complex questions when evaluating educational software.
authors.affiliation: Center for Dental Informatics, School of Dental Medicine, University of Pittsburgh, PA 15261, USA. titus@pitt.edu
authors.forename: Titus K L
authors.initials: TK
authors.lastname: Schleyer
authors.forename: Lynn A
authors.initials: LA
authors.lastname: Johnson
mesh_terms: D003198:Computer Simulation; D003194:Computer-Assisted Instruction / Q000379:methods*; D004497:Education, Dental / Q000379:methods*; D004521:Educational Measurement / Q000592:standards*; D005069:Evaluation Studies as Topic; D006801:Humans; D019982:Organizational Case Studies; D016544:Patient Simulation; D012984:Software / Q000592:standards; D012985:Software Design; D012986:Software Validation; D014481:United States; D014584:User-Computer Interface