title: A strategy for monitoring and evaluating massive open online courses.
abstract: We argue that the complex, innovative and adaptive nature of Massive Open Online Course (MOOC) initiatives poses particular challenges to monitoring and evaluation, in that any evaluation strategy will need to follow a systems approach. This article aims to guide organizations implementing MOOCs through a series of steps to assist them in developing a strategy to monitor, improve, and judge the merit of their initiatives. We describe how we operationalise our strategy by first defining the different layers of interacting agents in a given MOOC system. We then tailor our approach to these different layers. Specifically, a two-pronged approach was developed, where we suggest that individual projects be assessed through performance monitoring; assessment criteria for which would be defined at the outset to include coverage, participation, quality and student achievement. In contrast, the success of an overall initiative should be considered within a more adaptive, emergent evaluation inquiry framework. We present the inquiry framework we developed for MOOC initiatives, and show how this framework might be used to develop evaluation questions and an assessment methodology. We also define the more fixed indicators and measures for project performance monitoring. Our strategy is described as it was developed to inform the evaluation of a MOOC initiative at the University of Cape Town (UCT), South Africa.
authors.affiliation: Institute for Monitoring and Evaluation, Section for Organisational Psychology, The School of Management Studies, University of Cape Town, South Africa. Electronic address: s.kaschula@gmail.com.
authors.forename: S A
authors.initials: SA
authors.lastname: Chapman
authors.affiliation: Section for Organisational Psychology, The School of Management Studies, University of Cape Town, South Africa.
authors.forename: S
authors.initials: S
authors.lastname: Goodman
authors.affiliation: Centre for Innovation in Learning and Teaching, Centre for Higher Education Development, University of Cape Town, South Africa.
authors.forename: J
authors.initials: J
authors.lastname: Jawitz
authors.affiliation: Centre for Educational Technology, University of Cape Town, South Africa.
authors.forename: A
authors.initials: A
authors.lastname: Deacon
mesh_terms: D020375:Education, Distance / Q000458:organization & administration / Q000592:standards*; D004521:Educational Measurement / Q000379:methods / Q000592:standards*; D006801:Humans; D020407:Internet; D015397:Program Evaluation / Q000379:methods / Q000592:standards*; D013019:South Africa; D014495:Universities
keywords: Developmental evaluation; Evaluation; MOOC; Monitoring; Online course; Performance monitoring