The OLDS MOOC (Open Learning Design Studio MOOC) evaluation report by Simon Cross has just been published and I thought it was worth a quick look.  OLDS MOOC, based at The Open University, was a learning design MOOC that ran between January and March 2013. The course was structured to reflect a proposed process for design, and combined a number of design thinking methodologies such as the Ideo Design Thinking for Educators Toolkitinquiry based learning and educational design research.

As you’d expect, the report is thorough and systematic.  The data for the evaluation was a combination of more formal surveys as well as capturing information from the spaces being used for the course.  (Interestingly, some of the criteria for the evaluation were provided by those taking a pre- course survey.)  Here are some of the main points that I think come out of the report:

  • It’s hard to evaluate the experiences of those who mainly lurk or just passively learn from the materials, even though they seem to be a majority
  • Asking MOOC learners to use technologies with which they may not be familiar (in this case, Cloudworks) can present an additional challenge; this has implications for the use of open tools or technologies with which users may be less familiar
  • Only 50% of those who completed the post-course survey felt that their original learning goals were met; some participants seemed to have struggled with scheduling
  • Using traditional evaluation criteria, 63% said they were satisfied with the quality of the course and 72% were satisfied with the quality of facilitation
  • Between 1/3 and 1/2 of participants applied for badges that were awarded earlier in the course but the more advanced badges based in peer-review activity fared less well

In one respect, the main conclusion of the report is that there remain questions about how best to evaluate MOOC learning and in particular (i) how to form a balanced view of highly diverse experiences; and (ii) how to better understand and meet learner expectations and goals within MOOC.  Maybe we can summarise this with the view that there is a need for  MOOE (Massively Open Online Evaluation)…

You can download the report from

Another approach is found in Gilly Salmon’s recent suggestion that we should evaluate MOOCs “by their capacity and capability to create positive and successful change for universities of the future” where MOOCs are the ‘tugboats’ that can change the direction of the slow-moving university ‘supertankers’.  I’m not sure this is quite right, though: if the only measure is disruption then this can be achieved in a number of ways (not all of which are desirable).  Our own hypotheses tend to be framed in terms of the complementarity of open and traditional education models though of course the question remains open.  But perhaps the crises affecting education are such that the idea of disruption becomes appealing for its own sake.  Here’s what she has to say:

What might the qualities of this disruption be? Well, for me the constructive ones are pointing to the fuller development and deployment of open education resources, of the appreciation of the potential and reach for huge scale learning, addressing and solving challenges of very large numbers of participants, global reach, accessibility and participation and the enormous advantages of flexible, entirely digital learning provision. There’s also something about learners and participants determining their own choices and pathways, and following their own rather than the providers’ motivations, outcomes and determinants. But business cases and credentialing… well they are still in the murk, with only the odd lighthouse shining. Probably not clearly enough to turn the supertankers yet. The serious negative disruptor for me is the strange and mysterious view of the pedagogy of the many MOOCs based on face-to-face knowledge transmission. As others have pointed out, it’s not as if we don’t know some of the key qualities of learning online… truly we do.

My first thought in response was that MOOC designers are generally aware of good principles of design in online learning although perhaps I am being rather optimistic!  It seems to me that there is nonetheless a tension between the idea of the autonomous learner (described more fully in my paper with Marcus Deimann) and the case for credentialled learning which MOOC providers can monetize.  It’s not clear to me where the real potential of MOOCs lie here.  Perhaps it is this kind of confusion that makes MOOC so hard to evaluate.