The Evaluation of e-learning project undertook a collaborative review of literature on the subject. Over one hundred reviews have been added to the project database.

This is what we found out.

Categorising the literature

In general there is an overwhelming preponderance on the evaluation of technology, usually through ethnographic research methods (questionnaires to participants).

However, it is possible to identify identified ten main categories of literature.

Case studies of specific e- training programmes

For the most part these are descriptive rather than analytic or predictive, predominantly American, mainly located in a Higher Education rather than vocational training environment and focussed on the ‘virtual classroom’ model. They also tend to be restricted to particular subject areas, in particular I.T, languages and engineering disciplines. (This is not necessarily to say that e-learning is restricted to these areas, rather that they are over-represented in evaluation reports.) 
 

Comparisons with traditional learning

There are some (but surprisingly few) systematic studies that compare e-learning effectiveness with traditional learning and which are empirically robust. Those that exist are mainly small-scale studies, often using a matched pairs design and are frequently of very specific instances of e-learning in which the e-learning methodologies are idiosyncratic and the conclusions cannot be generalised.
Tools and instruments for evaluation of e-learning
There is an abundance of literature detailing tools for the evaluation of e-learning. However, these are mainly divided into two types. Firstly there are many on-line data gathering instruments for assessing, typically, the user interface characteristics of software (e.g. student perception questionnaires) or secondly, there are devices to record and analyse usage by duration and frequency of log-in, pages accessed, user profile etc. Many of these are sophisticated in their design and ingenuity but lack guidance on interpretation and analysis.

Return on Investment (ROI) reports

There are surprisingly few ROI reports, considering the huge investments into e-learning at all levels. The majority of those that exist draw mainly from industry based examples and are written from an HRD perspective. The conclusion is inevitably that the investment was cost-effective and represented value-for-money but often the savings are defined in efficiency rather than effectiveness with no long-term impact analysis that takes account of unintended outcomes and consequences. It is also difficult to compare figures across reports because the distinctions between net and gross costs, capital and revenue costs, displacement of existing funds, costs over time etc. are often blurred or missing. Many ROI type evaluation reports appear to be justifying investment rather than evaluating it and more geared to an audience of shareholders rather than researchers.

Benchmarking models

There have been several attempts to generate sets of criteria for quality assuring e-learning. However, these tend to be skewed towards proposing quality standards for e-learning systems and software which often disregard key variables in the wider learning environment or are based on criteria associated with evaluating traditional learning processes (and which disregard the technology) or criteria associated with measuring learner achievement through traditional pedagogies. An additional problem is that the designers of these benchmarking systems are often locked in to a particular model of e-learning which limits their transferability.

Product evaluation

By far the greatest number of `hits’ on evaluation of e-learning are reports describing (and extolling the virtues of) particular education software. The vast majority of these reports are commissioned or published by the software developers. This is not to question the usefulness of these reports or necessarily to doubt their validity but evaluation of `de-contextualised’ software is not an acceptable substitute for the rigorous evaluation of e-learning systems.

Performance evaluation

Scrivens (2000) in the USA, uses the term `performance evaluation’ for what would, in European terms, be called student assessment. Whilst it is true that an examination of student performance is a powerful indicator of the effectiveness of e-learning, it is by no means the only one.  Moreover, a survey of reports on performance evaluation in the context of e-learning were mainly concerned with on-line tools and instruments for examining knowledge-based learner performance and could therefore be categorised under that heading.

Handbooks for the evaluation of e-learning

There is an increasing number of handbooks for e-learning which focus primarily on evaluation. The evaluation methods and tools differ widely. What they do have in common is that they recognise the importance of evaluation and many propose that evaluation should be an integral part of any e-learning initiatives or development. In this regard, they tend toward a management model of evaluation; the primary aim of the evaluation is to provide feedback to influence e-learning implementation and future development.

Many of the handbooks appear to have been produced by education advisors and advisory services and agencies. They are frequently quite basic and lack a theoretical approach. A surprising number cite The Evaluation Cookbook and appear to have borrowed many of their ideas from this influential publication. More worrying is the technology centred approach that many of the guides espouse.

Meta-studies

There are a number of meta-studies of the evaluation of e-learning, all of them based on US literature. These are attempting to answer the question of the effectiveness of e-learning by combining or bringing together the results of a series of different studies to provide a larger sample base. Although, obviously, the methodology is open to some question, these studies are interesting and offer a new approach to the issue,

Studies on the contribution of evaluation to metadata

This cannot be called a category of literature on evaluation as such, as this literature only includes evaluation as one of the factors to be taken into consideration in developing metadata. But, it is very important for the future of e-learning evaluation and will be explored further in the notes below.

Disciplinary backgrounds

It was notable that there are considerable differences in the disciplinary background of the authors and in the journals in which they were published or conferences they were presented at. These include:

• Education and training journals and conferences
• Journals and conferences relating to the use of Information and Communication Technologies for learning
• Specialist evaluation journals and conferences
• Journals and conferences around the psychology of learning.

The disciplinary background of the researchers / authors tends to determine their approach to evaluation. For instance there is more likelihood that learning technologists focus on the development of tools for evaluation, whist those coming from evaluation studies are more likely to consider how traditional models of evaluation can be applied to e-learning. Psychologists are more likely to undertake comparative studies. What is the importance of this? It suggests that we need teams of evaluators or of researchers from different disciplines to develop cross-disciplinary approaches to the evaluation of e-learning.

Issues and gaps

First, the positive things which have emerged form the literature review.

There appears to be a growing realisation of the importance of evaluation. This seems to be linked to concerns that e-learning is not succeeding in the way that had been expected. Evaluation is needed to gain a better understanding of the problems and issues regarding e-learning.

There is also an increasing focus on evaluation methodologies and a realisation that the evaluation of e-learning is complex and requires the development of new models and approaches.

Evaluation and assessment

However there remains confusion between evaluation and assessment. To some extent this can be explained by different understandings of evaluation in different countries and by the linguistic confusions between the two processes. However, I think it goes further than this. There appears to be an over reliance on assessment or achievement as the basis for evaluation. That is not to say that learner achievement is not an evaluation factor. But the assumption that if something has not been assessed it has not been learnt is surely wrong. Furthermore, such an approach misses informal learning and learning not included in the assessments.

Pedagogic and curricular approaches

Two big gaps stand out in the literature reviewed. The first is the complete lack of any evaluation – or attempts to evaluate – pedagogic approaches of e-learning. This is in contradiction to the increasing emphasis of e-learning researchers and developers on the pedagogy of e-learning.

Evaluation approaches still tend to focus on the functionality of the learning technologies and not on the learning which is facilitated by the functionality. If pedagogy is such an important factor in e-learning, then an important role for evaluation is to assist researchers through providing an understanding of the impact of different pedagogic approaches. At the same time, in the European literature there is often an assumption that constructivism is the proven and best pedagogy for e-learning. Furthermore, it is often hard to know what the authors mean or understand by constructivism. In the way the term is being used constructivism is only a means of describing in one category all the varieties of creative, perceptive and innovative approaches to facilitating the acquisition of new knowledge.

Linked to this lack of clear focus on pedagogy is a similar gap in the evaluation of curricular approaches. Why is this important? In the technical developers world a debate has been raging for the last three years over something called learning objects. Learning objects have been seen as one of the main answers to the problem of sustainable content creation.) A learning object is any digital, reproducible and addressable resource used to perform learning activities or learning support activities, made available for others to use. The problem is that is order to develop reusability, objects need to be granular. This means that e-learning developers are promoting modularity as a curricular process, driven by technical need. Many critics are concerned that modularisation and granular learning object will challenge the coherence of learning programmes. Since this is such a key debate in fort e-learning development, surely evaluation should focus on what is happening and whether modular programmes built from learning object can work.

Perhaps the lack of such studies just reflects a time delay, before evaluation catches up with technical and e-learning development. Yet I think it reflects the need for accompanying evaluation where evaluation takes place alongside technical development and innovation. Secondly, it suggests to me that a discourse is needed between technical developers and innovators and evaluators with a better understanding or both of what the other is doing.

Programme and policy evaluation

The second gap in the literature review is programme and policy evaluation. Every country and most large institutions have active policy driven programmes to develop e-learning. However, there are few  programme level evaluation reports, and still less models or theoretical approaches for evaluating e-learning policy. This is both surprising and worrying. Without evaluation, how do we know which policy approaches are working and which are not? I remain suspicious that e-learning is merely seen by policy makers as a “good thing” and that money is being invested with little understanding of where or why.

Metadata

 As I mentioned earlier there are a growing number of references to evaluation in the technical literature around metadata. I think this is a significant development. Meta data is simply data about data. However, it is crucial in allowing computers to know what exists on other computers and providing a machine-readable description of learning resources. At first it had been assumed that creators of learning resources would provide the metadata according to some kind of agreed common schema or standard. More recently, is a realisation that in education many different people have an interest and role in providing the metadata associated with any given object – developers, teachers, trainers, curriculum developers, technical developers, librarians and archivist, students and trainees andÉevaluators. Clearly, it would be absurd to expect material developers or creators to add an objective description of the quality of their own learning materials and evaluation may play a critical role in describing quality. Technically the debate is around distributed metadata and how all the different data which becomes naturally associated with an object or learning materials in the course of their development, deployment and use, can be found and aggregated.
Once more, I think this growing debate shows a necessity for evaluators to work alongside technical developers and at a more theoretical level for a discourse around ideas.