Academics and educational developers who like to develop structured approaches to evaluation will find the evaluation cookbook very useful. In addition, webpages derived from The Evaluation of Learning Technologies Project do give a reasoned description of many of the issues and problems involved in formal evaluation. The work has evolved into comprehensive toolkits that attempt to determine the 'learning and teaching media' most applicable to particular circumstances and evaluation techniques that can be used: Media Advisor and Evaluation of Learning Toolkits .

Evaluation is not a simple concept and it is unlikely that any 'toolkit' will point to a definative methodology. Practice and theory diverge considerably in these areas and sometimes the results of even 'staightforward' interventions are masked by a range of problems. These often include:lack of institutional or staff committment to 'experimenting' with new learning and teaching processes; inadeqate technical support or infrastructure; learning outcomes or resources out of step with staff and student skills; lack of staff time to plan for changes and to evaluate their impact; inadequate embedding of the learning resource into the learning programme. Faced with this array of problems, sometimes very simple evaluation methodologies are the best; they may not provide the most comprehensive answers but at least they are achievable. Here are some pointers to help to design evaluation processes.

Evaluation and design Do these learning resources help students to achieve their learning outcomes? Are they better than other approaches or can they yeild additional learning outcomes not accessible from traditional resources? Are the additional workloads, technical skills-bases and computer resources 'cost effective'? Is the innovation sustainable and transferrable? Are suitable videos available or do they need to be commissioned? What re-engineering might be necessary to make streaming viable? How can the vide be linked to student learning and tutor teaching? How will staff and students access the streaming video? What IT skills are needed by staff and students?

Potential evaluation tools for formative and summative evaluation User-tracking (eg by web-logs); observation; student and peer interviews; focus group; peer and expert review; reflective journals; confidence logs; attitude surveys; outcome-relevant assessment; questionnaires for students, tutors and related departmental staff. Much of the data that is collected is qualitative and controlled experiments are rare. Data analysis is often less of a problem than is the reasonable interpretation of results. An important point to make is that it is rare for any learning and teaching process to be extensively evaluated and particularly rare for traditional teaching and learning to undergo the checks that are mandatory for those that use learning technologies.Dissemmination of evaluation results is even more tricky. Often the 'best' interventions are reported in peer reviewed journals. Generally these are the ones with comprehensive evaluation that demonstrated to critcal peers that the intervention worked. The changes that didn't work, or those with partial ( but the best available in the circumstances) evaluative coverage are rarely reported. They should be as they are important contributions to this 'field of public enquiry'. On-line case studies have become an important alternative to peer-reviewed journal articles in this respect. Lifesign will endeavour to publish all relevent information on the successes and failures with streaming video in the life sciences.

Some Lifesign evaluation tools

Formative evaluation:In an ideal world evaluators would have access to real students while they were using learning resources. Often this is not possible. Some Lifesign video applications will make the most of server records of access to streaming URLs and some reflective logs kept by developers, tutors and support staff.

Summative evaluation:In an ideal world evaluators would have access to outcome-relevant assessment to judge the success of specific learning resources in supporting specific learning outcomes. Often this is not possible. Some Lifesign video applications will make the most of questionnaires that attempt to gauge the perceptions of staff and students. Some typical questions are; Qs for the tutor; Why did you feel the need to change your teaching? Why did you chose video as the format for the teaching/learning resource used? Before you introduced the change, what were your concerns about its use? Having now taught with the new resource, what are your major concerns about its use? Do you think that the change that you have introduced will have an effect beyond the 'unit' itself? If so what? Within your existing teaching environment, how sustainable do you think the changes that you have introduced are? Qs for the student about the particular use of video ; These questions assume that video was used in some way to support particular and specified learning outcomes. How EASILY do you think you LEARNED from the particular teaching method used. How WELL do you think you LEARNED from the particular teaching method used. Could ANYTHING have been done to make the learning process more easy? If so what? What do you think were the BENEFITS of the method you used? What do you think were the DISADVANTAGES of the method you used?