Evaluation, outcomes and education

The elusive outcome measurement for health professional education is what is commonly referred to as Kirkpatrick level 4b (in the adaption of the original Kirkpatrick framework by Barr and colleagues): benefits to patients (or service users), such as improvements in health and well-being.1 This outcome is very difficult to evaluate at the prequalification or undergraduate level, for obvious reasons. We therefore continue to quantify the success of educational interventions in terms of what is learned compared with the defined learning outcomes. Frequently, the ‘what is learned’ is self-reported on participants’ evaluation forms, alongside satisfaction ratings and changes in confidence. The poor correlation between external performance data and self-assessment has been shown in many studies,2 although this basic level of evaluation is important for pilot studies. If participants report that they haven't learned anything or hate the innovation, there is little point in a larger scale roll out without any revision. Indeed, if educators do not react to learners’ feedback, learners will be sub-sequently demotivated to give any feedback at all. All evaluation should have a purpose if we expect learners to participate. In this issue of the journal there are examples of learner feedback beyond the post-course survey; however, interviewing participants for a more in-depth exploration of process and outcomes is time consuming. Analysis of the data is resource intensive, but if done well makes interesting reading and may help answer the important question of the realist approach: what kinds of educational interventions will tend to work, for what kinds of learners, in what kinds of contexts, to what degree and why?3 If clinical teachers want to replicate an intervention in their own environment, they need to know if it will have the same effect as reported in a paper, or what they have to do to adapt it for local circumstances. Whereas Maurice and colleagues use a simple questionnaire survey for medical students to evaluate their endoscopy simulation training,4 Nelson and colleagues evaluate their simulation intervention through a qualitative methodology. Although the interviews still focus on what the participants feel has changed, the semi-structured process allows for a deeper exploration of participants’ answers.5 Self-confidence is the outcome measure for a study on enhancing students’ telephone communication skills. The authors acknowledge the limitations of this approach, but an interesting finding is that, until this particular intervention, the students stated that they had received no training in this important area of practice.6 Readers involved with undergraduate medical education elsewhere may be stimulated by this project to consider whether they need to introduce telephone consultation skills into their own curricula. Woods and colleagues evaluated near-peer bedside teaching through feedback surveys. But another outcome beyond satisfaction was that students who had received bedside tutorials from interns in the pilot project were likely to volunteer to run the tutorials once they became interns.7 Evaluations of educational innovations were a major focus of the presentations at the 2014 Association for the Study of Medical Education (ASME) annual scientific meeting in Brighton, in July, the themes of which were ‘Education, scholarship and leadership in pursuit of excellent patient care’. Clarence Braddock, a professor of medicine at the University of California (UCLA), opened the meeting with a keynote address that emphasised the need for outcomes research in health professional education, particularly with a focus on outcomes relating to patient-centredness and shared decision-making. He drew attention to the importance of educationally sensitive patient outcomes (EPSOs). Many patient outcomes, such as reduction of blood pressure and weight loss, are affected by complex and contextual factors, including public health messages, and are therefore difficult to attribute to specific health professional education. EPSOs are pathways that do directly link patient health outcomes to educational interventions.8 For example, training in motivational interviewing and enhancing patients’ health literacy are likely to have a direct effect on patients’ well-being; however, demonstrating such effects requires large multi-institutional projects and a longitudinal approach. At The Clinical Teacher we will continue to publish smaller scale less ambitious studies, which stimulate ideas and are of more immediate practical value to clinical teachers, while acknowledging the importance of larger evaluation studies for evidence-guided education. The final plenary speaker in Brighton, Professor Madeleine Abrandt Dahlgren of Linköping University, Sweden, discussed how her evaluation methodology for the interprofessional training wards was modified in response to reviewers’ feedback, when she and her team submitted the initial results for publication. Because of criticism that the findings were not particularly new, they adopted a sociological theoretical framework for more in-depth analysis of the training in relation to professional identity formation, and how the proximity of students from different professional groups helps in the development of negotiation and teamworking skills.9 This presentation highlighted the importance of the peer-review process in feedback and for stimulating new ways of looking at evaluation. I would also like to mention briefly that the Medical Journal of Australia (the MJA) celebrates its 100th anniversary this year. To mark this, a day of presentations and discussion about medical journals (including peer review of submitted papers) was held in Sydney led by Richard Smith (ex-editor of the British Medical Journal) and Stephen Leeder (current editor of the MJA). What will be the future of journals in this electronic age of mass communication? At conferences audience participation is no longer solely confined to questions and answers from the floor: the synchronous interaction through tweets allows more people to have their say. Such a facility is also available to journal readers: slowly through letters to the editor and quickly through online discussion boards. Readers of The Clinical Teacher and Medical Education are able to participate in online conversations via the revamped blog/discussion boards at http://www.mededucconversations.com – we look forward to ‘chatting’ with you there.