Assessing the effectiveness of medical student teaching methods

Degnan et al. [1] studied the effects of an online learning module on medical students' drug administration skills during a simulated scenario. They conclude that ‘the additional teaching significantly improves medical students’ performance'. We have a number of concerns that lead us to believe that this conclusion may not be valid from the evidence as presented. In a climate of limited budgets for medical teaching and ever more expensive resources such as simulators, there is a great need for medical education studies of excellent quality [2]. In a study trying to demonstrate an improvement in students' performance, the most robust method would be to perform a baseline measurement, an intervention and then a repeat measurement in the same subject. Degnan et al. do not discuss their reasons for their rejection of this methodology in favour of an unpaired study design. In their discussion, the authors consider whether their statistical analysis should have used the principles of ‘intention to treat’[3]. This is a strategy for the analysis of randomised controlled trials that compares subjects in their originally assigned groups, even if they subsequently change groups or drop out. It gives a more pragmatic estimate of the realistic (rather than the potential) benefit of an intervention. Effectiveness may be overestimated if an intention to treat analysis is not done. Although most commonly applied to trials of medical interventions in patients, the principles hold true for any interventional trial. Degnan et al. discuss the effect on their analysis of following an ‘intention to attend’ principle, as four of the students recruited did not attend the scenario session. They acknowledge that this could make some of their results non-significant. We believe that a more important application of intention to treat would be ‘intention to teach’. For the analysis, students offered the extra teaching but who did not view it are grouped with those not offered it. They combine to form a large ‘no extra teaching’ group. This introduces significant bias in two ways. Firstly, the extra teaching group comprises nine individuals entirely self-selected by their decision to view the extra material: a ‘keen’ group. This nullifies the randomised allocation to groups. Secondly, the no extra teaching group contains a significant number (30%) of students who chose not to take up the offer of extra teaching: an ‘indifferent’ group. This invalidates it as a control group. One could hypothesise that ‘keen’ students will perform better than those less motivated. Indeed, the difference found between the two groups supports this conclusion as much as the authors' own. This is interesting but does not appear to be the hypothesis they set out to test, although they do not explicitly state their specific research question. We suggest that the comparison groups should have been ‘directed to online teaching module’ vs ‘not’, permitting adherence to the principle of intention to treat within the subsequent analysis. This would provide a more realistic idea of the effect magnitude for the intervention of inviting students to undertake extra learning amongst a real student population. Alternatively, the extra teaching could have been compulsory for the intervention group. Errors in drug administration are an important cause of patient harm: if it were shown that this extra learning module does usefully improve drug administration skills, it would seem inappropriate to make it optional. This brings us to the additional caveat of this study, in that less then half of the students directed to the extra teaching viewed it. It would have been valuable to study the reasons behind some students' decision not to undertake the extra teaching module.