I was at Ben Goldacre’s fascinating talk in the RCSI yesterday and much of what he had to say about bad drug trials resonated with me as an educator. His point that some branches of medicine were riddled with “endless pilot studies” sounded to me like it could have been said about teaching and learning research
One of his more interesting points concerned the issue of “surrogate outcomes”. An example of the might be if you were to test a new heart drug and use, say, blood pressure as a measure of the drug’s efficacy. You might find that the drug does lower blood pressure but when you dig down into the data you might find that the drug has absolutely no effect on the incidence of heart attacks. In some cases, the surrogate outcome might even be in direct opposition to the desired outcome.
It seems to me that education research has a big problem with surrogate outcomes. Whether it’s students answering questionnaires or teachers/lecturers making a judgement about student ‘engagement’, the fundamental question as to whether students have learned better or not is rarely answered.