In 1999 Stephen Gorard published an article in this journal in which he provided a trenchant critique of what he termed the 'politician's error' in analysing differences in educational attainment. The main consequence of this error, he argued, has been the production of misleading findings in relation to trends in educational performance over time that have, in turn, led to misguided and potentially damaging policy interventions. By using gender differences in educational attainment as a case study, this article begins by (...) showing how Gorard's notion of the politician's error has been largely embraced and adopted uncritically by those within the field. However, the article goes on to demonstrate how Gorard's own preferred way of analysing such differences - by calculating and comparing proportionate changes in performance between groups - is also inherently problematic and can lead to the production of equally misleading findings. The article will argue that there is a need to develop a more reliable and valid way of measuring trends in educational performance over time and will show that one of the simplest ways of doing this is to make use of existing, and widely accepted, measures of effect size. (shrink)

This paper is intended to be a consideration of the role of multi-level modelling in educational research. It is not a guide on how to design or perform such an analysis. There are several references in the text to sources that teach the practicalities perfectly well, and the technique is anyway similar to other forms of regression and to analysis of variance. Rather, the paper describes what multi-level modelling is, why it is used, and what its limitations are. It does (...) so in the hope that some readers will be enthused sufficiently to become appropriately critical 'consumers' of research using this approach, so building research capacity, and easing pressures on 'specialist' reviewers. Anyone who can read or perform standard multivariate analyses can understand, referee, or conduct a multi-level model. Additionally, the paper makes three key points. The generally small sample size in each cluster at the lowest level of any multi-level model means that there is a danger of a greater bias in the results than in standard analyses that pool the data from all clusters. Even where there are genuine gains through the use of multi-level models these have to be set against a loss in simple intuitive grasp of the results, especially amongst policy-makers and practitioners. Therefore, long term, we are probably better advised to improve our research designs and the quality of the data we collect than to focus on more and more complex forms of analysis to overcome deficiencies in the datasets we already have. (shrink)