Randomized controlled trials in educational research tend to be small. Small trials can have large, chance, imbalances in important covariates. For studies with sample sizes greater than 50, chance imbalances can be corrected using analysis of covariance; for small trials, however, statistical power is maximized if the trial is balanced and analysis of covariance is used in the analysis. The aim of the present study was to discuss methods of improving covariate balance in trial design and to demonstrate the method (...) of minimization. Using an exemplar of a cluster or class‐randomized trial with 29 classes, we employed minimization to achieve covariate balance. Minimization achieved good balance on four prognostic variables. Many trialists in education use restricted forms of allocation, including pairing or stratified randomization. These approaches have disadvantages. Another approach rarely used in educational research is minimization. Minimization uses a simple arithmetic algorithm to produce balanced groups across a number of important covariates and should be more widely used in educational and psychological research. (shrink)

Randomised controlled trials (RCTs) are often seen as the 'gold standard' of evaluative research. However, whilst randomisation will ensure comparable groups, trials are still vulnerable to a range of biases that can undermine their internal validity. In this paper we describe a number of common threats to the internal validity of RCTs and methods of countering them. We highlight a number of examples from randomised trials in education and health care where problems of execution and analysis of the RCT has (...) undermined their internal validity. However, awareness of these potential biases can lead to careful planning to avoid or reduce their occurrence. If good quality randomised trials are to inform policy and practice in education then rigorous trials need to be designed that are the least susceptible to threats to their validity. (shrink)

It is only fair to hold someone accountable for outcomes over which they have sufficient control. The evidence-based approach to education (“evidence-based education,” or EBE) promises to give educators sufficient control over their students’ outcomes by providing access to interventions that are effective according to scientific research. I argue that EBE fails to secure sufficient control because the research on which it relies doesn't establish that interventions are generally effective. If they are to be fair, accountability practices must reflect the (...) limited control educators have, even when using evidence-based interventions that have improved outcomes in other settings. I glean relevant insights by considering accountability for medical practitioners within the context of evidence-based medicine (EBM). (shrink)

This paper is intended to be a consideration of the role of multi-level modelling in educational research. It is not a guide on how to design or perform such an analysis. There are several references in the text to sources that teach the practicalities perfectly well, and the technique is anyway similar to other forms of regression and to analysis of variance. Rather, the paper describes what multi-level modelling is, why it is used, and what its limitations are. It does (...) so in the hope that some readers will be enthused sufficiently to become appropriately critical 'consumers' of research using this approach, so building research capacity, and easing pressures on 'specialist' reviewers. Anyone who can read or perform standard multivariate analyses can understand, referee, or conduct a multi-level model. Additionally, the paper makes three key points. The generally small sample size in each cluster at the lowest level of any multi-level model means that there is a danger of a greater bias in the results than in standard analyses that pool the data from all clusters. Even where there are genuine gains through the use of multi-level models these have to be set against a loss in simple intuitive grasp of the results, especially amongst policy-makers and practitioners. Therefore, long term, we are probably better advised to improve our research designs and the quality of the data we collect than to focus on more and more complex forms of analysis to overcome deficiencies in the datasets we already have. (shrink)