Do marking reliability studies have validity?

Abstract

Marking reliability studies are essential to ensure that examiners' marking of high stakes public assessments is appropriate, consistent and fair to candidates. The importance of such studies is confirmed by an extensive marking reliability literature which spans the early twentieth century to the present day, and covers many different forms of examination papers.

There are many challenges in designing a marking reliability study, and perhaps the most significant of these is deciding whether to undertake research in a live or non-live examination context. While a live context has the greatest external validity, it is not always feasible or fair to candidates or examiners.

Alternatively, using a non-live context allows for greater control of the research environment but brings into question the validity of the outcomes. For example, if participants are aware that the examination scripts have no bearing on candidates' real examination results, will they mark the research scripts with the same diligence as live scripts.

This paper overviews the outcomes of a questionnaire distributed to 89 examiners who took part in a controlled, experimental marking reliability study. The questionnaire was designed to compare the degree of conscientiousness applied during the experimental study and under normal live marking conditions. In doing so, the questionnaire aimed to gauge the representativeness and naturalness of participants' behaviours and thus contribute to assessing the ecological validity of the primary (marking reliability) study.

It is found that a small proportion of participants acknowledged that they responded differently during the study. It is suggested that these differences could represent a threat to the validity of the study. It is concluded, however, that this is an inevitable drawback of controlled, non-live research, and that this is counter-balanced by the important insights into the reliability of examiners' marking that such studies give us.

How to cite

Chamberlain, S (2008). Do marking reliability studies have validity?, Machester: AQA Centre for Education Research and Policy.

Keywords

• Reliability
• Transparency