Marking consistency over time

Abstract

It is of great importance that examination bodies award candidates grades which correctly reflect the quality of work presented because, ultimately, examination results can affect life chances. Marking should be at a common high standard and free from bias, otherwise some candidates are placed at an unfair advantage and others at an unfair disadvantage.

In the past there has been much research into the reliability of marking (for example, Newton, 1996; Branthwaite et al., 1981; Murphy, 1978) and into the existence of marking bias (for example, Baird, 1998; Massey, 1983). Only a few studies, however, have considered the longitudinal reliability of marking, determining the influence that script sequence, or moment in time, can exert on the accuracy of assessment.

Spear (1997) examined the biasing influence of contrast effects and found that if a good piece of work was assessed before work of a lower standard, then the poorer-quality work would be assessed more harshly. On the other hand, if higher-standard work was preceded by a piece of lesser quality it would be assessed more favourably.

In a ten-year retrospective study Lunz and O’Neill (1997) showed that, although individual judges vary in their level of leniency, the leniency of most judges remains internally consistent throughout, in spite of retraining. This study corroborated earlier evidence presented by Lunz and Stahl (1990), where it was shown, albeit over a substantially smaller time scale, that judges’ leniency is reasonably consistent over time notwithstanding some variations across grading period.

Such findings are of considerable interest in the context of the public system of examining in England, Wales and Northern Ireland. After standardisation meetings, examiners assessing any of the national qualifications are required to mark an allocation of scripts over a period of two to three weeks. Anecdotal evidence provided by senior examiners and awarding body staff suggests that, as time since standardisation increases and pressure to complete the marking exercise increases, so the accuracy of examiners’ marking declines.

However, as the pressure increases for examiners, so does the pressure for those monitoring examiner performance. Work load is prioritised to focus on those examiners causing the most concern and, as such, the anecdotal evidence may be flawed, as the checks may be biased towards the poorest examiners at the end of the marking period. In the context of A-level examinations, this study sets out to explore the view that marking accuracy decreases through the marking period.

How to cite

Pinot de Moira, A., Massey, C., Baird, J-A. and Morrissy, M. (2002). Marking consistency over time, Research in Education, 67.

Keywords

• Marking
• Reliability