Assessing comparability of optional questions

Abstract

Optionality is any feature in an examination that allows different students to achieve the same qualification without responding to exactly the same set of questions. Where optionality is offered within an exam paper, it's intended that the options should be equivalent in demand. Since we cannot measure demand statistically, we instead consider the relative difficulty of the different options.

However, as the options may have appealed to different ability groups, it's not sufficient to simply compare the mean marks of each option; a more sophisticated analysis is needed. The Willmott-Nuttall index can be used to facilitate the comparison of the difficulty of optional questions but it does not assess whether any differences between options is consistent across the ability range.

This report illustrates possible analyses using two example papers where optionality was offered in the form of a choice between two sets of questions. The various analyses indicate that the optional sets for one of the papers were not of equal difficulty. The difference in performance of these two options was much more noticeable for low-ability students than for very high-ability students, indicating that a simple adjustment to align the options could not easily be made. The analyses also show that this is not a straightforward problem to assess statistically.

How to cite

Harrison, E. (2018). Assessing comparability of optional questions. Manchester: AQA Centre for Education Research and Practice.

Keywords

• Question difficulty
• Comparability