The art of setting questions
By Victoria Spalding
Published 22 Sep 2015
I’ve booked this great holiday next week. Can you tell me the best way to get there?
If I asked you this question your first response would be: where precisely are you going? Similarly, question writers need to know exactly what they want to test. This process is not as simple as it may first appear.
What is the best way to get there?
If my holiday was in Spain, you might suggest I fly, but if Bognor Regis was my destination then this would not be appropriate. Likewise, there is no point trying to assess physical skill in dance via a written exam. The mode of assessment must be the most valid way of testing the knowledge, understanding or skill the item writer is targeting. (An item is the whole question, instruction, stimulus and mark scheme). Even within a written exam there are different types of items. Multiple choice questions are good at testing recall over a wide topic range. Short answer questions are good at testing recall, application and understanding. Essay questions can test higher order skills, such as analysis.
You will need to know the exact destination in order to give good directions. Item writers need to be precise in order to enable the candidate to demonstrate the skills they seek to assess. If the examiner does not know exactly what they want to test they cannot write a valid assessment. Sounds obvious, but it is much harder than you might think.
Item writers need to know how to control the demand of their questions to set them at an appropriate level. Giving GCSE foundation students A-level difficulty items would not be a good way to assess what they have learned over the course of their study.
How we can lose our way
Consider this question:
And now the mark scheme:
On the face of it, the question appears well written. The response includes four key points: ‘gravity’, ‘makes the glacier move slowly’, and ‘over time’ it ‘smooths the surface’.
The question asks for an explanation yet the answer, given full marks, does not explain at all. The mark scheme is not rewarding the skill the question intends to assess – the candidates’ ability to explain a process. A candidate who writes all the words they know on the topic will do better than a candidate who wrote ‘it scrapes the land like sandpaper’ (2 marks) – despite demonstrating a greater understanding of the mechanism of glacial abrasion. The mark scheme is rewarding the wrong skill and is invalidating the item. Had this question asked for a description then the mark scheme would have been appropriate. However, as it is asking for an explanation the item writer should have used a mark scheme like the one below:
The importance of importance
Keeping items valid is crucial for good assessment. Assessment researchers Alastair Pollitt and Ayesha Ahmed came up with the following model to ensure that validity drives the question-setting process.
Item writers need to be creative but also methodical in their approach: question setting is both an art and a science.
About our blog
Discover more about the work our researchers are doing to help improve and develop our assessments, expertise and resources.