Context in exam questions: the important dos and don’ts

By Phoebe Surridge
Published 31 May 2022

Phoebe Surridge highlights some of the research we’ve been doing to make our qualifications as accessible and inclusive as possible.

The publication of the Ofqual consultation on designing and developing accessible assessments has put the spotlight on ensuring that exam questions test the knowledge and skills they’re designed to test.

This consultation has guided some of our recent research activities, particularly around how we can make sure that accessibility considerations are embedded right at the outset of the question-writing and paper production process. This will better enable us to remove any barriers that may prevent students demonstrating their full potential. One aspect of this work has focused on how to use context appropriately in exam questions.

For a teacher in the classroom, using real-life examples or contexts can be a great way to spark interest and explain topics to students, but how might this play out in an exam situation?

A student unfamiliar with the context of an exam question might end up panicking and doubting their ability to answer correctly because they feel they are missing some essential knowledge. On the other hand, appropriate context can make concepts more familiar for students who may not yet be invested in the subject and can also test students’ ability to apply learning to real-life situations.

At AQA, we think very carefully about how to frame questions, with the aim to provide all students with the opportunity to demonstrate the knowledge, skills and understanding they possess. Research plays an important role in underpinning this process.

Organising and interpreting information

As we go through life, we use what are described in psychology as ‘schemas’ to help us make sense of the world around us. For example, when a young child sees Garfield – a comic-strip cat – for the first time, they might not immediately recognise him as a cat. However, once they know that he is, his features are added to their schema about cats. Subsequently, they might be able to recognise that animated characters like Tom or Felix are also cats. Schemas provide us with a script of what to expect. However, sometimes the schemas activated can be misleading. For example, we might expect a cat’s favourite food to be fish, based on what we have seen them eat, but Garfield prefers lasagne!

Exam questions that are set in context activate schemas within students’ minds. However, if the schemas activated are not relevant to answering the question, students may be misled.

In one research study, students were presented with two versions of the same chemistry question. One version was accompanied by context about toothpaste and an image of toiletry products while the other used the context of book preservation and an image of books. When asked to identify the product of the chemical reaction in the question, some students who were presented with the first version responded with answers such as ‘soap’ or ‘shampoo’, presumably because they saw the word ‘product’ combined with the image of consumer products. This was not the case for students who were presented with the second version of the question. It is likely that the context in the first version misled some students – who may have actually known the correct answer – by activating schemas unrelated to the science behind the question.

Supplements to context questions such as pictures or tables may also negatively impact students’ performance, as they can introduce irrelevant information and increase cognitive load. If supplements are included, they need to be relevant. Including images that do not provide information relevant to answering the question, for example, may lead to students wasting time.

Context and student accessibility

The use of context in exam questions can interact with other student characteristics to advantage or disadvantage particular groups of students.

In subjects such as maths, the extra reading and potentially confusing language introduced by context may unfairly impact students for whom English is an additional language, especially if reading comprehension is not part of what is being tested.

Using contexts that may be more familiar to some students than others can introduce bias and create unfair advantage. It is also worth considering how traditional gender roles and stereotypes may influence the way students engage with exam questions.

As an exam board, we know that we need to bear in mind the day-to-day lives and experiences of our students and how these could affect their approach to contextualised tasks. There is some evidence that socio-economic background can have an impact on how students respond to questions set in various contexts. Providing opportunities for students to practise answering these types of questions might be one way to help ensure fairness. Research into the use of context in maths has suggested that students do not approach context questions differently to regular questions, provided that they have encountered similar styles of contextualised problems in a classroom environment.

Getting it right

To sum up, context questions can be a handy addition to the question writer’s toolbox, provided that the context does not introduce irrelevant information or lead to any disadvantage or extra strain for students. However, research has shown that context should not be used simply to make a task more interesting. Well-written, real-world context – of the type that students will have come across in the classroom – can enable students to activate schemas related to the knowledge needed to answer the question. It can also assist with students’ recall and their ability to apply their learning. For now, then, perhaps the best advice to question writers is to ‘use with caution’.

As we look to the future, these are some of the lessons and principles that we will continue to embed in our assessments so that every learner has a fair opportunity to demonstrate their ability.

Keywords

• Fairness
• Question difficulty
• Social equity
• Validity