Professional Development

In this video, we're going to be looking at understanding students’ minds, and particularly looking at where misunderstandings of questions can occur, and what causes these.

So, I'm going to talk a lot in this video about misinterpretations and where they come from. But I think it's worth taking a big step back at the very beginning just to think about the whole process of assessment taking, and where all of those misinterpretations can creep in. So, what we have, effectively, is a process where a lot of people are communicating with one another in written form alone, without that ability to ask questions or qualify where things are unclear.

So, at the start of the process, we have the question writer. They have to write the question and the mark scheme as clearly as humanly possible so that the student has a good understanding of the task in the exam hall and markers know how to reward the response they see in front of them. That student, bearing in mind they are in exams halls, they might be feeling a bit nervous, and they are a novice, has just the words written on the paper to explain to them what they need to do in order to answer that question. So, they have to interpret that question, hopefully, in the way that the question writer had intended, and then come up with their answer and their response in the exam hall as clearly as possible as they can in the conditions. That marker then has the really difficult task of looking at the student's response and trying to interpret what the student meant by that, and trying to infer what the students know, understand and can do from their written response without being able to ask the student if they are a little bit unclear about what the student meant – as well as looking at that mark scheme to make sure that they have interpreted that correctly, to make sure that they are rewarding the right things as the question writer had intended, and any marker training or guidance they may have received in standardisation. Once they've done all of that task, they come up with a final score or mark which hopefully reflects what the students knows, understands and can do.

As you can see, there's quite a lot of interpretations that are happening in this process, and if there is a misinterpretation at any step along the way, it might mean that that final mark or score doesn't truly reflect what the student knows, understands and can do and would be classed as measurement error. So, it's really important for us to try and minimise that ability to misinterpret it as much as we can here at AQA. So, in order to understand what's happening in students’ minds during the test-taking process, we need to really delve into the world of psychology. And one of the first things I'm going to talk about is working memory. And one of the best things to do to show you working memory and its limitations is to set you a little task.

So, what I want you to do in a minute is a quick memory recall game. You're going to need a pen and paper, so if you don't have one already, it's worth pausing the video now to go and get a pen and paper. Okay, so once you've got your pen and paper, what's going to happen next is a series of words are going to flash up on screen. Once they’ve finished, you're going to need to press pause on the video and simply write down as many as you can remember. When you're done, press play again, and we'll go through the answers. Okay, that's all the words, you're going to need to pause the video now, in order to have a chance to write down as many as you can remember.

Great, well done for giving that a go. What you've just done is a psychological experiment that was conducted by Miller, and what Miller was trying to measure was our ability to remember information. It was called our 'working memory'. You can kind of think of working memory as the part of your brain that can hold information while simultaneously processing it and remembering it. It's a little bit like RAM on a computer, if you think about it that way. The thing about working memory is it is finite – there is only so much that we can hold in our mind at any one given time. And what Miller found in this particular experiment was that people could remember somewhere between five and seven bits of information. So, if you remembered five or seven words, that means you're roughly average to most people's working memory capacity. If you remembered fewer, please don't worry. Ultimately, working memory is only really a snapshot of what your mind is doing at the moment. It may well be that you had lots of other things on your mind that may be distracting you from the task of remembering those words.

So, working memory will definitely impact students in the exam hall, and the amount that we ask of their working memory in our assessment is referred to as 'cognitive load'. So, we want to be really mindful that we're not asking too much of students in terms of how much they need to simultaneously process when writing questions. Students which are anxious are also far more likely to have problems with their working memory, because some of those bits of information that they can hold in their mind are occupied by things like 'What happens if I don't do well in the assessment?', 'What happens if I don't get the grades I need?'. They may well be thinking about how far they are along in the test, they might be clock-watching in the exam hall. And this is where you find students who could quite happily answer this question in the classroom really fall apart in the exam hall. And it's because their working memory is reduced so much by that anxiety level. So, it's something that we're quite mindful of when we write questions and thinking about the cognitive load of questions.

So, as well as working memory, when we're thinking about what students are doing in the exam hall we also need to be really mindful of what's going on in our minds as we read. So, one of the things to be thinking about, first of all, is how we read, and the fact that as soon as we read something, real time processing is occurring. We are building meaning of that sentence as we read it. And the way that this happens is often on quite an unconscious level, and it can really cause students, should we write the question a certain way, to misunderstand what they're reading, in terms of what's on the question paper.

So, in order to bring this to life, I'm going to tell you a little joke. I told my doctor I broke my arm in two places. He told me to stop going there. Why does that joke work? The reason it works is because you’re building understanding in real time. So, when you hear the first sentence about telling a doctor you broke your arm in two places, you're thinking about breaking the arm in two separate places of the arm. The reason you smile at the second sentence is because it's not matching what you had already built. When the doctor says, 'Stop going there', you realise the doctor's talking about going to two different places, and that's not what was going on in your mind. When you put this in the context of the exam hall, students are doing exactly the same thing. Information they pick up early on in the question, they already start building a mental representation about what that question is about. And sometimes what can happen is information that comes up later in the question, if it doesn't match that representation they've already built, can just be missed by the student. If their working memory is already at threshold, if they're very anxious, what can happen is they can fail to spot that this information doesn't marry up and disregard it subconsciously. They don't even know that they've done it. So, this is one of the things that can happen when students are answering questions, and it means that we have to be really mindful of that when we write our questions to make sure that there is a real consistency in the way that we're giving information, and students aren't likely to go off on the wrong tangent.

The second thing that we need to be mindful of is the question-answer process, and all of the unconscious activation in students’ minds with every word they read on that test paper. For example, it has been proven in studies that the sentence 'Yesterday I paid a large cheque into my account' subconsciously activates the word 'river' in your mind. Now, I'm sure you wouldn’t have consciously thought 'river' as I read that sentence, but what happens is that the word 'cheque' and 'accounts' activate the words 'bank', and 'bank' activates the other meaning of the word 'bank', ‘river’. This is all happening entirely unconsciously in your minds, but the same is happening for students. As soon as they see any words, subconsciously loads of activations and thoughts are happening, and then they have to filter out all the ones which are irrelevant, such as 'river', and make sure they home into the ones which are relevant. Again, for students whose minds were already got anxious, high-working memory demand, this subconscious filtering can go awry, and sometimes they end up activating the wrong thoughts due to what's there on the test. So, we have to be really thinking about that when we write the sentences, that we put our questions there, and try and put them on the right path as much as possible.

So, when it comes to this unconscious, real-time processing, test writers must do their utmost to minimise unconscious misinterpretations of questions. And as teachers, it's best to prepare your students by really good exam technique to support them, particularly those anxious students, to make sure they take the time to really carefully read those questions and make sure that they have got the full understanding of what the task is.

So, students have expectations of what an exam will include, and when a context conflicts with those student expectations, sometimes misinterpretations can occur. In this first example, the question is asking about a multipack of crisps. Having seen multipacks, students subconsciously will start thinking about questions like these. And commonly, a question with a multipack would be asking students to divide to find the price of one bag. The most common error for this type of question is students trying to divide £3.45 by five or eight. Students are failing to spot that this question is a multiplication question. In this next example, the context is in keeping with the maths involved. You were given the cost for a day, and you need to work out the cost for a week, keeping with the student's expectations of what the question is asking.

So, here is another example of this in action. This example came from one of our engineering papers, and it's actually a draft item. When this question came to the question review process, where all of our writers sit together to review the question papers in draft form before we put the paper together, there was some concern raised about this question. So, this question is ultimately about composite materials. But what the question writer had done on the first draft was to include a picture of a car. And the interesting thing about including images is, what we do as humans is as soon as we see the image, that is the first thing that we will look at. So, if you are reading a newspaper, or an online article, or a magazine, if there's a picture there, you will look at the image first, and then you will read the headline. And having seen the image, that will affect how you process that headline and any other information that follows that, because images are really salient to us as human beings. So, in this particular instance, when the student had seen the car, which is probably what they’d have looked at first, they may be starting to think about composite materials involved in car manufacture. That's not actually what the question is asking. And it might be, having seen the car, they may fail to come up with answers such as concrete or MDF material because that's not really involved in car manufacture. So, having reviewed this question, the team decided it was safer to take the image of that car out, because it might have just narrowed the range of responses students would have given in the exam hall. So, we have to be really careful, particularly around images, in making sure they’re activating the right thoughts and they’re not misleading students, or going to lead to misinterpretations, just because they are so very strong and salient to processing the question.

So, there are a couple of key things to take away from this section on understanding students' minds. For question writers, we need to be really mindful that assessment is a process which requires interpretation and is vulnerable to error. Test writers must do their utmost to minimise unconscious misinterpretation of questions. We aim to minimise cognitive load and ensure that the question contains the minimum amount of information required to answer the question. We avoid illogical contexts that could trip up students in the exam. And we use visual resources carefully, to avoid activating the wrong expectations from students. So, as a teacher, you need to be really aware that students make mistakes due to unconscious processes and working memory difficulties, and that we need to support them. So, we need to help students identify the key bits of information for answering the question. We need to help students pause and re-read questions to prevent expectations interfering with their processing. And we need to help students identify the command words in the questions and form a clear understanding of the task.

So, hopefully this video has given you a little insight into the question-answer process and understanding what's going on in students’ minds. We've talked about the role of working memory and how to minimise the opportunity for students to misinterpret questions insofar as possible.