Standardisation: super-standardise me

Craig:  Hello! And welcome to Inside Exams. I’m Craig Barton, a long serving maths teacher. I might have 15 years of classroom experience under my belt, but I can’t say I always feel clued up on exams. So, across this series I’m meeting the people who organise, write and mark national qualifications to gather information that will make our otherwise pretty hectic lives as teachers a little bit easier.I’ve got an Access All Areas pass. So, what is it you’d like to learn more about?

Victoria:  I’m Victoria and I teach Modern Languages, and we’ve heard that those who mark are trained to mark to the same standard. But we were wondering who gets to decide on the standard, and what are those decisions based on.

Craig: Yeah, I think you’ve hit on something really important here, Victoria. We have to feel safe in the knowledge that our students won’t be disadvantaged, or advantaged, as a result of which examiner is randomly assigned to mark their paper. I’ll tell you what this makes me think of. When I walk into a fast food chain, I want to feel safe in the knowledge that I’m getting the same delicious patty and gherkin combo, regardless of who’s on shift that day or which part of the country I’m in. I’m paying the same amount of money, so I don’t want to be short changed because somebody’s ability to salt fries isn’t up to standard.Huge businesses like that must have to relay a minute level of detail to franchisees across the globe, so surely similar principles apply to big organisations like exam boards. I’m going to sit down with AQA Preparation and Support Manager, [Ian White], and Preparation Coordinator, Jo Harland, to find out exactly how examiners are trained to guarantee they’ll all end up marking to the same standard.Ian and Joe. First off, thank you for inviting me to AQA Towers to discuss standardisation. Now, as a maths teacher, there are few things in life that make me happier than some number-based facts. And you sent some absolute crackers through about standardisation that I want the listeners to hear, so here we go. 16,166 examiners attended 920 standardisation meetings. 2,779 examiners attended 370 seeding meetings. 6,289 examiners attended 316 training events. 30,000 examiners standardised online. 7,400 teachers standardised online. And here’s my favourite, approximately 60,000 biscuits consumed by examiners at meetings.So my first question is, what type of biscuits are they eating?

Ian: Well, we have a selection. I think the favourite is generally the chocolate chip.

Craig:  Yeah, very good choice, good choice.

Jo: I get Ginger Nuts. That’s quite popular.

Craig: Nice

Ian: You get them in Manchester but we don’t get them in Guildford. [Laughter].

Craig:  Different biscuits, different tastes. Super. Now, we are going to dive very deep in terms of standardisation. I want to get to grips with the process. But before we start, this may be a really stupid question, but what does standardisation mean and why do we need it?

Ian: The purpose of standardisation is quite simple. It’s just to establish a common standard of marking. It’s the preparation that all examiners or moderators will do before they mark one of the, largely, the Summer Series, where most of the marking is done, but in any of the series. In short, so all the examiners understand the mark scheme that will have been produced when they produced the question paper, and that they can apply it accurately. You know, apply that consistently to the students’ work.Standardisation is essential in making sure that we get the correct marks to the right students and that they’re awarded fairly and consistently across the subjects and the question papers.

Craig:  Okay. Already I’m sensing this is going to be quite a complex process. Am I right in saying there’s not even just one type of standardisation? We’ve got examiner, moderator and teacher. Is that right?

Jo: Yes, so we have the examiner standardisation, which takes place just after when students have sat the exams, in May/June time. Then we have moderator standardisation that happens before that, and that is for subjects which have internally assessed components to them, to make sure that we are consistently moderating them as well. We also have teacher standardisation, which is completely by the teachers; again which is just for internally assessed components.

Craig:  Now, I’m sensing there seems to be a lot of people involved in this standardisation process. Can you just go through some of those people and what role they play in this?

Ian: Basically, for a team of markers there’s a hierarchy. So at the top there’s a lead examiner. They may or may not have been involved in writing the question paper. Some are lead assessment writers and some of them aren’t. Depending on the size of the panel, our largest, I think, is English Language, over 600,000 entries. Hundreds and hundreds of examiners, isn’t it?

Jo: I would say just below a thousand examiners for the English Language and English Literature papers.

Ian:  We’ve got a lead examiner in terms of the marking panel. So a general marking panel, you’ve got a lead examiner. If it’s large enough then you’ll have a set of assistant lead examiners. Then you have team leaders. And then you have examiners.

Craig: Okay.

Ian: There’s a chief and there’s a chair as well.

Craig: Oh, god. I’m picturing this hierarchy in my head here. We’ve got the lead examiners. Where does chief and chair fit? Who’s winning that one?

Ian:  In terms of marking, the lead examiner is.

Craig:  They’re the top dog.

Ian: They’re the top. The chief and chair come into their own more looking across the subjects, and also, they’re involved in the question paper production process, aren’t they?

Jo: Yes. So when you say subjects, obviously there’s a variety of components underneath a subject, so the chair and chief will cover all those subjects. And may even be a lead examiner on one of the components within the subject. So they are a knowledgeable part of the big pyramid hierarchy.

Craig: Well, I can hold off no longer. I want to know about this process of standardisation from start to finish. Now, don’t hold back on me. Go as deep and as detailed as you like. As a teacher, I’m fascinated by standardisation because I want to ensure that, when my kids sit a paper, that they’re fairly judged on what response they’ve given and they get the credit that they deserve. And every teacher listening will want the same thing. So how does it all start – standardisation?

Ian:  It starts with script selection. So the idea is that we select a set of, largely, it doesn’t sound a lot, but largely five standardisation scripts. But those scripts should reflect various levels of response, and across the topic, so cover all of the topics.

Craig: And this is after they’ve been marked. Is that right?

Ian:  No, this is right at the very beginning when the students have first completed a script. So there’s then, generally speaking, a small team – it can either just be the lead examiner or it’s a team of senior examiners based on the size of the panel – and they will be preparing for that standardisation window. Preparing the materials that the examiners will be expected to mark, to help them understand the standard and internalise the  mark scheme.

Craig: Okay.

Ian:  I’ll pass over to Jo for how that process works.

Craig:  Wow, okay. Go, Jo.

Jo:   So what I’ll add in as well is that it’s really important that we select clear responses. So as Ian mentioned, you want a variety of levels but you also want to have clear examples of them. So the aim is not to trip anybody up. So what we do is we get the scripts in. The lead examiner, with maybe a senior team, selects the scripts. And then what we do is, depending on the hierarchy, if there’s an assistant lead examiner, what we have is a pre pre-standardisation meeting

Craig:  [Laughs]. All right, okay.

Jo: [Laughs]. What happens there is you get the assistant principal examiner and the lead examiner in a room, and they start to work through the standardisation scripts, with support from their lead examiner, to make sure that they are clearly understanding the  mark scheme. But also, there might also be some discussion about the  mark scheme, depending on the responses that we get back from the candidates. Because even though the  mark scheme is produced before the exam is set, you never know what a student is going to respond with.

Craig: Absolutely.

Jo: But it’s really important that we capture that, and add that into the  mark scheme to support with fairly standardising all the different responses that we get.

Craig: Can I ask a really daft question at this point, Jo? I’m still not clear how you’ve selected these scripts. Again, if they haven’t been marked but you’re looking for different levels of response, so how does that happen?

Jo: So what the lead examiner would do is they’d have access to hundreds maybe, even up to a thousand scripts, and they will look through them and they will decide which ones show good examples of the variety of responses. So it’s up to the lead examiner to choose those standardisation scripts.

Craig: Got it. So they’re flicking through it electronically, and again, a bit of a sense of –

Jo:  – Yeah.

Craig: – you can pretty much see this child perhaps has made a few mistakes, not answered all the questions, whereas this one looks like they’ve nailed it, so we’re getting the range that way. That makes perfect sense. Okay, I’ve got me scripts and I’m in my pre pre-meeting. I’m having a load of biscuits. What’s happening next?

Ian:  We also, apart from the five [stand] scripts, they’ll also, if it’s marked electronically, will select a set of seeds. On some of the systems they have separate seeding meetings. Some of the platforms, seeding’s kind of built into standardisation rather than there being a definitive mark. So the lead examiner says, right, okay, this question, it should be 12 ... the response that’s given me is a definite 12 out of 30, say. It’s a long answer question, so something like –

Craig: – Yes, so English or something like that.

Ian: – something like that. Now, we have a tendency to say that there’s definitely a range of unacceptable marks but there would also be a range of acceptable marks.

Craig: Oh, wow.

Ian: Because particularly if something is subjective. So on both our stand script and our seeds, they’ll be marked and they will have a range of acceptable values. So the idea is that, with those scripts, yes, they’ll mark them how they feel they would mark them. If it’s a larger panel, that can be a bit of a collaborative and a collective discussion.

Craig: I bet that’s all kicking off in those meetings, isn’t it?

Ian: I don’t want to name out any subjects but yes, psychologists take a long time to make a decision.

Craig: [Laughs].

Ian: I would argue, from my own experience. And God bless them for doing it because they get to the right one.

Craig: Sure.

Ian: But yeah, I mean the importance here, and where’re we’re working with these ... and just to describe our senior team tend to be very experienced teachers. Some of them are still teaching or some of them have retired. And they have an absolute passion for their subjects.

Craig: Yeah, I bet.

Ian:  And I can certainly tell you that they’ve all got their own different personalities, as the question papers are different for the subjects. I used to work in science and I can tell you definitely that your chemists, your biologists and your physicists are very different people. And we need to make sure that we can provide the same level of service and make sure that we’re getting the right results for the students across all these different subjects.I mean, art and design is completely a different beast.

Craig:  Absolutely.

Ian: But the idea is that they will agree in that pre pre this range of acceptable marks for that particular script. And then, when the stand window opens, the examiners or moderators would mark those scripts cleanly, so they can’t see obviously what the marks are.

Craig: And individually, I assume.

Ian: Yeah, individually, absolutely. Most of them will be doing it at home in a two-day window that we’ve notified, and they’ve agreed that they would do it within. And once they’ve marked all of those five scripts, they press a button and they get told whether they’ve answered all the individual items within tolerance or not.

Craig:  Wow.

Ian: Some they will have passed, there’ll be tolerance on, say, on individual items and also on the total mark. At that point, even if they’ve marked all of them correctly, and you know, part of the processes they won’t have done, they will then be shown the range of marks or the definitive marks, and the comments that those seniors would have given it to explain why those marks should be given.So there’s a self-serve learning process going on there. And then, also, what will happen is their supervisor, so if they’ve got a team leader, they’ll be in contact with them and they’ll talk them through that.At that point, and again we’re basing this on our most experienced examiners – the supervisors, the team leaders, the assistant lead examiners, lead examiners – on their judgement they will decide whether that person has understood the standard and can start marking. And that’s at the point that a button is pressed, largely, or a telephone call has said, yes, you can start making the bits of paper again. They go ahead and they start marking at that.

Craig: We mentioned annotations. What does that actually look like? Can you give us some examples?

Ian: The different marking panels will have different sets of annotations. The annotations is literally quite often acronyms, I’m afraid –

Craig:  – Of course.

Ian:  – that the markers will use.

Craig: Can you give us some acronyms?

Ian: Well, BOD.

Craig:  This could be a little game for our listeners here.

Ian: We tend to do this. There are hundreds that you can use.

Craig: BOD?

Ian:  Benefit of the doubt.

Craig: Nice, I like it, I like it.

Ian:  So there’s lots of these, and the mathematicians really use some really bespoke ones. And you can go from some panels that will have maybe ten. As an ideal rule, and our guidance to our examiners is to use a limited amount, particularly for electronic marking now. They used to use quite a lot when they’re marking on paper, and that’s fine. And we tend to have something called a model marked script to explain that, because when people are marking at home, obviously, you can’t see what they’re using.When they’re marking electronically, the lead examiner will agree a set of these annotations, and tools as well, so things where you can underline. Squiggly lines is a big one. A tick with a, you know, different types of ticks. I never knew there was so much variety of tics in the world but there are a lot of different tics.

Craig: That mean different things.

Ian: That will mean different things.

Jo: So you’ve got AO, which is assessment objective.

Craig:  A classic, yeah.

Jo:  Or you’ve got AC, which is your assessment criteria.

Craig:  Nice, I like it. I’m just thinking of annotations I’ve used myself whenever I can’t quite understand what a kid’s done. WTF springs to mind.

Ian:  What I would say is, as part of the pre standardisation process, that the standardisation scripts and the seeds are double, triple checked. They’re checked by the senior team. There’ll be clerical checks done by our own teams.

Craig: I know that, I mean, whenever we sit down as a department and try and do, essentially, a bit of internal standardisation, whether we’re marking some longer form projects or whether we’re doing some mock exam, wherever we’ve got five or six mark questions, it’s all kicking off there. We can’t agree. And this is as mathematicians where there’s supposed to be kind of right or wrong answers. It must be a... I mean, you mentioned... I mean, English, I don’t even know where to start. But then you mentioned art. Like, I want to ... there must be so many back and forths. I’d be failing things left right and centre if I’m trying to mark this. How does it work in those other subjects?

Ian: They tend to have a wider tolerance.

Craig: Right. How wide are we talking? In percent terms.

Ian: Well, you don’t really want to go ... if you’re looking at a total mark, you don’t really want to go 5% much either, either way.

Craig:  No, of course, because again we’re back to disadvantaging the kids –

Ian:  – So it’s relatively tight. It is. The difference that they do for subjects like English is they deal with that issue on the  mark scheme, so they have a level of response  mark scheme. It’s very different to maths where there largely is a definitive. Although, having said that, with A-level further maths I find that, actually, certainly the workings out, can open to debate.

Craig:   I can imagine, oh absolutely, I can well imagine.

Ian: And what I would say for the mathematicians is they like an annotation. Whereas an English, they would tend to use probably tick and comments. And that would probably be it. Because that’s the way that you can explain it.

Jo:    What I know that they talk about a lot is ranking, putting things in rank order. And I think, especially when, as a teacher, one of the things we look for is obviously experienced teachers when we employ people because that’s how you get to understand the ranking order, especially when it comes to English scripts. So then you can understand the standardisation, to a degree.

Craig:  You mentioned the standardisation for teachers that you’d like to talk about. Just talk to me about that.

Jo: Yes, so one of the services we provide to the teachers is teacher standardisation, and it’s really important because it allows teachers to understand the standards of what they need to do when marking internally assessed work or course work, as it might be more widely known. And there’s two ways we do that. We’ve got teacher online standardisation and we have face to face teacher standardisation. And the main difference about moderator standardisation is that you’re marking teachers’ work, which has already been marked by the teachers. So I know Ian mentioned earlier that they use clean scripts for examined. That’s actually a bit different for moderated. Because what you’re doing is you need the moderators to be able to look at the marks teacher has done and make a decision on whether that is the right mark for that piece of work.And the main aim is, obviously, that they will agree with the teacher because what they do for moderators is they don’t mark all the pieces of work. They mark a sample, which is a mixture of the high, the medium and the lows. So it is really important that it gets done right first time. So now I’ll hand over to Ian to go through the exhibition.

Craig: The exhibition. Sounds exciting.

Ian:  So, obviously for art and design the whole process is very different. We can’t mark electronically. I mean, as good as computers are these days, you can’t scan a piece of textile and put it on and it be marked fairly or correctly.

Craig:  Of course, of course.

Ian:     To ensure that the moderators, again, are standardised before they’re sampling the schools and the students’ work, normally, a little bit earlier for moderation, is that we carry out the moderation standardisation. We have meetings in Manchester and in London and you’re looking at hundreds of moderators across Manchester and the London area. Largely, a lot of them will be assessing this work. And so, effectively, what we need to do is we need to put together a travelling art exhibition. Because across art and design, there are many different forms of art that can be produced, they literally, they go round in teams and they stand in front of it and they mark it. And then they all come back to the room and then they have a really good discussion.

Craig: Ian and Jo, this has been an absolute pleasure, this. Once again, this is an area of the examination process I didn’t know enough about. And yet it’s one of the most important ones because, again, whether we’re teachers, whether we’re involved with the awarding bodies, we all want the same thing. We want our students to be judged fairly on what they’ve produced. And standardisation is an absolutely key part of that.So, thank you so much for inviting me to AQA today to talk all things standardisation.

Jo: Thank you. Lovely to speak to you.

Ian:  Yes, you’re very welcome. All the best. [Music].

Craig:  The thing is, before our students reach the exam hall they’ll be relying on our marking throughout the year to gauge their performance compared to their peers. So do we need to be actively standardising our own marking to make sure we’re being fair to all our students? Ryan Wilson is a former Head of English and Assistant Head. And I want to talk to him about how subject teams in schools can ensure consistent marking within the department, and also, against the exam board’s standard. [Music].

Craig: Ryan, welcome to the show.

Ryan: Thanks for having me.

Craig: Now, we’ve spoken before to Ian and Jo from AQA and they’ve told me loads about the standardisation process that happens with their examiners and their team. What I’m fascinated about is standardisation within normal teaching, within normal lessons. So first off, how important is it for you to maintain a standard of marking for pupils in your class?

Ryan: It’s absolutely fundamental to have a sense of the standard that the work’s going to be marked at. Because the way schools are at the moment, as you will know, is that they like data. [Laughs].

Craig: Yes, God almighty, yes.

Ryan: And we can have a discussion about whether it’s right or wrong that teachers are asked for data so often. But I think for as long as that’s the case we might as well try and make that data useful. And so if you don’t know at what level a student in your class is working, then how can you know who’s underperforming? How can you know who ... other ones that need more attention?So I think, if you don’t have a sense of what standard each student in your class is working at, then I’d say it’s difficult to be as good a teacher as you could be.

Craig: Yes, it’s a strong claim but I think I’m with you there. And would I be right in saying that that’s true outside of your class, across your department as well. You need that same standard.

Ryan: I think that’s absolutely true. And it’s really hard.

Craig: I mean, I’m a maths teacher and I have a flipping nightmare. Anything above two marks and anything could happen when it comes to trying to get a consistent standard. But you’ve got essay questions, you’ve got long-answer questions. What are some of the specific obstacles that stand in your way of getting this consistent standard in marking?

Ryan:  When I was Head of English I had a big team. I think there were something like 15 or 16 teachers.

Craig:  Wow.

Ryan: And people just approached marking essay questions in very different ways.

Craig: What kind of ways?

Ryan: Well, it’s funny. There used to be people on the team who would say, almost as a badge of honour, that they were a harsh marker.

Craig: [Laughs] Yeah, I know those types.

Ryan: But it used to do my head in. I would say what you’re saying is that you’re a wrong marker. [Laughs]. Particularly I’m old enough to remember the days of course work and controlled assessments, and that was the last thing as Head of English you wanted to hear, that somebody was a hard marker. You think, don’t deprive the students of those marks.But I think the other thing that’s important to say is that, in those long answer questions, you know, I for AQA examine English Literature, and one of the things we always say in our meetings is that there’s no single right mark.

Craig: See, that is not music to my ears. Because that is just a minefield.

Ryan: [Laughs]. It is a minefield. But I think it’s more of a minefield to claim that there is.

Craig: Right, okay

Ryan:  Because then that’s when teachers start getting stressed and anxious and worried that they haven’t got the right mark. And the fact is that there can never be a single right mark. And what Ofqual say is that there’s a range of possible fair marks and it’s up to the teacher how they justify that.And I think that that’s really helpful to remember when you’re working with your teams. When I work with teachers and we look at examples of students’ work and I ask them what mark do you think this would get, everybody breathes a sigh of relief when you say, there is no right answer here. There might be wrong answers, there will be wrong answers, but there’s no single right mark.

Craig: I’m just thinking here, because I’m picturing our maths departmental meetings whenever we come across a four-mark question. And what we tend to do is we’ll project up a sample student response on the board and we’ll all play the game – how many marks would you give this response. Now, this is maths teachers. We’ve only got, well, five marks to play with. Zero, one, two, three or four. And you can guarantee at least three of those marks will come up within discussion. And people are quite passionate about it. And even when you see the  mark scheme next to it, so you’re not even like guessing in the dark here, people are saying, that one, of course it’s three marks. I’m like, is it three marks? Definitely two. Now, this is a four-mark maths question. What are some of these discussions like in English departmental meetings?

Ryan: And English teachers aren’t known for [laughs] holding back in our opinions. Yeah, I think you’re absolutely right. And people feel very strongly on it. The way we cope with it in AQA, which is also what we get in my department, was that you let everybody have their say and then you have somebody who’s in charge – at AQA that’s the principal examiner – everybody has their say and you have somebody who takes the decision at the end of the day. And there is a phrase which we sometimes use, which I find quite helpful also in the department, which was, okay, you might not have given it the mark that we’re deciding, but can you live with it?

Craig: Okay, I like that, yes. I like that.

Ryan: Can you go home and have your dinner and go to bed and not be worrying about that mark? And that was something that we find useful and people generally say, yes, I can live with it. And there’s also a whole other level of, that’s if we’re looking at, say, an anonymous piece of work.I think, when you’re moderating work that’s already been marked, which is another thing we often do in the department. People bring their, you know, it used to be marked course work. It might be marked mocks these days. You put them all down on a big table. And you’re having to have that conversation where you’re saying to somebody, actually, I don’t necessarily agree with the mark that you’ve given that. and then that’s also difficult. And again, people are understandably defensive of the marks that they’ve given. That’s human nature.What I tried to remember, when I was Head of English, all the time was, firstly, most people want to get it right. Most teachers want to get their marking right. That’s the starting point. And secondly, to understand that people are anxious about making themselves vulnerable and putting their mark out there and saying, this is what I would give this. That’s, on a human level, that’s a vulnerable thing to do. And to try and respond to it in that way. Because ultimately, you do have to have that conversation. But I think there’s ways you can do it that are sympathetic.And we always said, if you think somebody’s mark is wrong, get a second opinion as well. Otherwise you might just have one person who’s wrong changing it to their wrong mark.

Craig: Yes, of course. Do you find that you mark other teachers’ students work different to your own? And again, the reason I ask this is, that’s often something we’ve experimented, as a maths department, because you kind of ... you’re forced to be a bit more objective, I think, when you’re marking someone who you don’t teach. Is that a good thing or a bad thing? I don’t know.

Ryan:  Yeah, I think that’s really true. And again, maybe it depends what the point of the marking is. If you’re looking for that, exactly where the students working at, then I think it’s best to blind mark. Because for the reasons that I guess we were hinting at earlier, that if you teach a class you’ll know a student who’s worked really, really hard and it’s tempting then, subconsciously or otherwise, to give them extra marks for what they’ve done because you know they’ve worked so hard.

Craig: Yes, absolutely.

Ryan:   And all those other ... you know that they’ve come from a difficult home background or you know all those things are in your mind when you’re marking their work. So I think, if you want an absolutely accurate representation of exactly where they’re working then I think you’re right. It’s good to blind mark.But you don’t necessarily want that for every single piece of work, perhaps.

Craig: And it must be, as a rough rule of thumb, the more marks available for a single piece of something, the more difficult it is to standardise. That seems fair, does it?

Ryan: Yeah, I think that’s probably true. In English Literature we mark out of 30 –

Craig:  – Gee.

Ryan: – split into six levels. The process that I find most helpful and what I say if I’m training people to mark for AQA is, well, you first of all work out which level it’s in, and that’s not too difficult because at level one they’re kind of retelling the story. At level two they may be adding some quotations to that.

Craig:  So this isn’t the kind of specifics, it’s just a general feel, is it, from reading the full piece.

Ryan: Exactly. And then, once you’ve got a level, you might decide, let’s say it’s in level three, which is 11 to 15 marks, and then you’re just deciding how secure is it within that level. So you break it down a little bit rather than trying to pluck a number.

Craig: That’s interesting. I just want to go practical then. So let’s imagine you’re working with your department and you’re trying to improve this standardisation process. We know it’s important, we want the kids to be getting the marks that they deserve and it shouldn’t matter whose class they're in or who’s marking their work. They should be getting, essentially, the same mark at the end of the day.What are some of the practical ways that you help that happen? I don’t think there’s anything better than an actual piece of work that’s been marked, with justifications, to use with your team or with your students. It’s so powerful, isn’t it?

Ryan: Absolutely. And AQA provides full marks answers, which I think have their role and are really useful, but actually, kids love nothing better than getting some work that they can critique and say, why did you do this or, you know.

Craig:  Absolutely.

Ryan: And it can work really well in lessons, I think.

Craig: So you come back to your department with, you’ve got these incredible resources, you’ve got these kids. Work that you know how it’s been officially marked and why and so on. What are the practicalities of running that meeting? Is it everybody has a go on their own? Or do you work in pairs? What do you find works best?

Ryan: Yeah, I’ve done it different ways. What I’ve sometimes done is gathered groups of teachers to look at responses in particular levels. So I might have a group of teachers looking at level five and six. That’s going to be grid seven, eight, nine responses. Another group looking at level three and four, and another group looking at one and two. And then they kind of feedback and teach the rest of the department about the ones that they’ve particularly been studying.I think, whatever you do, you want to just avoid putting people on the spot of what mark would you have given this. Because that just makes people feel uncomfortable and exposed, and there’s no need for that. It’s a learning exercise.Again, working with teachers through AQA, that’s the thing that people fear, that you’re going to be asked on the spot, what mark would you have given this.

Craig:Absolutely. And it can almost be... it can be quite a fun process, can't it? I mean, okay, I’m an absolute nerd for this kind of stuff but, essentially, it’s a bit of a game trying to guess what mark this got given and why it was given it and so on and so forth. It’s fun and really enlightening, isn’t it?

Ryan: Yes, absolutely, I think it definitely can be.

Craig:  Here’s a question for you – where does, if anywhere, where does this notion of comparative judgement fit into this? Because what we found dabbling is that whilst it can be quite difficult even in mathematics to assign and mark to a four or five mark question, what's a lot easier is certainly to say, well, this is a better answer than this answer. And this notion of getting a bit of a rank first before assigning marks, we found, just in small scale experiments, seems to be quite... it seems to improve the process a little bit.Have you dabbled and have you found this or not?

Ryan:  I think that’s really true, and particularly when you’re working with new teachers, marking is one of the areas that they often find very intimidating. And they’re worried, particularly if they’ve got exam classes, about getting it wrong. And actually, again, one of the things that I often say to new teachers is all we’re doing is putting them in a ranked order –

Craig:  – Yes, absolutely.

Ryan: – against the  mark scheme. And I think you’re exactly right, that that’s a useful way of doing that. And again, with [NQTs] or even trainee teachers, the thing I always start with when we talk about marking is, well, here’s five essays. Do you want to just go and read them and come back and we’ll talk about what rank order you put them in. And then we can talk about, well, this was level six, this was level five, level four and level three. And they’ve done it themselves without even knowing what they were doing.And it takes away all that mystery and fear.

Craig: Do you find in English that the  mark schemes can be quite difficult to follow? Is it a bit of an art form? And how can, and particularly we’ve mentioned inexperienced teachers, how can teachers get better at interpreting the  mark scheme?

Ryan: I think they are necessarily a little bit difficult, and I’m sure English is not the only subject this applies to. But they tend to be, you know, at level six there’s excellent analysis. At level five there’s very good analysis. And obviously, one person’s excellent is another person’s very good. So I just think there’s no substitute for looking at the marked students’ work.And of course, now you can get back students’ work as well after the exam series, so you can get their actual exam paper back and you can have as many exemplars there as you want. And that’s also helpful.

Craig: Let’s talk specifically about being an examiner. How did you find it helped, first with your own teaching? What were some of the practical ways that it improved your own teaching?

Ryan: I think it gives you confidence because you’re seen how the process works, you’ve seen real examples of students’ work. And I just feel, going into the classroom, that I feel I know what I’m doing. More so than before I examined.I also think it’s useful for dealing with misconceptions. Because with English Literature, and again I’m sure this is the case with all subjects, there are so many rumours spreading on Twitter and social media about what examiners are looking for. We have AO3, which is ideas and context, and there is a really widely held belief amongst teachers and, therefore, students, that they had to write a chunk of history in every literature essay. They had to have a bit about, whatever it was, the Battle of Hastings or whatever.And from marking, I just knew that that’s not true. So it helps, I think, with dealing with misconceptions. And also, the kids quite like it. The kids quite like it when you say, you know, I’m not going to be marking yours but I’m going to be marking ... I think it gives them a sort of a confidence in you as well.

Craig:  What about the practical benefits to your department and to other colleagues, of you being an examiner?

Ryan: I’m a little bit biased on it, obviously, but I feel that every department should have at least one marker. Because it’s absolutely invaluable. For all the reasons that we talked about. Having a standard across the department is important because it underpins so much of the work of the department, of the school. And if you’ve got somebody who’s got the authority and the confidence to go in and say, well, actually, this is what they’re looking for, this is the way it’s marked, this is what I learned from the principal examiner themselves, then that’s massive for a department.

Craig: And you have access to the resources as well that that can bring in.

Ryan:  Absolutely.

Craig: So to kind of wrap up and think of practical things here. So one thing we’re definitely saying is, if possible, either you or one of your colleagues in the department sign up to be a marker, because it’s going to be great for you, it’s going to be great for your department. In the short term, if people can't do that but they want to tap into some of these ... they want to get this consistent standard in their own marking, whatever subject it is, and across their department. Where should teachers start with this?

Ryan:  One of the key things, and I’m sure a lot of heads of departments who are listening to this will empathise, is that you need a leadership team who will give you time to do it. To sit down with your team and talk about these things. You know, I was always saying to the head, please can we have some time on a certain day when we’re not all sat in the hall but actually when we’re ... give us a full day to sit down with some kids’ work and just go through it as a team.And that thing we were talking about earlier of looking at students’ marked work and just really trying to work out how can we get it consistent across the team. I mean, getting a leadership team that will give you time to do that, I think, is really important.

Craig: I think you’re right. And it’s time dedicated to that and not bogged down with the usual nonsense that often this departmental time gets sucked up in.

Ryan:  Absolutely.

Craig: Well, Ryan, this has been again another revelation to me, this. You could not pay me to be an English teacher. I find it hard enough with maths. I’m going to stick with that for now. You have my deepest sympathies. But thanks so much for your time.

Ryan: Thanks a lot for having me. [Music].

Craig: Well, I’m certainly appreciating the importance of standardising marking across departments. But it’s also interesting to hear there is sometimes merit in adding a bit of your own human judgement to classroom marking too. If you want some extra homework, you’ll find the top benefits of using teacher standardisation for your NEA subjects in the podcast show notes. And Episode 6 of Series 1 on Grade Boundaries might be interesting after today’s chats too.I’ll be back in a fortnight to learn more tricks of the teacher trade. But in the meantime, make sure you rate, review and subscribe to the podcast.You can also join the conversation and ask your own questions on Twitter using #insideexams.Until next time, goodbye.