Welcome to this session looking at some aspects of the assessment of A-level Chemistry.
My name is Richard Grime, and I am one of the senior examiners on the AS and A-level Chemistry team.
The purpose of this training video is to examine the structure and outline of the assessment of A-level Chemistry, the use of ramping in questions to provide differentiation, some key features of the design of questions involving mathematics and practical work, the use of multiple choice questions and, finally, questions that use level of response mark schemes. This session follows on from the more general “What makes good assessment“ videos which you have hopefully watched already. Here is a reminder of some of the key principles covered in that video, with a strong focus on developing assessments that are valid.
In A-level Chemistry there are 3 assessment objectives. The first assessment objective, referred to as AO1, is “demonstrate knowledge and understanding” of the chemistry on the specification. 30% of all marks in the qualification are based upon AO1. AO1 questions can include definitions, for example, asking the definition of electronegativity or what a carbon-neutral fuel is. An AO1 question could include knowing and explaining trends in physical/chemical properties such as the trend in ionization energy across a period or the trend in reducing power of halides down Group 7. It could also include knowing what happens in reactions and why, for example what happens when aqueous iron(II) and iron(III) react with sodium carbonate and why their reactions are so different from each other.
The second assessment objective is applying knowledge and understanding in unfamiliar situations – and this carries the greatest weighting with 45% of the marks.
Questions assessing AO2 skills might include many calculations, most organic reactions and mechanisms, or many explanations, for example, explaining how a specific buffer solution works. The final assessment objective AO3 involves analysing, interpreting and evaluating information - and this carries 25% of the marks. AO3 style questions might include, for example, looking at spectra and/or test-tube reaction results to identify a compound; suggesting what might happen in an unfamiliar reaction ; suggesting improvements to an experiment method or perhaps designing an experiment method.
Now for an activity to check your understanding of the assessment objectives. In this first one, look at this question and decide whether it is examining AO1, 2 or 3. You may wish to pause the video while you read the question and decide upon your answer. Hopefully you chose Assessment Objective 1 as this question requires an answer stating a definition - it’s simply a student demonstrating their knowledge of the term enthalpy of lattice dissociation.
For this second question, consider which assessment objective this is assessing. Again, you may wish to pause the video while you decide. This question is assessing AO2 – the student needs to apply their knowledge and understanding of enthalpies of hydration and solution to create a Hess cycle to find the enthalpy of lattice dissociation for ammonium nitrate. Now for the final question reflecting on assessment objectives. You know what to do by now, please feel free to pause the video to give yourself time to read the whole question. This question focuses upon AO3 – the student needs to analyse and interpret the information provided, and then evaluate it to suggest a change that reduces the apparatus uncertainty in the temperature change.
When assessment writers write questions, as part of ensuring our assessment is valid, we have to ensure that we assess each of AO1, 2 and 3, and in the right proportions. Now we are going to look at the assessment structure across the 3 A level papers in relation to their content, style, and their weighting towards the overall grade.
Paper 1 covers inorganic chemistry and the relevant physical topics (e.g. electrode potentials, acids and bases) and it requires both short and longer written answers. It’s worth 35% of a student’s total attainment. Paper 2 covers organic chemistry and the relevant physical topics (e.g. rates of reactions) and like paper 1 it requires a mixture of short and longer written answers. There are some physical topics that are covered in both papers 1 and 2, including most AS topics such as structure & bonding or energetics. Paper 3 is a bit different: it can cover content from the whole course and has a stronger slant on practical work. In addition to the short and longer written answers it also contains 30 multiple choice questions allowing areas of the course not assessed elsewhere to be assessed. This paper contributes 30% towards a student’s overall grade.
Now we shall look at some key features of question writing. Command words are imperative as it is these that inform students exactly what they should do. Here is a standard list of command words that are used in questions and if you write your own bespoke assessment material for your students you should look to stick to these command words. Each has a specific meaning, but these terms are generally well interpreted by students; for example, students generally know if asked to state how something might change (typically worth one mark) how this differs from being asked to explain why something changes (typically worth several marks). Assessment writers are very careful to choose the appropriate command word to ensure students do what we want them to do to assess what we are looking to assess.
Another key feature of many questions in Chemistry is ramping. We set questions with different levels of difficulty in order to differentiate across the whole grade range, which helps with the validity of the assessment. Often, different parts of the same question have different levels of difficulty. We often design questions with the easier parts at the beginning and the harder parts at the end, to make questions as accessible as possible. We will look at some examples of this in a moment, but first of all we shall look at some of the ways in which we analyse the effectiveness of questions. The performance of questions is very carefully analysed and evaluated after the exam. This helps in designing future questions. The following charts are examples of two of the tools that are used.
Firstly, this chart shows for a 6-mark question what proportion of students scored each of the available marks from 0-6. On this question there is quite an even spread. However, this second chart on the right is even more valuable as it shows on average what % of the 6 available marks students who finished with grades A*, A, B, C, etc. attained on this question. You can see that A* candidates scored 80% of the marks available on this question, whereas E grade candidates only scored 14% of the available marks. This question, from 2023 has a very good spread showing that it discriminated well between candidates across the whole grade range. This type of diagram can help to show how we ramp questions.
Here is the analysis of a 4-part question from paper 2 in 2023. The data suggests the first two parts, 7.1 and 7.2, were more straightforward as all students performed well, including students who eventually attained an E-grade who, for example, gained 70% of the marks available on 7.1. However, as the question is ramped to being more challenging in part 7.3 , we can see attainment drops. For example, grade C candidates only attained 23% of the marks. Part 7.4 was even more challenging with even A* candidates only attaining 54% of the available marks. This is a classic ramped question as it gets progressively harder, allowing lower grade candidates to demonstrate knowledge, understanding and skills but also facilitating discrimination across the whole grade range.
Here is a different question from 2023 which I’ve included to show that sometimes a harder part of a question may come earlier if it better fits the flow of the question. As you can see, this is a 7-part question and most parts of the question show discrimination but similar outcomes. Data shows that question parts 1.3 and 1.7 were much harder, with grade A students represented by the orange bar scoring only about a quarter of the marks and grade A* students represented by the pink bar about half the marks. Across this question as a whole, E-grade candidates represented by the red bar scored several marks and were able to demonstrate knowledge and understanding at A-level standard, but the ramping also facilitated effective discrimination at the top end of the grade range.
Assessment writers build ramping into questions to allow effective discrimination across the full grade range. One key feature of A level Chemistry is the mathematical content. In fact, 20% of the marks in A-level are for demonstrating level 2, that is GCSE (Higher tier level), mathematical skills. When question writers assess these skills, they write questions and mark schemes carefully to allow students to gain credit wherever possible. For example, one mark is allocated for each discrete step in a calculation – this allows question writers to account for any errors from previous steps, so students are not penalized more than once for making an error. This is called ‘error carried forward’ or ECF for short. This approach allows senior associates to assess and credit each step in a multi-step calculation independently. Here is an example.
This question from 2021 is about an equilibrium mixture involving SO2, O2 and SO3.
Students have to deduce how many moles of each gas are present at equilibrium and then use these to calculate the partial pressure of SO2. Now, if they make a mistake deducing the moles of either SO2 or O2 then they would get the final answer wrong. To avoid students being penalized unfairly, the mark scheme works as follows:
- M1, that is Mark 1, – is awarded for a student deducing the amount of SO2.
- M2 is for deducing the amount of O2 (which does not depend on their M1 answer).
- M3 involves finding the total amount of gases at equilibrium – and crucially examiners use student’s answers to M1 and M2 for this, even if they got these wrong, as examiners are assessing a student’s ability to find the mole fraction based on the numbers they have – they should not be penalized again for errors that have already been penalized for earlier in the question.
- M4 – Examiners use the candidate’s answer to M1, divided by their answer to M3 to see if they should be awarded the mark.
So, by using error carried forward, examiners are assessing the four steps independently and a student who makes one mistake remains able to score 3 out of 4 - which is fair. In some questions, the answers to a calculation from an earlier part of the question might be needed in a later part of that question. Question writers do not want to deny a candidate the chance to be assessed on separate skills because they struggled with something earlier in a question.
These next examples show how questions are designed to allow candidates to demonstrate their skills and understanding in these circumstances. In this question candidates are asked to use the numerical answer from the previous question. They are asked to use their answer from 8.3 to answer 8.4. The mark scheme shows that M1 here is based entirely on the application of the candidate’s answer from Q8.3 whether it was right or wrong. However, if candidates could not do Q8.3, we have given them an alternative value (it says – if you were unable to complete the calculation in 8.3, assume a value of Kc = 0.825 mol-2 dm6 – which is not the correct value but is one that candidates without a value from 8.3 can use). This means they can be assessed and gain credit on the chemistry we want to assess in Q8.4 even if they could not do the previous question.
As a slight alternative, this is a similar question where students are using partial pressures to find Kp. They have been asked in a previous part to calculate the partial pressures, but in order to avoid any issue with students getting those wrong or being unable to answer that, writers have given them a different set of values to work with in this part of the question. An extra advantage of this approach is that examiners are all looking at answers based on the same values rather than having to check back and determine the answer using whatever incorrect values candidates might have been found in the previous question.
Multiple choice questions are another key part of the assessment in A-level Chemistry. These allow us to assess parts of the course that are not covered elsewhere meaning that we are assessing as much of the specification as possible – and this avoids construct under-representation. Most areas of skills, knowledge and understanding can be assessed through multiple choice questions. Incorrect answers are called ‘distractors’. In A-level Chemistry effective multiple choice questions have three plausible distractors. They should be answers that the students might have given if a similar written question had been presented. These distractors will stem from common misconceptions or errors.
Here is a recent example of a multiple choice question asking for the main products of the reaction between chlorine and an excess of methane in the presence of ultraviolet radiation. I’d like for you to look at this question and work out the answer (which we call the key). Then, look at each of the distractors and suggest why students might think they are correct. Pause the video to do this, and then click play when you are ready. The correct answer, the key, is D – the products are chloromethane and hydrogen chloride. For distractor B – this is what would be produced if it was the chlorine rather than the methane that was in excess; candidates often get this the wrong way round. For distractor C – here the candidate thinks that H2 is the other product rather than HCl – this is another common mistake that candidates make. Distractor A is a combination of those two common errors.
Another key area within A-level Chemistry is the assessment of practical work. While AQA have the CPAC practical endorsement, question writers also assess knowledge and understanding of practical work on the written papers with at least 15% of the marks coming from this. Questions are set to discriminate between the candidates who understand the practicals they have completed, from those who did not understand them. As an example of this, this question focuses upon the position of the meniscus when using a pipette. The mark scheme for this question shows that examiners were looking for the curvature of the liquid with the bottom of the curve touching the graduation line. Students who learned how to use a pipette accurately would likely have attained this mark. This question is about the use of a burette and what a student should do before this burette can be used; namely, remove the funnel and allow some liquid out to fill the space below the tap. Here is a question asking why we use a conical flask rather than a beaker during a titration. The actual experience of doing a titration and swirling a flask will help students appreciate that it is harder to swirl a beaker than a flask without spilling any liquid. This question is about recrystallisation. It is about why we undertake various actions during a recrystallisation. It asks why we use a minimum volume of hot water; why we cool the flask to room temperature before filtering off the crystals, and why we pour some cold water through the crystals. Many students have undertaken one or more recrystallizations, but not all understand how the technique works – this question is assessing their understanding of the technique.
One final area of A-level Chemistry exams is questions assessed using level of response mark schemes. There is one of these on each paper. Each question is worth 6 marks and is marked in a different way from all the other questions. These questions are assessed by allocating a level – level 1, 2 or 3 (or zero marks if not worth level 1). These questions need very careful design to ensure that they can be reliably marked with all examiners placing answers in the same level.
Each question that requires the application of a level of response mark scheme is designed to have 3 stages – we must be careful here not to confuse the 3 stages within the answer with the 3 levels we use to assess it. We will look at an example in a minute.
The level is based on how much progress has been made with the 3 stages.
- For level 3 – the candidate needs to make strong progress in all 3 stages.
- For level 2 – they need to have made strong progress in two stages, or at least some progress in all 3 stages.
- For level 1 – they either need to have made strong progress in one stage, or at least some progress in 2 of the 3 stages.
Of course, if a student has made little to no progress, then they won’t attain any marks.
Each level itself has a higher mark and a lower mark – for example – level 2 can have the higher mark 4 or lower mark 3. Once the level has been determined, the higher or lower mark in the level is judged based on whether it is a strong or weak answer for that level.
This AS level question is a good example – the question has been carefully designed to have 3 stages. The student has to plan a way to distinguish 4 organic compounds (an aldehyde, an alcohol and two carboxylic acids that are structural isomers). Two of them can be distinguished by chemical tests (the aldehyde and alcohol), and then the remaining two carboxylic acids can be distinguished by using the fingerprint region of their infrared spectra (NMR could also be used but AS candidates have not met that).
- Stage 1 is identifying a chemical test to distinguish the aldehyde.
- Stage 2 is identifying a chemical test to distinguish the alcohol.
- Stages 1 and 2 could be done with a range of reagents and in different orders.
- Stage 3 is using the fingerprint region of the infrared spectra to distinguish the two carboxylic acids from each other.
The question has been carefully designed so all candidates would have to follow this pattern making the judgment of the level clear for examiners to ensure reliable marking. Ultimately, for all questions whether they have a level of response mark scheme or not, we need the question to assess what we want to assess and we need them to be able to be marked reliably across all examiners.
I hope this session has been useful in giving you a deeper insight into how question writers go about designing questions to assess A-level chemistry. Thank you for watching.
