Physics

This video looks at how the general assessment principles are applied in A-level Physics. Starting with a detailed look at the assessment objectives, the video considers the assessment framework for Science, and the different types of questions. It looks at the language of questions and how we can vary the level of demand of a question to suit its purpose. The video also provides insights into how we develop our assessments.

Transcript

Hello. My name is David Homer, and I am the Chair of Examiners for AQA A-level Physics. This presentation is about the principles of assessment. It explains how these principles support the way questions are written for the AQA AS and A-level Physics specifications: 7407 and 7408. I want to begin by reminding you what is meant by good assessment and then we’ll consider the Assessment Objectives. I’ll explain how the AQA team of A-level Physics writers attempt to achieve these principles and the objectives in question papers.

I hope you have seen the videos on “What makes good assessment” produced by AQA. These cover the general principles of assessment. One of the slides from these presentations reminds us about the importance of several factors in assessment such as reliability, differentiation, reduction of bias and so on.  But, there is one further construct that is relevant to assessment and that is the need for assessments to be authentic. Authenticity means that we need to know the extent to which any test reflects real life: are we assessing the skills students have learned in a way that matches the way they have been taught, and are the students applying their skills to new and relevant situations?  The UK driving test is a good example of authentic assessment in action: we expect new drivers to demonstrate their competence by undertaking a supervised and assessed drive along real roads. The driver must deal convincingly with real situations that arise during the test. A test taken using a driving simulator would not be so authentic because the reality of the drive and the consequences of error are significantly reduced. The Driving Theory test is also an authentic way to test Highway code knowledge and the various road signs and rules of the road. These might not be tested in sufficient detail on a 30-minute test drive. But, of course, we are not assessing driving skills!

To make the assessment of physics practical skills authentic, we would choose in an ideal world to assess them using a practical examination undertaken in the student’s school laboratory. However, the present rules for the conduct of UK examinations do not allow this and so at AQA we assess practical skills and data analysis using two theoretical papers.  For full A-level, this is component 7408/3A, which is wholly practically based. For AS, the practical test is in one section of 7407/2. There is a debate to be had as to whether such paper exercises can ever be as authentic as a laboratory experience. That’s why it’s so important for students to have encountered the Required Practical Activities in the specification. This allows them to translate their practical experiences into something that they can describe, explain, and use in the context of a written paper.

The AQA suite of A-level papers contains questions that are usually focussed on a context. This context may come from within the school laboratory, or it may be a situation from outside school, but it is one that all students can understand, and to which they can all relate. You will see some of the contexts we have used during the course of this presentation. This focus on context within a physics assessment can also allow a high degree of authenticity. Paper writers and examiners are governed by these the principles of validity, reliability, and authenticity as I hope you will see in the rest of this presentation.

It’s perhaps worth spending a few minutes talking about Assessment Objectives – a phrase which lead assessment writers tend to abbreviate to AOs. As I said earlier, these Assessment Objectives have a major role in ensuring the validity of the assessments year on year. The AOs divide the assessment into three cognitive categories: AO1: This is the demonstration of knowledge and understanding of scientific ideas, processes, techniques, and procedures. Incidentally, you can find all these definitions and details about them in the specification itself. AO2: the application of knowledge and understanding of scientific ideas, processes, techniques, and procedures: in theoretical and practical contexts, and when handling qualitative and quantitative data. AO3: the analysis, interpretation and evaluation of scientific information, ideas, and evidence, including in relation to issues, to make judgements and reach conclusions, and to develop and refine practical design and procedures. You will recognise the influence of learning taxonomies here. I am going to use a taxonomy that focusses on the structure of observed learning outcomes – abbreviated as the SOLO taxonomy.  I could however have just as easily used the Bloom taxonomy or others.

AO1 tasks focus on the unistructural level in which the student demonstrates largely limited and isolated disciplinary knowledge. For example, at the unistructural level a student will state that water boils at 100 °C. AO2 tasks are multi-structural or relational so that a student succeeding at AO2 tasks will have a knowledge of multiple facts within the learning domain and can understand the relationships at work. They can now explain that water boils at 100 °C and that this is because there is increased kinetic energy of the water molecules that make up the liquid. AO3 tasks belong to the extended-abstract level of the taxonomy where high-attaining students can take the facts they have understood and assimilate them. They can apply their coherent knowledge to new scenarios. They might for example predict the behaviour of other boiling fluids based on their understanding of water. It is essential for the assessment writers to understand this underpinning ramp of skills in the construction of a paper. Demonstration of AO3 skills is required at AS- and A-level by Ofqual regulation but, in a more pragmatic way, they are necessary to empower an A-level-qualified physicist to be able to operate beyond the school laboratory, in tertiary education or elsewhere.

The assessment principles remind us that questions must contain accurate physics. This question would not achieve this aim because the resistance of the filament lamp is decreasing with increasing electrical current. I should add that I have invented this example; this is not a stem from a real AQA question. The question I have based it on actually deals with the resistance of the resistor and the effect of the two devices in parallel. Even though the error in the graph would not prevent any student answering these questions, the ambiguity about the behaviour of the filament lamp would put many students off; they might waste time worrying about the issue. It would constitute a form of construct irrelevance.

Command words must be standard and clear. The command word for this question is “Calculate.” It will generally be the first word in the final (as here) or penultimate sentence of the question. The approved list of commands that we use is given on the AQA website. However, from time to time we do use words outside this list where they are appropriate. An example of this is the use of the word “estimate” which does not appear on the list. There is no other word that can quite express so well what a student must do when making an educated guess or making a calculation about a quantity that cannot be exactly determined.

Language must be accessible to the student. The rule of thumb often used in writing examination prose is that the reading age should be about two years below that of the average student sitting the examination. So, we aim for a reading age of sixteen years. Sentences are short and rarely contain more than one or two basic ideas. In this example, the stem could have been in one sentence. It could have read “Figure 3 shows the wheelchair and its user travelling up a hill that makes an angle of 4.5 degrees to the horizontal”. However, this would involve the cognitive requirement to interpret the meaning of “that” in the sentence. Two sentences are kinder. Language must be unbiased and sensitive. I shall give you an example of this in action in a moment.

Usually, we only provide material sufficient to answer the question. And, particularly in the AS papers for 7407, data is provided where they are required. However, analytic and evaluative assessment is, of necessity, demanding. So, putting data where it is needed is not always desirable or possible. High-achieving young physicists need the skills to identify and collect data from several parts of an extended question. The students need to be challenged by questions where selection of data is important and where judgement about the data needs to be made.

So, what governs the construction of the questions? A set of documents is used – this includes, as a major component, the specification itself. The specification sets out the topics which can be covered and suggests A-level of treatment for both the teacher and the examiner. It also describes the Assessment Objectives and the overall balance that these must have both within an individual paper and the assessment as a whole. It also sets out the mathematical requirements and their demand. The amount of mathematics that has to appear in the suite of papers is dictated by a blueprint that was agreed with Ofqual when the specification was constructed. The specification also goes into significant detail about the practical requirements that schools must address throughout the two-year course whether at AS or full A-level. Some of these requirements are dictated by Ofqual to all awarding organisations who operate an A-level physics qualification. Others were agreed within the individual awarding organisations when the specifications and their accompanying documentation were written. These documents include:

the specimen papers which are constructed at the beginning of the life of a specification and

the list of command terms that writers are expected to use. There is also a practical handbook that gives information about our expectations for practical work in the assessments. All these documents can be downloaded from the AQA website.

So, an overarching set of objectives for examination writers is that questions should be valid, reliable, and authentic - they should have an appropriate level of varying demand within the assessment objective framework, and they require a ramp of difficulty to allow students to demonstrate the full range of their individual abilities. I shall illustrate how this works in practice using a June 2019 question from paper 1 of the 7408 suite. The question begins with a stem that contains the context and a small amount of data. These data are here because they apply to most calculation parts of the question. You can pause the webinar to read the question parts before I discuss them.

The question concerns safety barriers but is in the context of the use of test dummies. This was deliberate. A context involving the suggestion of a real accident could cause distress to a student who might have been involved in some way with a car accident. We always try to avoid the possibility of such scenarios. This goes back to the need for unbiased and sensitive language in questions. Notice also that the main direction of the entire question – which you will see in a moment – is set up in the first two sentences of the stem.

5.1 asks for a simple calculation of kinetic energy. The only complication is that the speed is in kilometres per hour rather than metres per second. This is essentially AO1 material with a calculation that could equally be set for GCSE Combined Science. The writer now increases the demand of the question by introducing three sets of skills: vector components, understanding of momentum change, and a test of units. The level of complexity here lifts this simple calculation to AO2, here in a theoretical context. Theoretical because this is not a case that a student is likely to see or to investigate in a school laboratory. Phase three of the question begins with another relatively simple question. The student, however, needs to identify quantities that they have calculated earlier to attempt the task. Again, this makes this AO2 material.

Question 5.3 is a “show that” question. We provide a rounded ‘show that’ value for students who cannot access 5.3 or earlier parts. The expectation for a ‘show that’ deduce question is that students must demonstrate fluently how they arrive at their answer and that they have calculated it to a degree of precision greater than the ‘show that’ value. Students are now given data that will determine the energy stored in the barrier as it deforms. The 80 kJ value from 5.3 is used together with these data to predict whether the barrier will pass the safety test. There are now elements of AO3 in 5.4 as the question demands some judgement on the part of the student as to the best approach to take.

If you look at the mark scheme for this question you will see that there are no less than four separate marking routes to the answer. This is a characteristic of questions that test high-order cognitive skills. Such questions also demand flexibility of the examiners who will eventually mark the live scripts.

Finally, another AO3 question. A comparison between a steel barrier and a concrete one is made. Here the only additional information provided is “that the concrete wall does not deform.” This question is exploring the evaluative skills of the student. As you can see, a question of this type has a clear ramp of difficulty that leads the student through an increasing demand from low cognitive levels to sophisticated evaluative and judgemental arguments.

The process of writing the questions for a series begins with a planning meeting.  One of the elements of this is to review the performance of the question papers from the previous summer series. Statistical data is circulated to inform the team about the performance of the components. For example, what is the extent to which the questions discriminated between students at the A and E boundaries? Or, did a particular multi-mark question itself provide a good distribution? We want a 3-mark question to lead to the award of all four possible marks, zero through 3. It is important to avoid question – mark-scheme combinations where few students are awarded 2 out of 3 marks but many receive 1 or 3.

The meeting then turns to decisions for the new assessment that is about to be written. Much of the balance of the components is pre-determined by the blueprint for the specification. This blueprint was produced at the same time as the specification and the specimen papers before the series was first examined in 2016. These documents, taken together, give very detailed instructions and guidance about the composition of every component. They list, for example:

• the number of marks that can be assigned to Assessment Objectives 1 2 and 3,
• the minimum numbers of marks that must be assigned to level 2 mathematical skills,
• the mathematical skills that must be tested,
• the maximum number of straight recall marks that can be set.

The paper writing process itself goes through a number of stages. Each of these ends with a review by a large team of writers and practising physics teachers who all work with the component.  Drafts are checked for overlapping material between components.  There must be adherence to the blueprint and the paper rubric, questions must again be cross-referenced with the specification, the physics must be checked and double-checked, the likely timings of the paper worked out and more.

The mark scheme is constructed at the same time as the Question Paper – not once the paper has been sat by candidates. Indeed, some examiners argue that the best assessments begin with the mark scheme and the question paper is written to match the scheme itself. While this is rarely true in practice, because the context usually comes first, we often find ourselves at an early stage in a paper’s life asking questions like ‘does the question truly reflect the requirements of the mark scheme’ or ‘could the question be phrased better to trigger the student response that we require’?

Decisions about the readability of a paper go beyond matters of text and language. We also spend time discussing the optimum layout of the pages to give as much help to students as possible, minimising turn backs and so on.  Finally, you will know that we put multiple-choice questions as the final section of our papers. This was a request from teachers and students that arose in focus groups at an early stage of the specification development. Students feel that putting the multiple-choice section first means that they spend too long on it. We were happy to make the change from what was our existing practice.

The comment about the multiple-choice section allows me to turn briefly to the construction of the multiple-choice questions themselves. Students answer this style of question in 7407/2, 7408/1 and 7408/2. Multiple-choice questions (MCQ) come under the general category of objective-test questions (OTQ).  Broadly speaking, objective-test questions are those where the responses provided are either correct or incorrect. The advantages of OTQ of all types are:

• that they are less reliant on a student’s writing skills.
• that they can sample a wide range of the specification in a relatively short time and so they can assess the basic learning of the student.

The drawback, for A-level, is that multiple-choice items tend to focus on lower-order skills. This is because the chain of argument involved in an effective AO3 test is worth more than the single-mark tariff available for a multiple-choice question. However, both AO1 and AO2 tests are relatively easy to construct within the MCQ framework.

Multiple-choice questions are constructed at the same time as the written questions throughout the writing process to ensure appropriate specification coverage. The process for their construction revision, scrutiny and review is identical to that of the written parts of the components.  Additionally, each component that does not have a formal MCQ section has at least one multiple-choice style question. However, these can have slight differences from the questions on the formal multiple-choice sections and I will go into this later.

Items in multiple choice testing have a stem, followed by a question, a series of possible answers – the responses – which must be unambiguously correct or incorrect. There is only one correct response, known as the key. The incorrect responses are the distractors. In AQA physics there are three distractors and one key.

For a multiple-choice question to be reliable and valid the language must be familiar throughout; students are unlikely to consider distractors with unfamiliar terms or ideas. There must be no verbal association between stem and key, students are more likely to select the key on this basis.  So, as far as possible, a student should be able to select the key on the basis of the stem alone – although this is not always possible particularly in verbal rather than numerical items, therefore all the relevant information must be contained in the stem. The physics team tries to avoid negative wording, e.g. “what is not correct about nuclear fission?” as this can be misread by students in the heat of the moment but sometimes it is the best way to ask a particular question. In such cases we always embolden the word ‘not’ and we limit ourselves to a small number of such questions per component.

Just a final word about the single objective question found in the Option sections and Paper 3A. These by and large obey similar rules to the items in the formal multiple-choice section. However, a significant difference is that these single multiple-choice questions can have variations in the number of distractors – these have varied in the past between 2 and 4 distractors – meaning a total of 3 to 5 responses for students to consider. This allows assessment writers more variation in the construction of the item.

Just to show how times change, here is a multiple-choice question from a JMB examination set in 1980. Four distractors were used then. This is a complex calculation for students to undertake with only one mark available. It would be difficult to justify this question today. How would you modify this question to make it more accessible? Pause the video to give yourself time to consider the task in detail.

I suspect that today we might ask students to consider the temperature rise of a fixed mass of water after absorption of all the radiation. The use of an intermediate concave mirror is no longer on the specification so that would have to go too. And, rather than have a continuous flow process, we would probably ask for a calculation over a fixed time period, of perhaps one minute or, more than likely, one hundred seconds. However, what is common to today’s questions and also those of over forty years ago is the care and effort that goes into making all our assessments as fair, valid and reliable as they possibly can be.

I hope you have found all information in this presentation both helpful and enlightening. Thank you for watching.

Questions you may want to think about

• How can you use these insights to prepare your learners for exams?
• Do your internal assessments reflect the approach of the exam? To what extent do you want them to?
• What’s the most important or surprising thing that you’ve learned? How might it influence your teaching?