Moving towards on-screen assessment – why building familiarity is key

By Victoria Armstrong
Published 01 Mar 2022

Our researchers shed some light on student and teacher experiences of piloting an on-screen GCSE English assessment.

Today’s students are living in unprecedented times, with school closures related to Covid-19 meaning they’ve experienced more online and on-screen education than any previous cohort. This has been a driver for change across the assessment sector. AQA’s CEO Colin Hughes noted recently that teachers “are now wondering when digital assessment will follow: heads are asking us not whether it’s going to happen, but when”. As such, exam boards are moving at pace to explore how to embed digital assessments into both formative and high-stakes testing, and AQA will be trialling large-scale, on-screen pilots in GCSE English, Maths and Science in summer 2022.

Last year, my colleague Katy Finch and I led an initial, smaller study to evaluate student and teacher experiences of taking an on-screen GCSE English assessment pilot. This study took place during the Covid-19 school closures of January–March 2021. Doublestruck, an educational technology company that’s part of the AQA family, delivered the pilot assessment via their online platform Exampro. Although the assessment was originally due to be carried out in the classroom, we eventually administered it remotely across nine schools, involving 255 students and 22 teachers.

Feedback overall was good and those taking the assessment reported few technical difficulties. The students identified many positives, including the ability to edit and restructure their answers on screen and to flip between different source texts. They also liked the increased legibility that resulted from typing their answers (and, perhaps not surprisingly, the teachers also noted this as a positive). There was general enthusiasm for digital assessment, and a sense of embracing change.

“Yeah, I think I can definitely see it becoming a possibility in the future, because it was really easy to use and I think everyone our age knows how to use a computer because we’ve been growing up around computers and technology. So I think it is something that most people can get access to and can do.”

However, a key research finding suggested that students found it challenging to adapt their current paper-based ways of working to completing an assessment on screen.

Participating teachers were generally positive about their marking experience of the digital assessment, although they stated a preference for a balance of on-screen and paper-based marking. Fourteen of the 22 teachers reported feeling positive about more on-screen marking.

Below, we highlight some useful pointers to bear in mind from our findings.

Challenge assumptions about young people’s digital experiences

There is a tendency to believe that children today are experts at everything digital by virtue of exposure to technology throughout their lifetime. The term ‘digital natives’, coined by Marc Prensky in 2001, is often used. However, children’s confidence, competence and creativity in digital spaces depends more on how they use these technologies than their level of exposure to them. We noted that the types of digital activities the students participated in required relatively low levels of skill and were predominantly passive in nature, mainly revolving around YouTube (88%), Instagram (76%) and Snapchat (74%).

Be mindful that not everyone has the same access to technology

The move to remote learning due to the initial school closures in March 2020 revealed significant differences in students’ experiences and use of digital technology. This influenced their transition to online learning. Access to appropriate devices was patchy, with students relying on whatever they had at home. This quote from one student was fairly typical: “At the time the schools weren’t offering any computers or anything, so I just worked on my phone and I found it really difficult.”

In terms of day-to-day use of digital devices, we found that 88% of students in our study used a mobile phone or tablet daily, while only 50% used a laptop every day. Some students reported feeling confident about typing their answers using a keyboard as it would enable them to “get everything down that they wanted”. For others, using a keyboard was an impediment. About 25% of those surveyed felt their typing speed was slower or much slower than writing by hand. These students often cited lower levels of laptop use or no access to a computer at home.

Support students to adopt new approaches to familiar tasks

Based on our findings, we shouldn’t assume that students will seamlessly transition from paper-based to on-screen assessments. For example, some students had difficulties when faced with an unfamiliar, on-screen layout of an exam paper. A significant number reported feeling “fazed” by having to type answers in a blank text box rather than writing in a lined booklet, describing the blank text box as “intimidating”. They were also uncertain about how to plan their essays on screen, leading some students to “write whatever came into my head”.

While most students had few problems navigating between the source text and answer box, a small number (around 4.71%) said they “got lost” when reading the source text so had to spend more time rereading it. Some also reported that they struggled with reading large amounts of text on screen.

The difficulty most frequently mentioned by students was how to go about underlining quotes and annotating the text. In the classroom, this can be done on paper.

“In English I’ve been taught to annotate the text and I couldn’t really do that online, so I had to do it on a piece of paper. So generally, I found that quite annoying.”

“I think it just made it harder to identify the quotes that I wanted to use as evidence, because it meant that I did lose them and that I would have to try and add in additional effort to try and find them again.”

A few students adopted workarounds to replicate their paper-based ways of working. These included opening up a separate Word document, using Post-it notes, printing off a copy of the source texts, or opening the source text in a separate document.

Help students to feel adequately prepared

On the whole, students and teachers in the study were very positive about the pilot on-screen assessment. They agreed that the transition from paper-based to digital exams is both inevitable and necessary, and they were enthusiastic about this. However, as we’ve outlined here, it will be vitally important to take student perspectives and their varied technological experiences into account.

Students frequently stressed the importance of “being prepared”. They would need time to develop and practise new ways of working that support on-screen assessment, as well as having more regular use of laptops and keyboards. It was highlighted that future generations would need to be exposed to digital assessments much earlier in their school careers, such as at the beginning of secondary education.

By embedding digital forms of assessment throughout a child’s learning journey, we could help mitigate some of the challenges experienced by the students who participated in this pilot. Our research team is very keen to explore and support this work and find out more about teachers’ and students’ experiences of on-screen assessment. Further work is planned for later this year, when our researchers will be running focus groups in schools as part of AQA’s summer 2022 GCSE digital assessment pilots.

Keywords

• On-screen tests
• Teachers
• Student voice