Can AI-generated questions meet the standards of high-stakes assessment?

As generative AI capabilities continue to evolve, there is growing interest in how these technologies can enhance aspects of assessment development. The key question is how they can support the process without compromising validity, reliability and fairness in high-stakes assessment.

Published

Friday 6 Nov 2026

AI infographic

Our study compared AI-generated and human-authored GCSE Biology items. The AI-generated items were used without human editing to explore both the potential contribution of AI and the continuing role of human expertise in producing high quality assessments. The research involved psychometric analysis, teacher review and
expert examiner evaluation.

Study at a glance

• 203 GCSE Biology students from four schools
• AI-generated and human-authored short response items (worth 1,2 or 3 marks)
• Marked by three senior assessment examiners
• Psychometric analysis of item performance
• Blind review by four teachers
• Expert evaluation by same three senior assessment examiners
• Comparison of question quality, difficulty and mark schemes

AI can generate credible assessment questions

The first important finding is that AI-generated items often looked authentic and performed comparably to human-authored questions.

Teachers were generally unable to reliably identify which items had been generated by AI and which had been written by humans. Classification accuracy was approximately 49%, effectively no better than chance.

AI-generated questions were also rated similarly to human-authored questions in terms of students’ perceived ability to answer them.

Consistency matters more than average performance

On average, AI-generated items performed similarly to human-authored items in
several psychometric measures.

However, they showed much greater variability. AI produced both some of the strongest and some of the weakest performing items in the study.

Human-authored questions tended to be more consistent, producing a steadier level of quality. AI-generated questions also showed greater variation in difficulty.

The biggest challenge was not the question but the mark scheme

Senior assessment examiners consistently identified weaknesses in AI-generated mark schemes.

These included unclear marking guidance, mismatches between command words and marking criteria, insufficient detail for consistent marking and excessive wordiness and ambiguity.

This is significant as developing the mark scheme is not secondary to item writing. It is central to assessment quality.

Implications

The findings suggest that generative AI is most effective when used as a drafting tool within a controlled assessment process, rather than as an autonomous author.

The study points to three priorities:

  • Use AI to support efficiency, not replace expertise. AI can accelerate early-stage content generation but still requires expert review.
  • Give mark schemes equal attention. Quality assurance processes should focus on marking guidance as well as question wording.
  • Evaluate with evidence. Decisions about AI adoption should be based on psychometric evidence, expert review and ongoing monitoring rather than assumptions about AI capabilities.

AI can generate assessment questions that look and feel authentic. However, high-stakes assessment depends on more than plausible questions.

Human expertise remains essential for calibration, quality assurance and, above all, the development of robust mark schemes.