Towards Human-Like Educational Question Generation with Large Language Models

Zichao Wang, Jakob Valdez, Debshila Basu Mallick, Richard G. Baraniuk

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We investigate the utility of large pretrained language models (PLMs) for automatic educational assessment question generation. While PLMs have shown increasing promise in a wide range of natural language applications, including question generation, they can generate unreliable and undesirable content. For high-stakes applications such as educational assessments, it is not only critical to ensure that the generated content is of high quality but also relates to the specific content being assessed. In this paper, we investigate the impact of various PLM prompting strategies on the quality of generated questions. We design a series of generation scenarios to evaluate various generation strategies and evaluate generated questions via automatic metrics and manual examination. With empirical evaluation, we identify the prompting strategy that is most likely to lead to high-quality generated questions. Finally, we demonstrate the promising educational utility of generated questions using our concluded best generation strategy by presenting generated questions together with human-authored questions to a subject matter expert, who despite their expertise, could not effectively distinguish between generated and human-authored questions.

Original languageEnglish (US)
Title of host publicationArtificial Intelligence in Education - 23rd International Conference, AIED 2022, Proceedings
EditorsMaria Mercedes Rodrigo, Noburu Matsuda, Alexandra I. Cristea, Vania Dimitrova
PublisherSpringer Science and Business Media Deutschland GmbH
Pages153-166
Number of pages14
ISBN (Print)9783031116438
DOIs
StatePublished - 2022
Event23rd International Conference on Artificial Intelligence in Education, AIED 2022 - Durham, United Kingdom
Duration: Jul 27 2022Jul 31 2022

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13355 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference23rd International Conference on Artificial Intelligence in Education, AIED 2022
Country/TerritoryUnited Kingdom
CityDurham
Period7/27/227/31/22

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint

Dive into the research topics of 'Towards Human-Like Educational Question Generation with Large Language Models'. Together they form a unique fingerprint.

Cite this