Automated Scoring for Reading Comprehension via In-context BERT Tuning

Nigel Fernandez, Aritra Ghosh, Naiming Liu, Zichao Wang, Benoît Choffin, Richard Baraniuk, Andrew Lan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Automated scoring of open-ended student responses has the potential to significantly reduce human grader effort. Recent advances in automated scoring leverage textual representations from pre-trained language models like BERT. Existing approaches train a separate model for each item/question, suitable for scenarios like essay scoring where items can be different from one another. However, these approaches have two limitations: 1) they fail to leverage item linkage for scenarios such as reading comprehension where multiple items may share a reading passage; 2) they are not scalable since storing one model per item is difficult with large language models. We report our (grand prize-winning) solution to the National Assessment of Education Progress (NAEP) automated scoring challenge for reading comprehension. Our approach, in-context BERT fine-tuning, produces a single shared scoring model for all items with a carefully designed input structure to provide contextual information on each item. Our experiments demonstrate the effectiveness of our approach which outperforms existing methods. We also perform a qualitative analysis and discuss the limitations of our approach. (Full version of the paper can be found at: https://arxiv.org/abs/2205.09864 Our implementation can be found at: https://github.com/ni9elf/automated-scoring )

Original languageEnglish (US)
Title of host publicationArtificial Intelligence in Education - 23rd International Conference, AIED 2022, Proceedings
EditorsMaria Mercedes Rodrigo, Noburu Matsuda, Alexandra I. Cristea, Vania Dimitrova
PublisherSpringer Science and Business Media Deutschland GmbH
Pages691-697
Number of pages7
ISBN (Print)9783031116438
DOIs
StatePublished - 2022
Event23rd International Conference on Artificial Intelligence in Education, AIED 2022 - Durham, United Kingdom
Duration: Jul 27 2022Jul 31 2022

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume13355 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference23rd International Conference on Artificial Intelligence in Education, AIED 2022
Country/TerritoryUnited Kingdom
CityDurham
Period7/27/227/31/22

Keywords

  • Automated scoring
  • BERT
  • Reading comprehension

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint

Dive into the research topics of 'Automated Scoring for Reading Comprehension via In-context BERT Tuning'. Together they form a unique fingerprint.

Cite this