Evaluation of auto-generated distractors in multiple choice questions from a semantic network

Lishan Zhang, Kurt VanLehn

Research output: Contribution to journalArticle

Abstract

Despite their drawback, multiple-choice questions are an enduring feature in instruction because they can be answered more rapidly than open response questions and they are easily scored. However, it can be difficult to generate good incorrect choices (called “distractors”). We designed an algorithm to generate distractors from a semantic network for four types of multiple choice questions in biology. By recruiting 200 participants from Amazon Mechanical Turk, the machine-generated distractors were compared to human-generated distractors in terms of question difficulty, question discrimination and distractor usefulness. The machine-generated and human-generated distractors performed very closely on all the three measures, suggesting that generating distractors from a semantic network for simple multiple choice questions is a viable method.

Original languageEnglish (US)
JournalInteractive Learning Environments
DOIs
StatePublished - Jan 1 2019

Fingerprint

Semantics
semantics
Turk
evaluation
biology
discrimination
instruction

Keywords

  • crowd sourcing
  • distractor evaluation
  • item analysis
  • multiple choice question
  • Question generation
  • semantic network

ASJC Scopus subject areas

  • Education
  • Computer Science Applications

Cite this

@article{e765b6a41a2641b4b24c741d9d59d2ff,
title = "Evaluation of auto-generated distractors in multiple choice questions from a semantic network",
abstract = "Despite their drawback, multiple-choice questions are an enduring feature in instruction because they can be answered more rapidly than open response questions and they are easily scored. However, it can be difficult to generate good incorrect choices (called “distractors”). We designed an algorithm to generate distractors from a semantic network for four types of multiple choice questions in biology. By recruiting 200 participants from Amazon Mechanical Turk, the machine-generated distractors were compared to human-generated distractors in terms of question difficulty, question discrimination and distractor usefulness. The machine-generated and human-generated distractors performed very closely on all the three measures, suggesting that generating distractors from a semantic network for simple multiple choice questions is a viable method.",
keywords = "crowd sourcing, distractor evaluation, item analysis, multiple choice question, Question generation, semantic network",
author = "Lishan Zhang and Kurt VanLehn",
year = "2019",
month = "1",
day = "1",
doi = "10.1080/10494820.2019.1619586",
language = "English (US)",
journal = "Interactive Learning Environments",
issn = "1049-4820",
publisher = "Routledge",

}

TY - JOUR

T1 - Evaluation of auto-generated distractors in multiple choice questions from a semantic network

AU - Zhang, Lishan

AU - VanLehn, Kurt

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Despite their drawback, multiple-choice questions are an enduring feature in instruction because they can be answered more rapidly than open response questions and they are easily scored. However, it can be difficult to generate good incorrect choices (called “distractors”). We designed an algorithm to generate distractors from a semantic network for four types of multiple choice questions in biology. By recruiting 200 participants from Amazon Mechanical Turk, the machine-generated distractors were compared to human-generated distractors in terms of question difficulty, question discrimination and distractor usefulness. The machine-generated and human-generated distractors performed very closely on all the three measures, suggesting that generating distractors from a semantic network for simple multiple choice questions is a viable method.

AB - Despite their drawback, multiple-choice questions are an enduring feature in instruction because they can be answered more rapidly than open response questions and they are easily scored. However, it can be difficult to generate good incorrect choices (called “distractors”). We designed an algorithm to generate distractors from a semantic network for four types of multiple choice questions in biology. By recruiting 200 participants from Amazon Mechanical Turk, the machine-generated distractors were compared to human-generated distractors in terms of question difficulty, question discrimination and distractor usefulness. The machine-generated and human-generated distractors performed very closely on all the three measures, suggesting that generating distractors from a semantic network for simple multiple choice questions is a viable method.

KW - crowd sourcing

KW - distractor evaluation

KW - item analysis

KW - multiple choice question

KW - Question generation

KW - semantic network

UR - http://www.scopus.com/inward/record.url?scp=85066906007&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85066906007&partnerID=8YFLogxK

U2 - 10.1080/10494820.2019.1619586

DO - 10.1080/10494820.2019.1619586

M3 - Article

AN - SCOPUS:85066906007

JO - Interactive Learning Environments

JF - Interactive Learning Environments

SN - 1049-4820

ER -