Construct validity in TOEFL iBT speaking tasks

Insights from natural language processing

Kristopher Kyle, Scott A. Crossley, Danielle McNamara

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

This study explores the construct validity of speaking tasks included in the TOEFL iBT (e.g., integrated and independent speaking tasks). Specifically, advanced natural language processing (NLP) tools, MANOVA difference statistics, and discriminant function analyses (DFA) are used to assess the degree to which and in what ways responses to these tasks differ with regard to linguistic characteristics. The findings lend support to using a variety of speaking tasks to assess speaking proficiency. Namely, with regard to linguistic differences, the findings suggest that responses to performance tasks can be accurately grouped based on whether a task is independent or integrated. The findings also suggest that although the independent tasks included in the TOEFL iBT may represent a single construct, responses to integrated tasks vary across task sub-type.

Original languageEnglish (US)
Pages (from-to)319-340
Number of pages22
JournalLanguage Testing
Volume33
Issue number3
DOIs
StatePublished - Jul 1 2016

Fingerprint

construct validity
speaking
language
linguistics
statistics
TOEFL
Natural Language Processing
Construct Validity
performance

Keywords

  • Integrated tasks
  • language use domain
  • natural language processing
  • speaking assessment
  • TOEFL iBT

ASJC Scopus subject areas

  • Linguistics and Language
  • Social Sciences (miscellaneous)
  • Language and Linguistics

Cite this

Construct validity in TOEFL iBT speaking tasks : Insights from natural language processing. / Kyle, Kristopher; Crossley, Scott A.; McNamara, Danielle.

In: Language Testing, Vol. 33, No. 3, 01.07.2016, p. 319-340.

Research output: Contribution to journalArticle

@article{95207019cd7848c69d1c6f31726f817b,
title = "Construct validity in TOEFL iBT speaking tasks: Insights from natural language processing",
abstract = "This study explores the construct validity of speaking tasks included in the TOEFL iBT (e.g., integrated and independent speaking tasks). Specifically, advanced natural language processing (NLP) tools, MANOVA difference statistics, and discriminant function analyses (DFA) are used to assess the degree to which and in what ways responses to these tasks differ with regard to linguistic characteristics. The findings lend support to using a variety of speaking tasks to assess speaking proficiency. Namely, with regard to linguistic differences, the findings suggest that responses to performance tasks can be accurately grouped based on whether a task is independent or integrated. The findings also suggest that although the independent tasks included in the TOEFL iBT may represent a single construct, responses to integrated tasks vary across task sub-type.",
keywords = "Integrated tasks, language use domain, natural language processing, speaking assessment, TOEFL iBT",
author = "Kristopher Kyle and Crossley, {Scott A.} and Danielle McNamara",
year = "2016",
month = "7",
day = "1",
doi = "10.1177/0265532215587391",
language = "English (US)",
volume = "33",
pages = "319--340",
journal = "Language Testing",
issn = "0265-5322",
publisher = "SAGE Publications Ltd",
number = "3",

}

TY - JOUR

T1 - Construct validity in TOEFL iBT speaking tasks

T2 - Insights from natural language processing

AU - Kyle, Kristopher

AU - Crossley, Scott A.

AU - McNamara, Danielle

PY - 2016/7/1

Y1 - 2016/7/1

N2 - This study explores the construct validity of speaking tasks included in the TOEFL iBT (e.g., integrated and independent speaking tasks). Specifically, advanced natural language processing (NLP) tools, MANOVA difference statistics, and discriminant function analyses (DFA) are used to assess the degree to which and in what ways responses to these tasks differ with regard to linguistic characteristics. The findings lend support to using a variety of speaking tasks to assess speaking proficiency. Namely, with regard to linguistic differences, the findings suggest that responses to performance tasks can be accurately grouped based on whether a task is independent or integrated. The findings also suggest that although the independent tasks included in the TOEFL iBT may represent a single construct, responses to integrated tasks vary across task sub-type.

AB - This study explores the construct validity of speaking tasks included in the TOEFL iBT (e.g., integrated and independent speaking tasks). Specifically, advanced natural language processing (NLP) tools, MANOVA difference statistics, and discriminant function analyses (DFA) are used to assess the degree to which and in what ways responses to these tasks differ with regard to linguistic characteristics. The findings lend support to using a variety of speaking tasks to assess speaking proficiency. Namely, with regard to linguistic differences, the findings suggest that responses to performance tasks can be accurately grouped based on whether a task is independent or integrated. The findings also suggest that although the independent tasks included in the TOEFL iBT may represent a single construct, responses to integrated tasks vary across task sub-type.

KW - Integrated tasks

KW - language use domain

KW - natural language processing

KW - speaking assessment

KW - TOEFL iBT

UR - http://www.scopus.com/inward/record.url?scp=84976504604&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84976504604&partnerID=8YFLogxK

U2 - 10.1177/0265532215587391

DO - 10.1177/0265532215587391

M3 - Article

VL - 33

SP - 319

EP - 340

JO - Language Testing

JF - Language Testing

SN - 0265-5322

IS - 3

ER -