Objective Intelligibility Assessment by Automated Segmental and Suprasegmental Listening Error Analysis

Research output: Contribution to journalArticle

Abstract

Purpose Subjective speech intelligibility assessment is often preferred over more objective approaches that rely on transcript scoring. This is, in part, because of the intensive manual labor associated with extracting objective metrics from transcribed speech. In this study, we propose an automated approach for scoring transcripts that provides a holistic and objective representation of intelligibility degradation stemming from both segmental and suprasegmental contributions, and that corresponds with human perception. Method Phrases produced by 73 speakers with dysarthria were orthographically transcribed by 819 listeners via Mechanical Turk, resulting in 63,840 phrase transcriptions. A protocol was developed to filter the transcripts, which were then automatically analyzed using novel algorithms developed for measuring phoneme and lexical segmentation errors. The results were compared with manual labels on a randomly selected sample set of 40 transcribed phrases to assess validity. A linear regression analysis was conducted to examine how well the automated metrics predict a perceptual rating of severity and word accuracy. Results On the sample set, the automated metrics achieved 0.90 correlation coefficients with manual labels on measuring phoneme errors, and 100% accuracy on identifying and coding lexical segmentation errors. Linear regression models found that the estimated metrics could predict a significant portion of the variance in perceptual severity and word accuracy. Conclusions The results show the promising development of an objective speech intelligibility assessment that identifies intelligibility degradation on multiple levels of analysis.

Original languageEnglish (US)
Pages (from-to)3359-3366
Number of pages8
JournalJournal of speech, language, and hearing research : JSLHR
Volume62
Issue number9
DOIs
StatePublished - Sep 20 2019
Externally publishedYes

Fingerprint

Speech Intelligibility
Linear Models
manual labor
Dysarthria
Turk
listener
coding
regression analysis
rating
Regression Analysis
regression
Error Analysis
Intelligibility
Suprasegmentals
segmentation
Linear Regression
Phoneme
Degradation
Scoring
Lexical Segmentation

ASJC Scopus subject areas

  • Language and Linguistics
  • Linguistics and Language
  • Speech and Hearing

Cite this

@article{f4f3192c2ab44faab71b4e17542937c2,
title = "Objective Intelligibility Assessment by Automated Segmental and Suprasegmental Listening Error Analysis",
abstract = "Purpose Subjective speech intelligibility assessment is often preferred over more objective approaches that rely on transcript scoring. This is, in part, because of the intensive manual labor associated with extracting objective metrics from transcribed speech. In this study, we propose an automated approach for scoring transcripts that provides a holistic and objective representation of intelligibility degradation stemming from both segmental and suprasegmental contributions, and that corresponds with human perception. Method Phrases produced by 73 speakers with dysarthria were orthographically transcribed by 819 listeners via Mechanical Turk, resulting in 63,840 phrase transcriptions. A protocol was developed to filter the transcripts, which were then automatically analyzed using novel algorithms developed for measuring phoneme and lexical segmentation errors. The results were compared with manual labels on a randomly selected sample set of 40 transcribed phrases to assess validity. A linear regression analysis was conducted to examine how well the automated metrics predict a perceptual rating of severity and word accuracy. Results On the sample set, the automated metrics achieved 0.90 correlation coefficients with manual labels on measuring phoneme errors, and 100{\%} accuracy on identifying and coding lexical segmentation errors. Linear regression models found that the estimated metrics could predict a significant portion of the variance in perceptual severity and word accuracy. Conclusions The results show the promising development of an objective speech intelligibility assessment that identifies intelligibility degradation on multiple levels of analysis.",
author = "Yishan Jiao and Amy LaCross and Visar Berisha and Julie Liss",
year = "2019",
month = "9",
day = "20",
doi = "10.1044/2019_JSLHR-S-19-0119",
language = "English (US)",
volume = "62",
pages = "3359--3366",
journal = "Journal of Speech, Language, and Hearing Research",
issn = "1092-4388",
publisher = "American Speech-Language-Hearing Association (ASHA)",
number = "9",

}

TY - JOUR

T1 - Objective Intelligibility Assessment by Automated Segmental and Suprasegmental Listening Error Analysis

AU - Jiao, Yishan

AU - LaCross, Amy

AU - Berisha, Visar

AU - Liss, Julie

PY - 2019/9/20

Y1 - 2019/9/20

N2 - Purpose Subjective speech intelligibility assessment is often preferred over more objective approaches that rely on transcript scoring. This is, in part, because of the intensive manual labor associated with extracting objective metrics from transcribed speech. In this study, we propose an automated approach for scoring transcripts that provides a holistic and objective representation of intelligibility degradation stemming from both segmental and suprasegmental contributions, and that corresponds with human perception. Method Phrases produced by 73 speakers with dysarthria were orthographically transcribed by 819 listeners via Mechanical Turk, resulting in 63,840 phrase transcriptions. A protocol was developed to filter the transcripts, which were then automatically analyzed using novel algorithms developed for measuring phoneme and lexical segmentation errors. The results were compared with manual labels on a randomly selected sample set of 40 transcribed phrases to assess validity. A linear regression analysis was conducted to examine how well the automated metrics predict a perceptual rating of severity and word accuracy. Results On the sample set, the automated metrics achieved 0.90 correlation coefficients with manual labels on measuring phoneme errors, and 100% accuracy on identifying and coding lexical segmentation errors. Linear regression models found that the estimated metrics could predict a significant portion of the variance in perceptual severity and word accuracy. Conclusions The results show the promising development of an objective speech intelligibility assessment that identifies intelligibility degradation on multiple levels of analysis.

AB - Purpose Subjective speech intelligibility assessment is often preferred over more objective approaches that rely on transcript scoring. This is, in part, because of the intensive manual labor associated with extracting objective metrics from transcribed speech. In this study, we propose an automated approach for scoring transcripts that provides a holistic and objective representation of intelligibility degradation stemming from both segmental and suprasegmental contributions, and that corresponds with human perception. Method Phrases produced by 73 speakers with dysarthria were orthographically transcribed by 819 listeners via Mechanical Turk, resulting in 63,840 phrase transcriptions. A protocol was developed to filter the transcripts, which were then automatically analyzed using novel algorithms developed for measuring phoneme and lexical segmentation errors. The results were compared with manual labels on a randomly selected sample set of 40 transcribed phrases to assess validity. A linear regression analysis was conducted to examine how well the automated metrics predict a perceptual rating of severity and word accuracy. Results On the sample set, the automated metrics achieved 0.90 correlation coefficients with manual labels on measuring phoneme errors, and 100% accuracy on identifying and coding lexical segmentation errors. Linear regression models found that the estimated metrics could predict a significant portion of the variance in perceptual severity and word accuracy. Conclusions The results show the promising development of an objective speech intelligibility assessment that identifies intelligibility degradation on multiple levels of analysis.

UR - http://www.scopus.com/inward/record.url?scp=85072546655&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85072546655&partnerID=8YFLogxK

U2 - 10.1044/2019_JSLHR-S-19-0119

DO - 10.1044/2019_JSLHR-S-19-0119

M3 - Article

C2 - 31525112

AN - SCOPUS:85072546655

VL - 62

SP - 3359

EP - 3366

JO - Journal of Speech, Language, and Hearing Research

JF - Journal of Speech, Language, and Hearing Research

SN - 1092-4388

IS - 9

ER -