The use of latent semantic analysis as a tool for the quantitative assessment of understanding and knowledge

Amy M. Shapiro, Danielle McNamara

Research output: Contribution to journalArticle

20 Citations (Scopus)

Abstract

Latent Semantic Analysis (LSA) is a statistical model of word usage that has been used for a variety of applications. One of these applications is the quantitative assessment of the semantic content within written text. While the technology has been successful in correlating with the qualitative ratings of human experts, it is unclear what aspect of knowledge is being reflected in an LSA output. The two experiments presented here were designed to address this general question. We were particularly interested in whether an LSA analysis more accurately reflects the factual or conceptual knowledge contained in written material. Experiment 1 explored this issue by comparing LSA analyses of essays to human-generated scores. It also compared the LSA output to several measures of conceptual structure. Experiment 2 correlated LSA analyses of transcribed recall protocols with a series of comprehension measures that were designed to vary in the degree to which they reflect conceptual or factual knowledge. We found compelling evidence that LSA analyses are a stronger reflection of the text-based knowledge represented by essays and recall protocols than conceptual knowledge. Both studies also explored a methodological issue pertaining to the use of LSA. Specifically, does LSA have to be "trained" in the particular content area of the text to be analyzed? This question was addressed by running multiple LSA analyses, each performed with differing "semantic spaces" created through training in domain specific or general content areas. We found that LSA performed best when trained in a content area specific to the material to be analyzed. These results are discussed with respect to the application of LSA analyses in the classroom and laboratory.

Original languageEnglish (US)
Pages (from-to)1-36
Number of pages36
JournalJournal of Educational Computing Research
Volume22
Issue number1
StatePublished - 2000
Externally publishedYes

Fingerprint

Semantics
semantics
experiment
Experiments
comprehension
rating
expert
classroom

ASJC Scopus subject areas

  • Education

Cite this

@article{93146626ba8b4a2f9ede95ceb229e32e,
title = "The use of latent semantic analysis as a tool for the quantitative assessment of understanding and knowledge",
abstract = "Latent Semantic Analysis (LSA) is a statistical model of word usage that has been used for a variety of applications. One of these applications is the quantitative assessment of the semantic content within written text. While the technology has been successful in correlating with the qualitative ratings of human experts, it is unclear what aspect of knowledge is being reflected in an LSA output. The two experiments presented here were designed to address this general question. We were particularly interested in whether an LSA analysis more accurately reflects the factual or conceptual knowledge contained in written material. Experiment 1 explored this issue by comparing LSA analyses of essays to human-generated scores. It also compared the LSA output to several measures of conceptual structure. Experiment 2 correlated LSA analyses of transcribed recall protocols with a series of comprehension measures that were designed to vary in the degree to which they reflect conceptual or factual knowledge. We found compelling evidence that LSA analyses are a stronger reflection of the text-based knowledge represented by essays and recall protocols than conceptual knowledge. Both studies also explored a methodological issue pertaining to the use of LSA. Specifically, does LSA have to be {"}trained{"} in the particular content area of the text to be analyzed? This question was addressed by running multiple LSA analyses, each performed with differing {"}semantic spaces{"} created through training in domain specific or general content areas. We found that LSA performed best when trained in a content area specific to the material to be analyzed. These results are discussed with respect to the application of LSA analyses in the classroom and laboratory.",
author = "Shapiro, {Amy M.} and Danielle McNamara",
year = "2000",
language = "English (US)",
volume = "22",
pages = "1--36",
journal = "Journal of Educational Computing Research",
issn = "0735-6331",
publisher = "Baywood Publishing Co. Inc.",
number = "1",

}

TY - JOUR

T1 - The use of latent semantic analysis as a tool for the quantitative assessment of understanding and knowledge

AU - Shapiro, Amy M.

AU - McNamara, Danielle

PY - 2000

Y1 - 2000

N2 - Latent Semantic Analysis (LSA) is a statistical model of word usage that has been used for a variety of applications. One of these applications is the quantitative assessment of the semantic content within written text. While the technology has been successful in correlating with the qualitative ratings of human experts, it is unclear what aspect of knowledge is being reflected in an LSA output. The two experiments presented here were designed to address this general question. We were particularly interested in whether an LSA analysis more accurately reflects the factual or conceptual knowledge contained in written material. Experiment 1 explored this issue by comparing LSA analyses of essays to human-generated scores. It also compared the LSA output to several measures of conceptual structure. Experiment 2 correlated LSA analyses of transcribed recall protocols with a series of comprehension measures that were designed to vary in the degree to which they reflect conceptual or factual knowledge. We found compelling evidence that LSA analyses are a stronger reflection of the text-based knowledge represented by essays and recall protocols than conceptual knowledge. Both studies also explored a methodological issue pertaining to the use of LSA. Specifically, does LSA have to be "trained" in the particular content area of the text to be analyzed? This question was addressed by running multiple LSA analyses, each performed with differing "semantic spaces" created through training in domain specific or general content areas. We found that LSA performed best when trained in a content area specific to the material to be analyzed. These results are discussed with respect to the application of LSA analyses in the classroom and laboratory.

AB - Latent Semantic Analysis (LSA) is a statistical model of word usage that has been used for a variety of applications. One of these applications is the quantitative assessment of the semantic content within written text. While the technology has been successful in correlating with the qualitative ratings of human experts, it is unclear what aspect of knowledge is being reflected in an LSA output. The two experiments presented here were designed to address this general question. We were particularly interested in whether an LSA analysis more accurately reflects the factual or conceptual knowledge contained in written material. Experiment 1 explored this issue by comparing LSA analyses of essays to human-generated scores. It also compared the LSA output to several measures of conceptual structure. Experiment 2 correlated LSA analyses of transcribed recall protocols with a series of comprehension measures that were designed to vary in the degree to which they reflect conceptual or factual knowledge. We found compelling evidence that LSA analyses are a stronger reflection of the text-based knowledge represented by essays and recall protocols than conceptual knowledge. Both studies also explored a methodological issue pertaining to the use of LSA. Specifically, does LSA have to be "trained" in the particular content area of the text to be analyzed? This question was addressed by running multiple LSA analyses, each performed with differing "semantic spaces" created through training in domain specific or general content areas. We found that LSA performed best when trained in a content area specific to the material to be analyzed. These results are discussed with respect to the application of LSA analyses in the classroom and laboratory.

UR - http://www.scopus.com/inward/record.url?scp=0034378819&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0034378819&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:0034378819

VL - 22

SP - 1

EP - 36

JO - Journal of Educational Computing Research

JF - Journal of Educational Computing Research

SN - 0735-6331

IS - 1

ER -