Text, topics, and turkers: A consensus measure for statistical topics

Fred Morstatter; Jürgen Pfeffer; Katja Mayer; Huan Liu

doi:10.1145/2700171.2791028

Text, topics, and turkers: A consensus measure for statistical topics

Fred Morstatter, Jürgen Pfeffer, Katja Mayer, Huan Liu

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

4 Scopus citations

Abstract

Topic modeling is an important tool in social media analysis, allowing researchers to quickly understand large text corpora by investigating the topics underlying them. One of the fundamental problems of topic models lies in how to assess the quality of the topics from the perspective of human interpretability. How well can humans understand the meaning of topics generated by statistical topic modeling algorithms? In this work we advance the study of this question by introducing Topic Consensus: a new measure that calculates the quality of a topic through investigating its consensus with some known topics underlying the data. We view the quality of the topics from three perspectives: 1) topic interpretability, 2) how documents relate to the underlying topics, and 3) how interpretable the topics are when the corpus has an underlying categorization. We provide insights into how well the results of Mechanical Turk match automated methods for calculating topic quality. The probability distribution of the words in the topic best fit the Topic Coherence measure, in terms of both correlation as well as finding the best topics.

Original language	English (US)
Title of host publication	HT 2015 - Proceedings of the 26th ACM Conference on Hypertext and Social Media
Publisher	Association for Computing Machinery, Inc
Pages	123-131
Number of pages	9
ISBN (Electronic)	9781450333955
DOIs	https://doi.org/10.1145/2700171.2791028
State	Published - Aug 24 2015
Event	26th ACM Conference on Hypertext and Social Media, HT 2015 - Guzelyurt, Cyprus Duration: Sep 1 2015 → Sep 4 2015

Publication series

Name	HT 2015 - Proceedings of the 26th ACM Conference on Hypertext and Social Media

Conference

Conference	26th ACM Conference on Hypertext and Social Media, HT 2015
Country/Territory	Cyprus
City	Guzelyurt
Period	9/1/15 → 9/4/15

Keywords

Text analysis
Text mining
Topic analysis
Topic modeling

ASJC Scopus subject areas

Artificial Intelligence
Software
Computer Graphics and Computer-Aided Design
Human-Computer Interaction

Access to Document

10.1145/2700171.2791028

Cite this

Morstatter, F., Pfeffer, J., Mayer, K., & Liu, H. (2015). Text, topics, and turkers: A consensus measure for statistical topics. In HT 2015 - Proceedings of the 26th ACM Conference on Hypertext and Social Media (pp. 123-131). (HT 2015 - Proceedings of the 26th ACM Conference on Hypertext and Social Media). Association for Computing Machinery, Inc. https://doi.org/10.1145/2700171.2791028

Text, topics, and turkers: A consensus measure for statistical topics. / Morstatter, Fred; Pfeffer, Jürgen; Mayer, Katja et al.
HT 2015 - Proceedings of the 26th ACM Conference on Hypertext and Social Media. Association for Computing Machinery, Inc, 2015. p. 123-131 (HT 2015 - Proceedings of the 26th ACM Conference on Hypertext and Social Media).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Morstatter, F, Pfeffer, J, Mayer, K & Liu, H 2015, Text, topics, and turkers: A consensus measure for statistical topics. in HT 2015 - Proceedings of the 26th ACM Conference on Hypertext and Social Media. HT 2015 - Proceedings of the 26th ACM Conference on Hypertext and Social Media, Association for Computing Machinery, Inc, pp. 123-131, 26th ACM Conference on Hypertext and Social Media, HT 2015, Guzelyurt, Cyprus, 9/1/15. https://doi.org/10.1145/2700171.2791028

@inproceedings{e3da00fdf77f4ed2aa21bb1d4ecaf57a,

title = "Text, topics, and turkers: A consensus measure for statistical topics",

abstract = "Topic modeling is an important tool in social media analysis, allowing researchers to quickly understand large text corpora by investigating the topics underlying them. One of the fundamental problems of topic models lies in how to assess the quality of the topics from the perspective of human interpretability. How well can humans understand the meaning of topics generated by statistical topic modeling algorithms? In this work we advance the study of this question by introducing Topic Consensus: a new measure that calculates the quality of a topic through investigating its consensus with some known topics underlying the data. We view the quality of the topics from three perspectives: 1) topic interpretability, 2) how documents relate to the underlying topics, and 3) how interpretable the topics are when the corpus has an underlying categorization. We provide insights into how well the results of Mechanical Turk match automated methods for calculating topic quality. The probability distribution of the words in the topic best fit the Topic Coherence measure, in terms of both correlation as well as finding the best topics.",

keywords = "Text analysis, Text mining, Topic analysis, Topic modeling",

author = "Fred Morstatter and J{\"u}rgen Pfeffer and Katja Mayer and Huan Liu",

note = "Funding Information: This work is sponsored, in part, by O_ce of Naval Research grant N000141410095 as well as LexisNexis and HPCC Systems.; 26th ACM Conference on Hypertext and Social Media, HT 2015 ; Conference date: 01-09-2015 Through 04-09-2015",

year = "2015",

month = aug,

day = "24",

doi = "10.1145/2700171.2791028",

language = "English (US)",

series = "HT 2015 - Proceedings of the 26th ACM Conference on Hypertext and Social Media",

publisher = "Association for Computing Machinery, Inc",

pages = "123--131",

booktitle = "HT 2015 - Proceedings of the 26th ACM Conference on Hypertext and Social Media",

}

TY - GEN

T1 - Text, topics, and turkers

T2 - 26th ACM Conference on Hypertext and Social Media, HT 2015

AU - Morstatter, Fred

AU - Pfeffer, Jürgen

AU - Mayer, Katja

AU - Liu, Huan

N1 - Funding Information: This work is sponsored, in part, by O_ce of Naval Research grant N000141410095 as well as LexisNexis and HPCC Systems.

PY - 2015/8/24

Y1 - 2015/8/24

N2 - Topic modeling is an important tool in social media analysis, allowing researchers to quickly understand large text corpora by investigating the topics underlying them. One of the fundamental problems of topic models lies in how to assess the quality of the topics from the perspective of human interpretability. How well can humans understand the meaning of topics generated by statistical topic modeling algorithms? In this work we advance the study of this question by introducing Topic Consensus: a new measure that calculates the quality of a topic through investigating its consensus with some known topics underlying the data. We view the quality of the topics from three perspectives: 1) topic interpretability, 2) how documents relate to the underlying topics, and 3) how interpretable the topics are when the corpus has an underlying categorization. We provide insights into how well the results of Mechanical Turk match automated methods for calculating topic quality. The probability distribution of the words in the topic best fit the Topic Coherence measure, in terms of both correlation as well as finding the best topics.

AB - Topic modeling is an important tool in social media analysis, allowing researchers to quickly understand large text corpora by investigating the topics underlying them. One of the fundamental problems of topic models lies in how to assess the quality of the topics from the perspective of human interpretability. How well can humans understand the meaning of topics generated by statistical topic modeling algorithms? In this work we advance the study of this question by introducing Topic Consensus: a new measure that calculates the quality of a topic through investigating its consensus with some known topics underlying the data. We view the quality of the topics from three perspectives: 1) topic interpretability, 2) how documents relate to the underlying topics, and 3) how interpretable the topics are when the corpus has an underlying categorization. We provide insights into how well the results of Mechanical Turk match automated methods for calculating topic quality. The probability distribution of the words in the topic best fit the Topic Coherence measure, in terms of both correlation as well as finding the best topics.

KW - Text analysis

KW - Text mining

KW - Topic analysis

KW - Topic modeling

UR - http://www.scopus.com/inward/record.url?scp=84956970377&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84956970377&partnerID=8YFLogxK

U2 - 10.1145/2700171.2791028

DO - 10.1145/2700171.2791028

M3 - Conference contribution

AN - SCOPUS:84956970377

T3 - HT 2015 - Proceedings of the 26th ACM Conference on Hypertext and Social Media

SP - 123

EP - 131

BT - HT 2015 - Proceedings of the 26th ACM Conference on Hypertext and Social Media

PB - Association for Computing Machinery, Inc

Y2 - 1 September 2015 through 4 September 2015

ER -

Text, topics, and turkers: A consensus measure for statistical topics

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this