Bias analysis in text classification for highly skewed data

Lei Tang, Huan Liu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

29 Scopus citations

Abstract

Feature selection is often applied to high-dimensional data as a preprocessing step in text classification. When dealing with highly skewed data, we observe that typical feature selection metrics like information gain or chi-squared are biased toward selecting features for the minor class, and the metric of bi-normal separation can select features for both minor and major classes. In this work, we investigate how these feature selection metrics impact on the performance of frequently used classifiers such as Decision Trees, Naïve Bayes, and Support Vector Machines via bias analysis for highly skewed data. Three types of biases are metric bias, class bias, and classifier bias. Extensive experiments are designed to understand how these biases can be employed in concert and efficiently to achieve good classification performance. We report our findings and present recommended approaches to text classification based on bias analysis and the empirical study.

Original languageEnglish (US)
Title of host publicationProceedings - Fifth IEEE International Conference on Data Mining, ICDM 2005
Pages781-784
Number of pages4
DOIs
StatePublished - Dec 1 2005
Event5th IEEE International Conference on Data Mining, ICDM 2005 - Houston, TX, United States
Duration: Nov 27 2005Nov 30 2005

Publication series

NameProceedings - IEEE International Conference on Data Mining, ICDM
ISSN (Print)1550-4786

Other

Other5th IEEE International Conference on Data Mining, ICDM 2005
CountryUnited States
CityHouston, TX
Period11/27/0511/30/05

ASJC Scopus subject areas

  • Engineering(all)

Fingerprint Dive into the research topics of 'Bias analysis in text classification for highly skewed data'. Together they form a unique fingerprint.

Cite this