Language without words: A pointillist model for natural language processing

Peiyou Song, Anhei Shu, David Phipps, Mohit Tiwari, Dan S. Wallach, Jedidiah R. Crandall, George F. Luger

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

This paper explores two separate questions: Can we perform natural language processing tasks without a lexicon?; and, Should we? Existing natural language processing techniques are either based on words as units or use units such as grams only for basic classification tasks. How close can a machine come to reasoning about the meanings of words and phrases in a corpus without using any lexicon, based only on grams? Our own motivation for posing this question is based on our efforts to find popular trends in words and phrases from online Chinese social media. This form of written Chinese uses so many neologisms, creative character placements, and combinations of writing systems that it has been dubbed the 'Martian Language.' Readers must often use visual queues, audible queues from reading out loud, and their knowledge and understanding of current events to understand a post. For analysis of popular trends, the specific problem is that it is difficult to build a lexicon when the invention of new ways to refer to a word or concept is easy and common. For natural language processing in general, we argue in this paper that new uses of language in social media will challenge machines' abilities to operate with words as the basic unit of understanding, not only in Chinese but potentially in other languages.

Original languageEnglish (US)
Title of host publication6th International Conference on Soft Computing and Intelligent Systems, and 13th International Symposium on Advanced Intelligence Systems, SCIS/ISIS 2012
Pages11-15
Number of pages5
DOIs
StatePublished - 2012
Externally publishedYes
Event2012 Joint 6th International Conference on Soft Computing and Intelligent Systems, SCIS 2012 and 13th International Symposium on Advanced Intelligence Systems, ISIS 2012 - Kobe, Japan
Duration: Nov 20 2012Nov 24 2012

Publication series

Name6th International Conference on Soft Computing and Intelligent Systems, and 13th International Symposium on Advanced Intelligence Systems, SCIS/ISIS 2012

Conference

Conference2012 Joint 6th International Conference on Soft Computing and Intelligent Systems, SCIS 2012 and 13th International Symposium on Advanced Intelligence Systems, ISIS 2012
CountryJapan
CityKobe
Period11/20/1211/24/12

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software

Fingerprint Dive into the research topics of 'Language without words: A pointillist model for natural language processing'. Together they form a unique fingerprint.

Cite this