Computer Vision for Augmentative and Alternative Communication

Augmentative and Alternative Communication (AAC) includes all forms of communication that are used to supplement speech for those with impairments in the production or comprehension of spoken language. AAC technology has mostly been used by people with severe speech or language impairments. Individuals with visual impairments also face a fundamental limitation in communicating with their sighted peers, as about 65% of the information during social interactions is conveyed using nonverbal cues. In this chapter, we present our computer vision research contributions in the design and development of a Social Interaction Assistant (SIA), which is an AAC technology that can enrich the communication experience of individuals with visual impairments.Moreover, individuals with visual impairments often have specific requirements that necessitate a personalized, adaptive approach to multimedia computing, rather than a "one-size-fits-all" approach. To address this challenge, our proposed solutions place emphasis on understanding the individual user's needs, expectations,and adaptations toward designing, developing, and deploying effective multimedia solutions. Our empirical results demonstrate the significant potential in using person-centered AAC technology to enrich the communication experience of individuals with visual impairments.