Generative multimodal models of nonverbal synchrony in close relationships

Joseph Grafsgaard, Nicholas Duran, Ashley Randall, Chun Tao, Sidney D'Mello

Research output: Chapter in Book/Report/Conference proceedingConference contribution

8 Scopus citations

Abstract

Positive interpersonal relationships require shared understanding along with a sense of rapport. A key facet of rapport is mirroring and convergence of facial expression and body language, known as nonverbal synchrony. We examined nonverbal synchrony in a study of 29 heterosexual romantic couples, in which audio, video, and bracelet accelerometer were recorded during three conversations. We extracted facial expression, body movement, and acoustic-prosodic features to train neural network models that predicted the nonverbal behaviors of one partner from those of the other. Recurrent models (LSTMs) outperformed feed-forward neural networks and other chance baselines. The models learned behaviors encompassing facial responses, speech-related facial movements, and head movement. However, they did not capture fleeting or periodic behaviors, such as nodding, head turning, and hand gestures. Notably, a preliminary analysis of clinical measures showed greater association with our model outputs than correlation of raw signals. We discuss potential uses of these generative models as a research tool to complement current analytical methods along with real-world applications (e.g., as a tool in therapy).

Original languageEnglish (US)
Title of host publicationProceedings - 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages195-202
Number of pages8
ISBN (Electronic)9781538623350
DOIs
StatePublished - Jun 5 2018
Event13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018 - Xi'an, China
Duration: May 15 2018May 19 2018

Other

Other13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018
CountryChina
CityXi'an
Period5/15/185/19/18

Keywords

  • Close relationships
  • Couples therapy
  • Facial expression
  • LSTM
  • Neural networks
  • Nonverbal synchrony

ASJC Scopus subject areas

  • Computer Vision and Pattern Recognition
  • Control and Optimization

Fingerprint Dive into the research topics of 'Generative multimodal models of nonverbal synchrony in close relationships'. Together they form a unique fingerprint.

  • Cite this

    Grafsgaard, J., Duran, N., Randall, A., Tao, C., & D'Mello, S. (2018). Generative multimodal models of nonverbal synchrony in close relationships. In Proceedings - 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018 (pp. 195-202). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/FG.2018.00037