Generative multimodal models of nonverbal synchrony in close relationships

Joseph Grafsgaard, Nicholas Duran, Ashley Randall, Chun Tao, Sidney D'Mello

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

Positive interpersonal relationships require shared understanding along with a sense of rapport. A key facet of rapport is mirroring and convergence of facial expression and body language, known as nonverbal synchrony. We examined nonverbal synchrony in a study of 29 heterosexual romantic couples, in which audio, video, and bracelet accelerometer were recorded during three conversations. We extracted facial expression, body movement, and acoustic-prosodic features to train neural network models that predicted the nonverbal behaviors of one partner from those of the other. Recurrent models (LSTMs) outperformed feed-forward neural networks and other chance baselines. The models learned behaviors encompassing facial responses, speech-related facial movements, and head movement. However, they did not capture fleeting or periodic behaviors, such as nodding, head turning, and hand gestures. Notably, a preliminary analysis of clinical measures showed greater association with our model outputs than correlation of raw signals. We discuss potential uses of these generative models as a research tool to complement current analytical methods along with real-world applications (e.g., as a tool in therapy).

Original languageEnglish (US)
Title of host publicationProceedings - 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages195-202
Number of pages8
ISBN (Electronic)9781538623350
DOIs
StatePublished - Jun 5 2018
Event13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018 - Xi'an, China
Duration: May 15 2018May 19 2018

Other

Other13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018
CountryChina
CityXi'an
Period5/15/185/19/18

Fingerprint

Synchrony
Facial Expression
Generative Models
Accelerometer
Feedforward Neural Networks
Gesture
Real-world Applications
Analytical Methods
Neural Network Model
Facet
Therapy
Baseline
Acoustics
Complement
Model
Feedforward neural networks
Accelerometers
Output
Movement
Relationships

Keywords

  • Close relationships
  • Couples therapy
  • Facial expression
  • LSTM
  • Neural networks
  • Nonverbal synchrony

ASJC Scopus subject areas

  • Computer Vision and Pattern Recognition
  • Control and Optimization

Cite this

Grafsgaard, J., Duran, N., Randall, A., Tao, C., & D'Mello, S. (2018). Generative multimodal models of nonverbal synchrony in close relationships. In Proceedings - 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018 (pp. 195-202). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/FG.2018.00037

Generative multimodal models of nonverbal synchrony in close relationships. / Grafsgaard, Joseph; Duran, Nicholas; Randall, Ashley; Tao, Chun; D'Mello, Sidney.

Proceedings - 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 195-202.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Grafsgaard, J, Duran, N, Randall, A, Tao, C & D'Mello, S 2018, Generative multimodal models of nonverbal synchrony in close relationships. in Proceedings - 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018. Institute of Electrical and Electronics Engineers Inc., pp. 195-202, 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018, Xi'an, China, 5/15/18. https://doi.org/10.1109/FG.2018.00037
Grafsgaard J, Duran N, Randall A, Tao C, D'Mello S. Generative multimodal models of nonverbal synchrony in close relationships. In Proceedings - 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 195-202 https://doi.org/10.1109/FG.2018.00037
Grafsgaard, Joseph ; Duran, Nicholas ; Randall, Ashley ; Tao, Chun ; D'Mello, Sidney. / Generative multimodal models of nonverbal synchrony in close relationships. Proceedings - 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 195-202
@inproceedings{3c2e1fd98e61473eba59cc69f07f98bf,
title = "Generative multimodal models of nonverbal synchrony in close relationships",
abstract = "Positive interpersonal relationships require shared understanding along with a sense of rapport. A key facet of rapport is mirroring and convergence of facial expression and body language, known as nonverbal synchrony. We examined nonverbal synchrony in a study of 29 heterosexual romantic couples, in which audio, video, and bracelet accelerometer were recorded during three conversations. We extracted facial expression, body movement, and acoustic-prosodic features to train neural network models that predicted the nonverbal behaviors of one partner from those of the other. Recurrent models (LSTMs) outperformed feed-forward neural networks and other chance baselines. The models learned behaviors encompassing facial responses, speech-related facial movements, and head movement. However, they did not capture fleeting or periodic behaviors, such as nodding, head turning, and hand gestures. Notably, a preliminary analysis of clinical measures showed greater association with our model outputs than correlation of raw signals. We discuss potential uses of these generative models as a research tool to complement current analytical methods along with real-world applications (e.g., as a tool in therapy).",
keywords = "Close relationships, Couples therapy, Facial expression, LSTM, Neural networks, Nonverbal synchrony",
author = "Joseph Grafsgaard and Nicholas Duran and Ashley Randall and Chun Tao and Sidney D'Mello",
year = "2018",
month = "6",
day = "5",
doi = "10.1109/FG.2018.00037",
language = "English (US)",
pages = "195--202",
booktitle = "Proceedings - 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Generative multimodal models of nonverbal synchrony in close relationships

AU - Grafsgaard, Joseph

AU - Duran, Nicholas

AU - Randall, Ashley

AU - Tao, Chun

AU - D'Mello, Sidney

PY - 2018/6/5

Y1 - 2018/6/5

N2 - Positive interpersonal relationships require shared understanding along with a sense of rapport. A key facet of rapport is mirroring and convergence of facial expression and body language, known as nonverbal synchrony. We examined nonverbal synchrony in a study of 29 heterosexual romantic couples, in which audio, video, and bracelet accelerometer were recorded during three conversations. We extracted facial expression, body movement, and acoustic-prosodic features to train neural network models that predicted the nonverbal behaviors of one partner from those of the other. Recurrent models (LSTMs) outperformed feed-forward neural networks and other chance baselines. The models learned behaviors encompassing facial responses, speech-related facial movements, and head movement. However, they did not capture fleeting or periodic behaviors, such as nodding, head turning, and hand gestures. Notably, a preliminary analysis of clinical measures showed greater association with our model outputs than correlation of raw signals. We discuss potential uses of these generative models as a research tool to complement current analytical methods along with real-world applications (e.g., as a tool in therapy).

AB - Positive interpersonal relationships require shared understanding along with a sense of rapport. A key facet of rapport is mirroring and convergence of facial expression and body language, known as nonverbal synchrony. We examined nonverbal synchrony in a study of 29 heterosexual romantic couples, in which audio, video, and bracelet accelerometer were recorded during three conversations. We extracted facial expression, body movement, and acoustic-prosodic features to train neural network models that predicted the nonverbal behaviors of one partner from those of the other. Recurrent models (LSTMs) outperformed feed-forward neural networks and other chance baselines. The models learned behaviors encompassing facial responses, speech-related facial movements, and head movement. However, they did not capture fleeting or periodic behaviors, such as nodding, head turning, and hand gestures. Notably, a preliminary analysis of clinical measures showed greater association with our model outputs than correlation of raw signals. We discuss potential uses of these generative models as a research tool to complement current analytical methods along with real-world applications (e.g., as a tool in therapy).

KW - Close relationships

KW - Couples therapy

KW - Facial expression

KW - LSTM

KW - Neural networks

KW - Nonverbal synchrony

UR - http://www.scopus.com/inward/record.url?scp=85049389284&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85049389284&partnerID=8YFLogxK

U2 - 10.1109/FG.2018.00037

DO - 10.1109/FG.2018.00037

M3 - Conference contribution

SP - 195

EP - 202

BT - Proceedings - 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -