Researcher-centered design of statistics: Why Bayesian statistics better fit the culture and incentives of HCI

Matthew Kay, Gregory L. Nelson, Eric B. Hekler

Research output: Chapter in Book/Report/Conference proceedingConference contribution

32 Scopus citations

Abstract

A core tradition of HCI lies in the experimental evaluation of the effects of techniques and interfaces to determine if they are useful for achieving their purpose. However, our individual analyses tend to stand alone, and study results rarely accrue in more precise estimates via meta-analysis: in a literature search, we found only 56 meta-analyses in HCI in the ACM Digital Library, 3 of which were published at CHI (often called the top HCI venue). Yet meta-analysis is the gold standard for demonstrating robust quantitative knowledge. We treat this as a user-centered design problem: the failure to accrue quantitative knowledge is not the users' (i.e. researchers') failure, but a failure to consider those users' needs when designing statistical practice. Using simulation, we compare hypothetical publication worlds following existing frequentist against Bayesian practice. We show that Bayesian analysis yields more precise effects with each new study, facilitating knowledge accrual without traditional meta-analyses. Bayesian practices also allow more principled conclusions from small-n studies of novel techniques. These advantages make Bayesian practices a likely better fit for the culture and incentives of the field. Instead of admonishing ourselves to spend resources on larger studies, we propose using tools that more appropriately analyze small studies and encourage knowledge accrual from one study to the next. We also believe Bayesian methods can be adopted from the bottom up without the need for new incentives for replication or meta-analysis. These techniques offer the potential for a more user-(i.e. researcher-) centered approach to statistical analysis in HCI.

Original languageEnglish (US)
Title of host publicationCHI 2016 - Proceedings, 34th Annual CHI Conference on Human Factors in Computing Systems
PublisherAssociation for Computing Machinery
Pages4521-4532
Number of pages12
ISBN (Electronic)9781450333627
DOIs
StatePublished - May 7 2016
Event34th Annual Conference on Human Factors in Computing Systems, CHI 2016 - San Jose, United States
Duration: May 7 2016May 12 2016

Publication series

NameConference on Human Factors in Computing Systems - Proceedings

Other

Other34th Annual Conference on Human Factors in Computing Systems, CHI 2016
CountryUnited States
CitySan Jose
Period5/7/165/12/16

Keywords

  • Bayesian statistics
  • Effect size
  • Estimation
  • Meta-analysis
  • Replication
  • Small studies

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction
  • Computer Graphics and Computer-Aided Design

Fingerprint Dive into the research topics of 'Researcher-centered design of statistics: Why Bayesian statistics better fit the culture and incentives of HCI'. Together they form a unique fingerprint.

  • Cite this

    Kay, M., Nelson, G. L., & Hekler, E. B. (2016). Researcher-centered design of statistics: Why Bayesian statistics better fit the culture and incentives of HCI. In CHI 2016 - Proceedings, 34th Annual CHI Conference on Human Factors in Computing Systems (pp. 4521-4532). (Conference on Human Factors in Computing Systems - Proceedings). Association for Computing Machinery. https://doi.org/10.1145/2858036.2858465