Mode effects in cultural domain analysis: Comparing pile sort data collected via internet versus face-to-face interviews

Clarence C. Gravlee, Chad R. Maxwell, Aryeh Jacobsohn, Harvey Bernard

    Research output: Contribution to journalArticle

    1 Citation (Scopus)

    Abstract

    This article tests whether collecting pile-sort data online produces results similar to those obtained with face-to-face methods. We collected pile sorts from 227 university students in the cultural domain of emotions. To test for mode and design effects, we randomly assigned participants to faceto- face or internet modes and to either a 15- or 30-item sort. We found no evidence of mode effects in semantic structure, but the level of agreement among respondents varied by mode, depending on the number of items in the task. Agreement was higher among online respondents in the 15-item task but among face-to-face respondents in the 30-item one. Because the central purpose of pile sorting is to elicit semantic structure, we conclude that online data collection is a viable option for most researchers, but the mode effect in level of agreement implies that other design elements may affect comparability of face-to-face and web-based methods.

    Original languageEnglish (US)
    Pages (from-to)165-176
    Number of pages12
    JournalInternational Journal of Social Research Methodology
    Volume21
    Issue number2
    DOIs
    StatePublished - Jun 24 2018

    Fingerprint

    cultural analysis
    Internet
    interview
    semantics
    emotion
    university
    evidence
    student

    Keywords

    • Cultural domain analysis
    • Mode effects
    • Pile sort

    ASJC Scopus subject areas

    • Social Sciences(all)

    Cite this

    Mode effects in cultural domain analysis : Comparing pile sort data collected via internet versus face-to-face interviews. / Gravlee, Clarence C.; Maxwell, Chad R.; Jacobsohn, Aryeh; Bernard, Harvey.

    In: International Journal of Social Research Methodology, Vol. 21, No. 2, 24.06.2018, p. 165-176.

    Research output: Contribution to journalArticle

    @article{9adb2cadad624d90905edc33feccf75b,
    title = "Mode effects in cultural domain analysis: Comparing pile sort data collected via internet versus face-to-face interviews",
    abstract = "This article tests whether collecting pile-sort data online produces results similar to those obtained with face-to-face methods. We collected pile sorts from 227 university students in the cultural domain of emotions. To test for mode and design effects, we randomly assigned participants to faceto- face or internet modes and to either a 15- or 30-item sort. We found no evidence of mode effects in semantic structure, but the level of agreement among respondents varied by mode, depending on the number of items in the task. Agreement was higher among online respondents in the 15-item task but among face-to-face respondents in the 30-item one. Because the central purpose of pile sorting is to elicit semantic structure, we conclude that online data collection is a viable option for most researchers, but the mode effect in level of agreement implies that other design elements may affect comparability of face-to-face and web-based methods.",
    keywords = "Cultural domain analysis, Mode effects, Pile sort",
    author = "Gravlee, {Clarence C.} and Maxwell, {Chad R.} and Aryeh Jacobsohn and Harvey Bernard",
    year = "2018",
    month = "6",
    day = "24",
    doi = "10.1080/13645579.2017.1341187",
    language = "English (US)",
    volume = "21",
    pages = "165--176",
    journal = "International Journal of Social Research Methodology: Theory and Practice",
    issn = "1364-5579",
    publisher = "Taylor and Francis Ltd.",
    number = "2",

    }

    TY - JOUR

    T1 - Mode effects in cultural domain analysis

    T2 - Comparing pile sort data collected via internet versus face-to-face interviews

    AU - Gravlee, Clarence C.

    AU - Maxwell, Chad R.

    AU - Jacobsohn, Aryeh

    AU - Bernard, Harvey

    PY - 2018/6/24

    Y1 - 2018/6/24

    N2 - This article tests whether collecting pile-sort data online produces results similar to those obtained with face-to-face methods. We collected pile sorts from 227 university students in the cultural domain of emotions. To test for mode and design effects, we randomly assigned participants to faceto- face or internet modes and to either a 15- or 30-item sort. We found no evidence of mode effects in semantic structure, but the level of agreement among respondents varied by mode, depending on the number of items in the task. Agreement was higher among online respondents in the 15-item task but among face-to-face respondents in the 30-item one. Because the central purpose of pile sorting is to elicit semantic structure, we conclude that online data collection is a viable option for most researchers, but the mode effect in level of agreement implies that other design elements may affect comparability of face-to-face and web-based methods.

    AB - This article tests whether collecting pile-sort data online produces results similar to those obtained with face-to-face methods. We collected pile sorts from 227 university students in the cultural domain of emotions. To test for mode and design effects, we randomly assigned participants to faceto- face or internet modes and to either a 15- or 30-item sort. We found no evidence of mode effects in semantic structure, but the level of agreement among respondents varied by mode, depending on the number of items in the task. Agreement was higher among online respondents in the 15-item task but among face-to-face respondents in the 30-item one. Because the central purpose of pile sorting is to elicit semantic structure, we conclude that online data collection is a viable option for most researchers, but the mode effect in level of agreement implies that other design elements may affect comparability of face-to-face and web-based methods.

    KW - Cultural domain analysis

    KW - Mode effects

    KW - Pile sort

    UR - http://www.scopus.com/inward/record.url?scp=85021277762&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85021277762&partnerID=8YFLogxK

    U2 - 10.1080/13645579.2017.1341187

    DO - 10.1080/13645579.2017.1341187

    M3 - Article

    VL - 21

    SP - 165

    EP - 176

    JO - International Journal of Social Research Methodology: Theory and Practice

    JF - International Journal of Social Research Methodology: Theory and Practice

    SN - 1364-5579

    IS - 2

    ER -