An iterative approach to case study analysis: Insights from qualitative analysis of quantitative inconsistencies

Allain J. Barnett, Jacopo A. Baggio, Hoon C. Shin, David J. Yu, Irene Perez-Ibarra, Cathy Rubiños, Ute Brady, Elicia Ratajczyk, Nathan Rollins, Rimjhim Aggarwal, John M. Anderies, Marco A. Janssen

Research output: Contribution to journalArticle

  • 6 Citations

Abstract

Large-N comparative studies have helped common pool resource scholars gain general insights into the factors that influence collective action and governance outcomes. However, these studies are often limited by missing data, and suffer from the methodological limitation that important information is lost when we reduce textual information to quantitative data. This study was motivated by nine case studies that appeared to be inconsistent with the expectation that the presence of Ostrom’s Design Principles increases the likelihood of successful common pool resource governance. These cases highlight the limitations of coding and analysing large-N case studies. We examine two issues: 1) the challenge of missing data and 2) potential approaches that rely on context (which is often lost in the coding process) to address inconsistencies between empirical observations and theoretical predictions. For the latter, we conduct a post-hoc qualitative analysis of a large-N comparative study to explore 2 types of inconsistencies: 1) cases where evidence for nearly all design principles was found, but available evidence led to the assessment that the CPR system was unsuccessful and 2) cases where the CPR system was deemed successful despite finding limited or no evidence for design principles. We describe inherent challenges to large-N comparative analysis and to coding complex and dynamically changing common pool resource systems for the presence or absence of design principles and the determination of “success”. Finally, we illustrate how, in some cases, our qualitative analysis revealed that the identity of absent design principles explained inconsistencies, and hence reconciled such apparent inconsistencies with theoretical predictions. This analysis demonstrates the value of combining quantitative and qualitative analysis, and using mixed-methods approaches iteratively to build comprehensive methodological and theoretical approaches to understanding common pool resource governance in a dynamically changing context.

Original languageEnglish (US)
Pages (from-to)467-494
Number of pages28
JournalInternational Journal of the Commons
Volume10
Issue number2
DOIs
StatePublished - 2016

Fingerprint

analysis
resources
coding
governance
case study
system
data
prognosis
information
collective behavior
expectation
observation
success
identity
assessment
understanding
influence exertion
method
process

Keywords

  • Case study analysis
  • Common pool resources
  • Coupled infrastructure systems
  • Design principles
  • Institutional analysis
  • Large-N
  • Mixed methods

ASJC Scopus subject areas

  • Sociology and Political Science

Cite this

An iterative approach to case study analysis : Insights from qualitative analysis of quantitative inconsistencies. / Barnett, Allain J.; Baggio, Jacopo A.; Shin, Hoon C.; Yu, David J.; Perez-Ibarra, Irene; Rubiños, Cathy; Brady, Ute; Ratajczyk, Elicia; Rollins, Nathan; Aggarwal, Rimjhim; Anderies, John M.; Janssen, Marco A.

In: International Journal of the Commons, Vol. 10, No. 2, 2016, p. 467-494.

Research output: Contribution to journalArticle

Barnett, AJ, Baggio, JA, Shin, HC, Yu, DJ, Perez-Ibarra, I, Rubiños, C, Brady, U, Ratajczyk, E, Rollins, N, Aggarwal, R, Anderies, JM & Janssen, MA 2016, 'An iterative approach to case study analysis: Insights from qualitative analysis of quantitative inconsistencies' International Journal of the Commons, vol 10, no. 2, pp. 467-494. DOI: 10.18352/ijc.632
Barnett AJ, Baggio JA, Shin HC, Yu DJ, Perez-Ibarra I, Rubiños C et al. An iterative approach to case study analysis: Insights from qualitative analysis of quantitative inconsistencies. International Journal of the Commons. 2016;10(2):467-494. Available from, DOI: 10.18352/ijc.632

Barnett, Allain J.; Baggio, Jacopo A.; Shin, Hoon C.; Yu, David J.; Perez-Ibarra, Irene; Rubiños, Cathy; Brady, Ute; Ratajczyk, Elicia; Rollins, Nathan; Aggarwal, Rimjhim; Anderies, John M.; Janssen, Marco A. / An iterative approach to case study analysis : Insights from qualitative analysis of quantitative inconsistencies.

In: International Journal of the Commons, Vol. 10, No. 2, 2016, p. 467-494.

Research output: Contribution to journalArticle

@article{f341b77a64ac4bdaafabf9a89509241f,
title = "An iterative approach to case study analysis: Insights from qualitative analysis of quantitative inconsistencies",
abstract = "Large-N comparative studies have helped common pool resource scholars gain general insights into the factors that influence collective action and governance outcomes. However, these studies are often limited by missing data, and suffer from the methodological limitation that important information is lost when we reduce textual information to quantitative data. This study was motivated by nine case studies that appeared to be inconsistent with the expectation that the presence of Ostrom’s Design Principles increases the likelihood of successful common pool resource governance. These cases highlight the limitations of coding and analysing large-N case studies. We examine two issues: 1) the challenge of missing data and 2) potential approaches that rely on context (which is often lost in the coding process) to address inconsistencies between empirical observations and theoretical predictions. For the latter, we conduct a post-hoc qualitative analysis of a large-N comparative study to explore 2 types of inconsistencies: 1) cases where evidence for nearly all design principles was found, but available evidence led to the assessment that the CPR system was unsuccessful and 2) cases where the CPR system was deemed successful despite finding limited or no evidence for design principles. We describe inherent challenges to large-N comparative analysis and to coding complex and dynamically changing common pool resource systems for the presence or absence of design principles and the determination of “success”. Finally, we illustrate how, in some cases, our qualitative analysis revealed that the identity of absent design principles explained inconsistencies, and hence reconciled such apparent inconsistencies with theoretical predictions. This analysis demonstrates the value of combining quantitative and qualitative analysis, and using mixed-methods approaches iteratively to build comprehensive methodological and theoretical approaches to understanding common pool resource governance in a dynamically changing context.",
keywords = "Case study analysis, Common pool resources, Coupled infrastructure systems, Design principles, Institutional analysis, Large-N, Mixed methods",
author = "Barnett, {Allain J.} and Baggio, {Jacopo A.} and Shin, {Hoon C.} and Yu, {David J.} and Irene Perez-Ibarra and Cathy Rubiños and Ute Brady and Elicia Ratajczyk and Nathan Rollins and Rimjhim Aggarwal and Anderies, {John M.} and Janssen, {Marco A.}",
year = "2016",
doi = "10.18352/ijc.632",
volume = "10",
pages = "467--494",
journal = "International Journal of the Commons",
issn = "1875-0281",
publisher = "International Association for the Study of the Commons",
number = "2",

}

TY - JOUR

T1 - An iterative approach to case study analysis

T2 - International Journal of the Commons

AU - Barnett,Allain J.

AU - Baggio,Jacopo A.

AU - Shin,Hoon C.

AU - Yu,David J.

AU - Perez-Ibarra,Irene

AU - Rubiños,Cathy

AU - Brady,Ute

AU - Ratajczyk,Elicia

AU - Rollins,Nathan

AU - Aggarwal,Rimjhim

AU - Anderies,John M.

AU - Janssen,Marco A.

PY - 2016

Y1 - 2016

N2 - Large-N comparative studies have helped common pool resource scholars gain general insights into the factors that influence collective action and governance outcomes. However, these studies are often limited by missing data, and suffer from the methodological limitation that important information is lost when we reduce textual information to quantitative data. This study was motivated by nine case studies that appeared to be inconsistent with the expectation that the presence of Ostrom’s Design Principles increases the likelihood of successful common pool resource governance. These cases highlight the limitations of coding and analysing large-N case studies. We examine two issues: 1) the challenge of missing data and 2) potential approaches that rely on context (which is often lost in the coding process) to address inconsistencies between empirical observations and theoretical predictions. For the latter, we conduct a post-hoc qualitative analysis of a large-N comparative study to explore 2 types of inconsistencies: 1) cases where evidence for nearly all design principles was found, but available evidence led to the assessment that the CPR system was unsuccessful and 2) cases where the CPR system was deemed successful despite finding limited or no evidence for design principles. We describe inherent challenges to large-N comparative analysis and to coding complex and dynamically changing common pool resource systems for the presence or absence of design principles and the determination of “success”. Finally, we illustrate how, in some cases, our qualitative analysis revealed that the identity of absent design principles explained inconsistencies, and hence reconciled such apparent inconsistencies with theoretical predictions. This analysis demonstrates the value of combining quantitative and qualitative analysis, and using mixed-methods approaches iteratively to build comprehensive methodological and theoretical approaches to understanding common pool resource governance in a dynamically changing context.

AB - Large-N comparative studies have helped common pool resource scholars gain general insights into the factors that influence collective action and governance outcomes. However, these studies are often limited by missing data, and suffer from the methodological limitation that important information is lost when we reduce textual information to quantitative data. This study was motivated by nine case studies that appeared to be inconsistent with the expectation that the presence of Ostrom’s Design Principles increases the likelihood of successful common pool resource governance. These cases highlight the limitations of coding and analysing large-N case studies. We examine two issues: 1) the challenge of missing data and 2) potential approaches that rely on context (which is often lost in the coding process) to address inconsistencies between empirical observations and theoretical predictions. For the latter, we conduct a post-hoc qualitative analysis of a large-N comparative study to explore 2 types of inconsistencies: 1) cases where evidence for nearly all design principles was found, but available evidence led to the assessment that the CPR system was unsuccessful and 2) cases where the CPR system was deemed successful despite finding limited or no evidence for design principles. We describe inherent challenges to large-N comparative analysis and to coding complex and dynamically changing common pool resource systems for the presence or absence of design principles and the determination of “success”. Finally, we illustrate how, in some cases, our qualitative analysis revealed that the identity of absent design principles explained inconsistencies, and hence reconciled such apparent inconsistencies with theoretical predictions. This analysis demonstrates the value of combining quantitative and qualitative analysis, and using mixed-methods approaches iteratively to build comprehensive methodological and theoretical approaches to understanding common pool resource governance in a dynamically changing context.

KW - Case study analysis

KW - Common pool resources

KW - Coupled infrastructure systems

KW - Design principles

KW - Institutional analysis

KW - Large-N

KW - Mixed methods

UR - http://www.scopus.com/inward/record.url?scp=84992153338&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84992153338&partnerID=8YFLogxK

U2 - 10.18352/ijc.632

DO - 10.18352/ijc.632

M3 - Article

VL - 10

SP - 467

EP - 494

JO - International Journal of the Commons

JF - International Journal of the Commons

SN - 1875-0281

IS - 2

ER -