Intraclass Correlations and Covariate Outcome Correlations for Planning Two- and Three-Level Cluster-Randomized Experiments in Education

Larry V. Hedges, E. C. Hedberg

Research output: Contribution to journalArticle

19 Citations (Scopus)

Abstract

Background: Cluster-randomized experiments that assign intact groups such as schools or school districts to treatment conditions are increasingly common in educational research. Such experiments are inherently multilevel designs whose sensitivity (statistical power and precision of estimates) depends on the variance decomposition across levels. This variance decomposition is usually summarized by the intraclass correlation (ICC) structure and, if covariates are used, the effectiveness of the covariates in explaining variation at each level of the design. Objectives: This article provides a compilation of school- and district-level ICC values of academic achievement and related covariate effectiveness based on state longitudinal data systems. These values are designed to be used for planning group-randomized experiments in education. The use of these values to compute statistical power and plan two- and three-level group-randomized experiments is illustrated. Research Design: We fit several hierarchical linear models to state data by grade and subject to estimate ICCs and covariate effectiveness. The total sample size is over 4.8 million students. We then compare our average of state estimates with the national work by Hedges and Hedberg.

Original languageEnglish (US)
Pages (from-to)445-489
Number of pages45
JournalEvaluation Review
Volume37
Issue number6
DOIs
StatePublished - 2013

Fingerprint

planning
experiment
education
district
school
Values
Group
linear model
educational research
academic achievement
research planning
Planning
Education
Randomized Experiments
Decomposition
student
Educational Research
Hierarchical Linear Models
School Districts
Sample Size

Keywords

  • content area
  • education
  • methodological development

ASJC Scopus subject areas

  • Social Sciences(all)
  • Arts and Humanities (miscellaneous)

Cite this

Intraclass Correlations and Covariate Outcome Correlations for Planning Two- and Three-Level Cluster-Randomized Experiments in Education. / Hedges, Larry V.; Hedberg, E. C.

In: Evaluation Review, Vol. 37, No. 6, 2013, p. 445-489.

Research output: Contribution to journalArticle

@article{5bbe1ee31ed248509d918c6ef8d6b86a,
title = "Intraclass Correlations and Covariate Outcome Correlations for Planning Two- and Three-Level Cluster-Randomized Experiments in Education",
abstract = "Background: Cluster-randomized experiments that assign intact groups such as schools or school districts to treatment conditions are increasingly common in educational research. Such experiments are inherently multilevel designs whose sensitivity (statistical power and precision of estimates) depends on the variance decomposition across levels. This variance decomposition is usually summarized by the intraclass correlation (ICC) structure and, if covariates are used, the effectiveness of the covariates in explaining variation at each level of the design. Objectives: This article provides a compilation of school- and district-level ICC values of academic achievement and related covariate effectiveness based on state longitudinal data systems. These values are designed to be used for planning group-randomized experiments in education. The use of these values to compute statistical power and plan two- and three-level group-randomized experiments is illustrated. Research Design: We fit several hierarchical linear models to state data by grade and subject to estimate ICCs and covariate effectiveness. The total sample size is over 4.8 million students. We then compare our average of state estimates with the national work by Hedges and Hedberg.",
keywords = "content area, education, methodological development",
author = "Hedges, {Larry V.} and Hedberg, {E. C.}",
year = "2013",
doi = "10.1177/0193841X14529126",
language = "English (US)",
volume = "37",
pages = "445--489",
journal = "Evaluation Review",
issn = "0193-841X",
publisher = "SAGE Publications Inc.",
number = "6",

}

TY - JOUR

T1 - Intraclass Correlations and Covariate Outcome Correlations for Planning Two- and Three-Level Cluster-Randomized Experiments in Education

AU - Hedges, Larry V.

AU - Hedberg, E. C.

PY - 2013

Y1 - 2013

N2 - Background: Cluster-randomized experiments that assign intact groups such as schools or school districts to treatment conditions are increasingly common in educational research. Such experiments are inherently multilevel designs whose sensitivity (statistical power and precision of estimates) depends on the variance decomposition across levels. This variance decomposition is usually summarized by the intraclass correlation (ICC) structure and, if covariates are used, the effectiveness of the covariates in explaining variation at each level of the design. Objectives: This article provides a compilation of school- and district-level ICC values of academic achievement and related covariate effectiveness based on state longitudinal data systems. These values are designed to be used for planning group-randomized experiments in education. The use of these values to compute statistical power and plan two- and three-level group-randomized experiments is illustrated. Research Design: We fit several hierarchical linear models to state data by grade and subject to estimate ICCs and covariate effectiveness. The total sample size is over 4.8 million students. We then compare our average of state estimates with the national work by Hedges and Hedberg.

AB - Background: Cluster-randomized experiments that assign intact groups such as schools or school districts to treatment conditions are increasingly common in educational research. Such experiments are inherently multilevel designs whose sensitivity (statistical power and precision of estimates) depends on the variance decomposition across levels. This variance decomposition is usually summarized by the intraclass correlation (ICC) structure and, if covariates are used, the effectiveness of the covariates in explaining variation at each level of the design. Objectives: This article provides a compilation of school- and district-level ICC values of academic achievement and related covariate effectiveness based on state longitudinal data systems. These values are designed to be used for planning group-randomized experiments in education. The use of these values to compute statistical power and plan two- and three-level group-randomized experiments is illustrated. Research Design: We fit several hierarchical linear models to state data by grade and subject to estimate ICCs and covariate effectiveness. The total sample size is over 4.8 million students. We then compare our average of state estimates with the national work by Hedges and Hedberg.

KW - content area

KW - education

KW - methodological development

UR - http://www.scopus.com/inward/record.url?scp=84903190149&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84903190149&partnerID=8YFLogxK

U2 - 10.1177/0193841X14529126

DO - 10.1177/0193841X14529126

M3 - Article

AN - SCOPUS:84903190149

VL - 37

SP - 445

EP - 489

JO - Evaluation Review

JF - Evaluation Review

SN - 0193-841X

IS - 6

ER -