An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention

Justin D. Smith, Thomas J. Dishion, Kimbree Brown, Karina Ramos, Naomi B. Knoble, Daniel S. Shaw, Melvin N. Wilson

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

The valid and reliable assessment of fidelity is critical at all stages of intervention research and is particularly germane to interpreting the results of efficacy and implementation trials. Ratings of protocol adherence typically are reliable, but ratings of therapist competence are plagued by low reliability. Because family context and case conceptualization guide the therapist’s delivery of interventions, the reliability of fidelity ratings might be improved if the coder is privy to client context in the form of an ecological assessment. We conducted a randomized experiment to test this hypothesis. A subsample of 46 families with 5-year-old children from a multisite randomized trial who participated in the feedback session of the Family Check-Up (FCU) intervention were selected. We randomly assigned FCU feedback sessions to be rated for fidelity to the protocol using the COACH rating system either after the coder reviewed the results of a recent ecological assessment or had not. Inter-rater reliability estimates of fidelity ratings were meaningfully higher for the assessment information condition compared to the no-information condition. Importantly, the reliability of the COACH mean score was found to be statistically significantly higher in the information condition. These findings suggest that the reliability of observational ratings of fidelity, particularly when the competence or quality of delivery is considered, could be improved by providing assessment data to the coders. Our findings might be most applicable to assessment-driven interventions, where assessment data explicitly guides therapist’s selection of intervention strategies tailored to the family’s context and needs, but they could also apply to other intervention programs and observational coding of context-dependent therapy processes, such as the working alliance.

Original languageEnglish (US)
Pages (from-to)62-70
Number of pages9
JournalPrevention Science
Volume17
Issue number1
DOIs
StatePublished - Jan 1 2016

Fingerprint

Mental Competency
Research
Therapeutics

Keywords

  • Competence
  • Family check-up
  • Fidelity
  • Implementation
  • Observational coding

ASJC Scopus subject areas

  • Public Health, Environmental and Occupational Health

Cite this

Smith, J. D., Dishion, T. J., Brown, K., Ramos, K., Knoble, N. B., Shaw, D. S., & Wilson, M. N. (2016). An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention. Prevention Science, 17(1), 62-70. https://doi.org/10.1007/s11121-015-0589-0

An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention. / Smith, Justin D.; Dishion, Thomas J.; Brown, Kimbree; Ramos, Karina; Knoble, Naomi B.; Shaw, Daniel S.; Wilson, Melvin N.

In: Prevention Science, Vol. 17, No. 1, 01.01.2016, p. 62-70.

Research output: Contribution to journalArticle

Smith, JD, Dishion, TJ, Brown, K, Ramos, K, Knoble, NB, Shaw, DS & Wilson, MN 2016, 'An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention', Prevention Science, vol. 17, no. 1, pp. 62-70. https://doi.org/10.1007/s11121-015-0589-0
Smith, Justin D. ; Dishion, Thomas J. ; Brown, Kimbree ; Ramos, Karina ; Knoble, Naomi B. ; Shaw, Daniel S. ; Wilson, Melvin N. / An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention. In: Prevention Science. 2016 ; Vol. 17, No. 1. pp. 62-70.
@article{b5fa1b0e2e3d40ef8b47ffbaad1c07c7,
title = "An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention",
abstract = "The valid and reliable assessment of fidelity is critical at all stages of intervention research and is particularly germane to interpreting the results of efficacy and implementation trials. Ratings of protocol adherence typically are reliable, but ratings of therapist competence are plagued by low reliability. Because family context and case conceptualization guide the therapist’s delivery of interventions, the reliability of fidelity ratings might be improved if the coder is privy to client context in the form of an ecological assessment. We conducted a randomized experiment to test this hypothesis. A subsample of 46 families with 5-year-old children from a multisite randomized trial who participated in the feedback session of the Family Check-Up (FCU) intervention were selected. We randomly assigned FCU feedback sessions to be rated for fidelity to the protocol using the COACH rating system either after the coder reviewed the results of a recent ecological assessment or had not. Inter-rater reliability estimates of fidelity ratings were meaningfully higher for the assessment information condition compared to the no-information condition. Importantly, the reliability of the COACH mean score was found to be statistically significantly higher in the information condition. These findings suggest that the reliability of observational ratings of fidelity, particularly when the competence or quality of delivery is considered, could be improved by providing assessment data to the coders. Our findings might be most applicable to assessment-driven interventions, where assessment data explicitly guides therapist’s selection of intervention strategies tailored to the family’s context and needs, but they could also apply to other intervention programs and observational coding of context-dependent therapy processes, such as the working alliance.",
keywords = "Competence, Family check-up, Fidelity, Implementation, Observational coding",
author = "Smith, {Justin D.} and Dishion, {Thomas J.} and Kimbree Brown and Karina Ramos and Knoble, {Naomi B.} and Shaw, {Daniel S.} and Wilson, {Melvin N.}",
year = "2016",
month = "1",
day = "1",
doi = "10.1007/s11121-015-0589-0",
language = "English (US)",
volume = "17",
pages = "62--70",
journal = "Prevention Science",
issn = "1389-4986",
publisher = "Springer New York",
number = "1",

}

TY - JOUR

T1 - An Experimental Study of Procedures to Enhance Ratings of Fidelity to an Evidence-Based Family Intervention

AU - Smith, Justin D.

AU - Dishion, Thomas J.

AU - Brown, Kimbree

AU - Ramos, Karina

AU - Knoble, Naomi B.

AU - Shaw, Daniel S.

AU - Wilson, Melvin N.

PY - 2016/1/1

Y1 - 2016/1/1

N2 - The valid and reliable assessment of fidelity is critical at all stages of intervention research and is particularly germane to interpreting the results of efficacy and implementation trials. Ratings of protocol adherence typically are reliable, but ratings of therapist competence are plagued by low reliability. Because family context and case conceptualization guide the therapist’s delivery of interventions, the reliability of fidelity ratings might be improved if the coder is privy to client context in the form of an ecological assessment. We conducted a randomized experiment to test this hypothesis. A subsample of 46 families with 5-year-old children from a multisite randomized trial who participated in the feedback session of the Family Check-Up (FCU) intervention were selected. We randomly assigned FCU feedback sessions to be rated for fidelity to the protocol using the COACH rating system either after the coder reviewed the results of a recent ecological assessment or had not. Inter-rater reliability estimates of fidelity ratings were meaningfully higher for the assessment information condition compared to the no-information condition. Importantly, the reliability of the COACH mean score was found to be statistically significantly higher in the information condition. These findings suggest that the reliability of observational ratings of fidelity, particularly when the competence or quality of delivery is considered, could be improved by providing assessment data to the coders. Our findings might be most applicable to assessment-driven interventions, where assessment data explicitly guides therapist’s selection of intervention strategies tailored to the family’s context and needs, but they could also apply to other intervention programs and observational coding of context-dependent therapy processes, such as the working alliance.

AB - The valid and reliable assessment of fidelity is critical at all stages of intervention research and is particularly germane to interpreting the results of efficacy and implementation trials. Ratings of protocol adherence typically are reliable, but ratings of therapist competence are plagued by low reliability. Because family context and case conceptualization guide the therapist’s delivery of interventions, the reliability of fidelity ratings might be improved if the coder is privy to client context in the form of an ecological assessment. We conducted a randomized experiment to test this hypothesis. A subsample of 46 families with 5-year-old children from a multisite randomized trial who participated in the feedback session of the Family Check-Up (FCU) intervention were selected. We randomly assigned FCU feedback sessions to be rated for fidelity to the protocol using the COACH rating system either after the coder reviewed the results of a recent ecological assessment or had not. Inter-rater reliability estimates of fidelity ratings were meaningfully higher for the assessment information condition compared to the no-information condition. Importantly, the reliability of the COACH mean score was found to be statistically significantly higher in the information condition. These findings suggest that the reliability of observational ratings of fidelity, particularly when the competence or quality of delivery is considered, could be improved by providing assessment data to the coders. Our findings might be most applicable to assessment-driven interventions, where assessment data explicitly guides therapist’s selection of intervention strategies tailored to the family’s context and needs, but they could also apply to other intervention programs and observational coding of context-dependent therapy processes, such as the working alliance.

KW - Competence

KW - Family check-up

KW - Fidelity

KW - Implementation

KW - Observational coding

UR - http://www.scopus.com/inward/record.url?scp=84952630212&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84952630212&partnerID=8YFLogxK

U2 - 10.1007/s11121-015-0589-0

DO - 10.1007/s11121-015-0589-0

M3 - Article

VL - 17

SP - 62

EP - 70

JO - Prevention Science

JF - Prevention Science

SN - 1389-4986

IS - 1

ER -