TY - JOUR
T1 - How Fair Is My Test?
T2 - A Ratio Coefficient to Help Represent Consequential Validity
AU - Dumas, Denis
AU - Dong, Yixiao
AU - McNeish, Daniel
N1 - Publisher Copyright:
© 2022 Hogrefe Publishing.
PY - 2023/11
Y1 - 2023/11
N2 - The degree to which test scores can support justified and fair decisions about demographically diverse participants has been an important aspect of educational and psychological testing for millennia. In the last 30 years, this aspect of measurement has come to be known as consequential validity, and it has sparked scholarly debate as to how responsible psychometricians should be for the fairness of the tests they create and how the field might be able to quantify that fairness and communicate it to applied researchers and other stakeholders of testing programs. Here, we formulate a relatively simple-to-calculate ratio coefficient that is meant to capture how well the scores from a given test can predict a criterion free from the undue influence of student demographics. We posit three example calculations of this Consequential Validity Ratio (CVR): one where the CVR is quite strong, another where the CVR is more moderate, and a third where the CVR is weak. We provide preliminary suggestions for interpreting the CVR and discuss its utility in instances where new tests are being developed, tests are being adapted to a new population, or the fairness of an established test has become an empirical question.
AB - The degree to which test scores can support justified and fair decisions about demographically diverse participants has been an important aspect of educational and psychological testing for millennia. In the last 30 years, this aspect of measurement has come to be known as consequential validity, and it has sparked scholarly debate as to how responsible psychometricians should be for the fairness of the tests they create and how the field might be able to quantify that fairness and communicate it to applied researchers and other stakeholders of testing programs. Here, we formulate a relatively simple-to-calculate ratio coefficient that is meant to capture how well the scores from a given test can predict a criterion free from the undue influence of student demographics. We posit three example calculations of this Consequential Validity Ratio (CVR): one where the CVR is quite strong, another where the CVR is more moderate, and a third where the CVR is weak. We provide preliminary suggestions for interpreting the CVR and discuss its utility in instances where new tests are being developed, tests are being adapted to a new population, or the fairness of an established test has become an empirical question.
KW - consequential validity
KW - educational measurement
KW - psychometrics
KW - testing fairness
UR - http://www.scopus.com/inward/record.url?scp=85136574301&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85136574301&partnerID=8YFLogxK
U2 - 10.1027/1015-5759/a000724
DO - 10.1027/1015-5759/a000724
M3 - Article
AN - SCOPUS:85136574301
SN - 1015-5759
VL - 39
SP - 416
EP - 423
JO - European Journal of Psychological Assessment
JF - European Journal of Psychological Assessment
IS - 6
ER -