Two recent articles have evaluated the measurement properties of the PEDro scale. This first article assessed the agreement between the PEDro scale and the Cochrane risk of bias tool, and is summarised in this post.
Critical appraisal of trial risk of bias (methodological quality) is one of the five steps of evidence-based practice. Two instruments that are commonly used to assess the risk of bias of trials of physiotherapy interventions are the Cochrane risk of bias (CROB) tool and the PEDro scale. While both instruments have different approaches to assessing risk of bias, they have six items in common (random allocation; concealed allocation; blinding of participants, personnel and assessors; and incomplete outcome data). To date, only one study has made a direct comparison between the PEDro scale and CROB tool in trials of physiotherapy interventions, concluding that there was poor agreement between the instruments. The primary objective of this study was to determine the convergent validity (level of agreement) between individual items from the PEDro scale and CROB tool that evaluate similar constructs and for summary scores. The secondary objective was to determine the level of agreement between different thresholds for “acceptable” risk of bias between the summary scores for the CROB tool and PEDro scale.
The Cochrane Library was searched to identify trials included in systematic reviews evaluating physiotherapy interventions. For trials that met the inclusion criteria (primary reference in Cochrane review, review used 2008 version of the CROB tool, and indexed in PEDro), CROB items were extracted from the reviews and PEDro items and total score were downloaded from PEDro. Kappa statistics were used to evaluate agreement between CROB and PEDro items that evaluate similar constructs. The total PEDro score was compared to the CROB summary score (% of items met) using an Intraclass Correlation Coefficient. Sensitivity analyses were used to explore the impact of the CROB “unclear” category and variants of CROB blinding items on agreement. Kappa statistics were used to determine agreement between different thresholds for “acceptable” risk of bias between CROB and PEDro scale summary scores.
1442 trials from 108 Cochrane reviews were included in the analyses. Agreement was “moderate” for three of the six CROB and PEDro scale items that evaluate similar constructs (allocation concealment, participant blinding, assessor blinding; Kappa = 0.479–0.582). Agreement between the summary scores was “poor” (Intraclass Correlation Coefficient = 0.285). Agreement was highest when the “unclear” CROB category was collapsed with “high” and when CROB participant, personnel and assessor blinding were evaluated separately. Agreement for different thresholds for “acceptable” risk of bias between instruments was, at best, “fair”.
There was moderate agreement for half of the PEDro scale and CROB items that evaluate similar constructs. Interpretation of the CROB “unclear” category and variants of the CROB blinding items substantially influenced agreement. The authors caution against the use of thresholds for “acceptable” risk of bias for both the CROB tool and PEDro scale. Either instrument can be used to quantify risk of bias, but they can’t be used interchangeably.
