Sources of variation and bias in studies of diagnostic accuracy: a systematic review

BACKGROUND: Studies of diagnostic accuracy are subject to different sources of
bias and variation than studies that evaluate the effectiveness of an
intervention. Little is known about the effects of these sources of bias and
variation.
PURPOSE: To summarize the evidence on factors that can lead to bias or variation
in the results of diagnostic accuracy studies.
DATA SOURCES: MEDLINE, EMBASE, and BIOSIS, and the methodologic databases of the
Centre for Reviews and Dissemination and the Cochrane Collaboration. Methodologic
experts in diagnostic tests were contacted.
STUDY SELECTION: Studies that investigated the effects of bias and variation on
measures of test performance were eligible for inclusion, which was assessed by
one reviewer and checked by a second reviewer. Discrepancies were resolved
through discussion.
DATA EXTRACTION: Data extraction was conducted by one reviewer and checked by a
second reviewer.
DATA SYNTHESIS: The best-documented effects of bias and variation were found for
demographic features, disease prevalence and severity, partial verification bias,
clinical review bias, and observer and instrument variation. For other sources,
such as distorted selection of participants, absent or inappropriate reference
standard, differential verification bias, and review bias, the amount of evidence
was limited. Evidence was lacking for other features, including incorporation
bias, treatment paradox, arbitrary choice of threshold value, and dropouts.
CONCLUSIONS: Many issues in the design and conduct of diagnostic accuracy studies
can lead to bias or variation; however, the empirical evidence about the size and
effect of these issues is limited.