Grading of Recommendations Assessment, Development, and Evaluation and Confidence in Network Meta-Analysis showed moderate to substantial concordance in the evaluation of certainty of the evidence

Objectives: Grading of Recommendations Assessment, Development, and Evaluation (GRADE) and Confidence in Network Meta-Analysis (CINeMA) are available to assess the confidence in network meta-analysis (NMA) results. They share common aspects, but their operationalization differs. We evaluated interrater reliability (IRR) among assessors, the approaches' concordance, and application time. Methods: Two dichotomous (“seizure response”, “ischemic stroke”) and two continuous (“change in attention deficit hyperactivity disorder (ADHD)”, “weight loss”) outcomes with networks of different sizes, structures and complexities were chosen from four NMAs. Thirteen assessors were randomly assigned to four groups to apply both GRADE and CINeMA on one continuous and one dichotomous outcome. We measured IRR and concordance using Gwet's AC on the overall evaluation and each tool's domain. We calculated time spent evaluating each network and tool, including time to consult the guidance papers. Results: IRR ranged from 0.49 (“seizure response”) to 0.70 (“ischemic stroke”) with GRADE, from 0.02 (“ischemic stroke”) to 0.73 (“change in ADHD symptoms”) with CINeMA. Overall concordance was 1, 0.90, 0.68, and 0.42 for “seizure response”, “ADHD symptoms”, “weight loss”, and “ischemic stroke”, respectively. The median time spent to assess each network ranged from 160 to 481 minutes with GRADE, from 150 to 330 minutes with CINeMA. Conclusion: IRR was moderate to substantial with both approaches, except for CINeMA when applied to the largest and most complex network (“ischemic stroke”). Concordance was good for small networks with no or few indirect comparisons (“seizure response”) and decreased as the number of comparisons and indirect evidence increased. Application time was long with both approaches, particularly with GRADE for large networks.