The inter-rater reliability and individual reviewer performance of the 2012 world heart federation guidelines for the echocardiographic diagnosis of latent rheumatic heart disease

Background: In 2012, the World Heart Federation (WHF) published guidelines for the echocardiographic diagnosis of rheumatic heart disease (RHD). This study assesses individual reviewer performance and inter-rater agreement and reliability on the presence of any RHD, as well classification of RHD based on the 2012 WHF criteria. Methods: Four cardiologists individually reviewed echocardiograms in the context of a randomized clinical trial (ClinicalTrials.gov:NCT03346525) and participated in a blinded adjudication panel. Panel decision was the reference standard for diagnosis. Performance of individual reviewers to panel adjudication was compared through sensitivity and specificity analyses and inter-rater reliability was assessed between individual panelists using Fleiss free marginal multirater kappa. Results: Echocardiograms from 784 children had two independent reports and panel adjudication. The accuracy of independent reviewers for any RHD had high sensitivity (94%, 95% CI 93–95%) and moderate specificity (62%, 95% CI 53–70%). Sensitivity and specificity for definite RHD was 61.3 (95% CI, 55.3–67.1) and 93.1 (95% CI, 91.6–94.4), with 86.8 (84.7–88.7) and 65.8 (61.0–70.4) for borderline RHD. There was moderate inter-rater agreement (κ = 0.66) on the presence of any RHD while agreement for specific 2012 WHF classification was only fair (κ = 0.51). Conclusions: The 2012 WHF guidelines are moderately reproducible when used by expert cardiologists. More cases of RHD were diagnosed by an consensus panel than by individual reviewers. A revision to the criteria is now warranted to further increase the reliability of the WHF criteria.