Inter-observer agreement and accuracy in classifying radiographs for pneumoconiosis among Asian physicians taking AIR Pneumo certification examination

This study examined inter-observer agreement and diagnostic accuracy in classifying radiographs for pneumoconiosis among Asian physicians taking the AIR Pneumo examination. We compared agreement and diagnostic accuracy for parenchymal and pleural lesions across residing countries, specialty training, and work experience using data on 93 physicians. Physicians demonstrated fair to good agreement with kappa values 0.30 (95% CI: 0.20-0.40), 0.29 (95% CI: 0.23-0.36), 0.59 (95% CI: 0.52-0.67), and 0.65 (95% CI: 0.55-0.74) in classifying pleural plaques, small opacity shapes, small opacity profusion, and large opacities, respectively. Kappa values among Asian countries ranging from 0.25 to 0.55 (pleural plaques), 0.47 to 0.73 (small opacity profusion), and 0.55 to 0.69 (large opacity size). The median Youden's J index (interquartile range) for classifying pleural plaque, small opacity, and large opacity was 61.1 (25.5), 76.8 (29.3), and 88.9 (23.3), respectively. Radiologists and recent graduates showed superior performance than other groups regarding agreement and accuracy in classifying all types of lesions. In conclusion, Asian physicians taking the AIR Pneumo examination were better at classifying parenchymal lesions than pleural plaques using the ILO classification. The degree of agreement and accuracy was different among countries and was associated with background specialty training.