Purpose of the Study: To test a new method of quantifying the degree of interobserver variation when evaluating signs of airways and interstitial lung disease (ILD) on HRCT. Methods: HRCT scans of two groups of 143 and 109 patients with proven airways and ILD respectively, were evaluated by two chest radiologists. Bronchial wall thickness (BWT) was visually graded according to a four-point scale, whereas the extent of the mosaic attenuation (MA) pattern and ILD were evaluated as a percentage of the lobar volume. The new method of agreement was tested by the same observers at one year following their original (standard) scoring. The new system entailed ‘‘continuous learning’’ with the observers rescoring cases and reaching a joint opinion whenever their score differences were higher than predetermined thresholds (>1 for BWT and ≥20% for both MA and ILD). Weighted kappas (kw) and single-determination standard deviation (SD) were calculated and compared between standard and new method systems. Correlations between the pulmonary function test results and the observers’ scores obtained using the standard and the new systems were compared. Results: Observer agreement for the standard and new systems showed improved agreement for the new system: BWT kw increased from 0.51 to 0.76, for MA kw increased from 0.34 to 0.81 and for ILD extent SD decreased from 7.0 to 4.1. When analyzing kw in succeeding groups of 20 observations with the new system we observed a straightforward improvement between the first group and the succeeding ones, although such incremental training effect was not observed in the ILD extent evaluations. The new scoring system did not significantly improve correlations between HRCT features and pulmonary function tests. Conclusions: The new scoring system reduces noise from observer variation and could, with benefit, be applied to scientific investigations which require quantification of HRCT signs of pulmonary disease.
A Method for Optimizing Observer Agreement for HRCT Signs of Airways and Interstitial Lung Disease / Sverzellati, Nicola; A., Devaraj; S. R., Desaiz; M., Quigley; A. U., Wellsy; D. M., Hansell. - In: JOURNAL OF THORACIC IMAGING. - ISSN 0883-5993. - 24:3(2009).
A Method for Optimizing Observer Agreement for HRCT Signs of Airways and Interstitial Lung Disease
SVERZELLATI, Nicola;
2009-01-01
Abstract
Purpose of the Study: To test a new method of quantifying the degree of interobserver variation when evaluating signs of airways and interstitial lung disease (ILD) on HRCT. Methods: HRCT scans of two groups of 143 and 109 patients with proven airways and ILD respectively, were evaluated by two chest radiologists. Bronchial wall thickness (BWT) was visually graded according to a four-point scale, whereas the extent of the mosaic attenuation (MA) pattern and ILD were evaluated as a percentage of the lobar volume. The new method of agreement was tested by the same observers at one year following their original (standard) scoring. The new system entailed ‘‘continuous learning’’ with the observers rescoring cases and reaching a joint opinion whenever their score differences were higher than predetermined thresholds (>1 for BWT and ≥20% for both MA and ILD). Weighted kappas (kw) and single-determination standard deviation (SD) were calculated and compared between standard and new method systems. Correlations between the pulmonary function test results and the observers’ scores obtained using the standard and the new systems were compared. Results: Observer agreement for the standard and new systems showed improved agreement for the new system: BWT kw increased from 0.51 to 0.76, for MA kw increased from 0.34 to 0.81 and for ILD extent SD decreased from 7.0 to 4.1. When analyzing kw in succeeding groups of 20 observations with the new system we observed a straightforward improvement between the first group and the succeeding ones, although such incremental training effect was not observed in the ILD extent evaluations. The new scoring system did not significantly improve correlations between HRCT features and pulmonary function tests. Conclusions: The new scoring system reduces noise from observer variation and could, with benefit, be applied to scientific investigations which require quantification of HRCT signs of pulmonary disease.I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
