Comparison of the RSNA chest CT classification system and CO-RADS system in reporting COVID-19 pneumonia in symptomatic and asymptomatic patients

Abstract

Background

Coronavirus disease (COVID-19) is a new infection with three pandemic waves up till now. CT plays an important role in diagnosis with multiple reporting systems that can be used during CT analysis. We aimed to compare reporting using the recommendations of the radiological society of North America (RSNA) versus the coronavirus disease reporting and data system (CO-RADS) and to assess the performance of CT if used in asymptomatic patients as a screening. Two hundred and fifty-one patients who underwent chest CT scanning either due to clinical suspicion or as screening before hospital admission were included in this retrospective observational cross-sectional study. This was followed by RT-PCR for confirmation. Three radiologists with different years of experience interpreted the CT findings using the RSNA recommendations and the CO-RADS reporting. The data were collected and compared.

Results

There was no statistically significant difference noted in the diagnostic accuracy obtained while using the RSNA recommendations and the CO-RADS reporting system. Also, a good inter-rater agreement was noticed while using the two reporting systems. The CT showed a highly significant value while used in the assessment of symptomatic patients in controversy to the screening of asymptomatic patients.

Conclusion

Both reporting systems show similar diagnostic accuracy with a good almost similar inter-rater agreement. Both can be used while interpreting the CT images of cases with suspected COVID-19 infection. CT can be used effectively in the detection of COVID-19 infection between symptomatic patients while it is of a lower value in the screening of asymptomatic patients.

Background

Coronavirus disease 2019 (COVID-19) infection was first reported in December 2019 in Wuhan, China [1]. The widespread human-to-human transmission led to the declaration of a pandemic by the World Health Organization (WHO) on January 30, 2020 [2].

Although the gold standard for diagnosis of COVID-19 is the reverse transcription-polymerase chain reaction (RT-PCR) assay, its sensitivity ranges from 42 to 83% depending on the viral load, in addition, there is a relative scarcity in some developing countries that occurred during the first wave and the long time it takes for the results to be delivered [3]. With the COVID-19 pandemic, many rapid tests appeared depending on rapid detection of the IgG and IgM within 10 to 30 min providing a rapid decision yet still of much lower diagnostic accuracy compared to the RT-PCR test [4].

This led to the evolution of the role of computed tomography (CT) in many countries as a screening tool for patients with clinically suspected COVID-19 pneumonia [5]. COVID-19 has CT-specific features yet may show some overlap with other diseases especially viral pneumonia [6]

In an attempt to standardize CT reporting in patients with suspected COVID-19, the Radiological Society of North America (RSNA) Expert Consensus Statement was published [7]. In early March 2020, the Dutch Radiological Society developed a standardized reporting system for reporting suspected cases of COVID-19 pneumonia, which was named COVID-19 Reporting and Data System (CO-RADS) [5]. Reporting cases requires the presence of high interobserver agreement to avoid confusion and guide physicians during patient management [8]. This was the hypothesis of this article trying to show the degree of interobserver agreement between different reporting systems that are commonly used during CT interpretation of cases with suspected COVID-19 infection to avoid clinicians’ confusion while using any of them. To our knowledge, limited studies are available comparing the two reporting systems and correlating them to RT-PCR findings, and showing the difference in CT accuracy while used in the screening of asymptomatic patients.

We aimed to compare the current reporting systems for suspected cases of COVID-19 pneumonia (RSNA reporting system and CO-RADS classification) and to correlate them with RT-PCR results. Also, we assessed the degree of interobserver agreement in both reporting systems.

The secondary aim was to assess the performance of CT in the screening of asymptomatic patients.

Methods

Patients

Two hundred and fifty-one patients were referred from the Emergency Room (ER) or the Triage clinic of Ain Shams University Hospital (a tertiary hospital that was allocated to partially receive COVID-19 cases) to the Radiology Department for a non-contrast CT study of the chest to exclude the presence of COVID-19 infection, followed by RT-PCR that had been taken within the same day. They were enrolled in this retrospective observational cross-sectional study. The study was conducted over the period from June 2020–May 2021. The authors received the ethical approval to use the patients’ data and images in the Picture Archiving and Communication System (PACS) from the Faculty of Medicine - Ain Shams Unniversity Ethical Committee, and the patients’ consent was waived being a retrospective study.

Inclusion criteria

Any patient with clinical suspicion of COVID-19 infection, and any asymptomatic patients who were indicated for hospital admission or prepared for any surgical interference who underwent non-contrast CT to screen for COVID-19 infection, followed by RT-PCR.

Exclusion criteria

Patients with non-available RT-PCR results and patients with excessive breathing motion artifacts that affected the accuracy of the CT image interpretation.

Patients’ data

The patients’ records were revised to collect their demographic data including age and sex. Also, the presence of any comorbidities was recorded (Diabetes mellitus DM, hypertension HTN, cardiac disease, chronic kidney disease, chronic liver disease, and chronic respiratory disease). The patients presenting symptoms were recorded from their records to classify them into cases having symptoms of COVID-19 infection and asymptomatic cases. Symptomatic cases were defined as those with any of the following symptoms (Fever, cough, dyspnea, sore throat, malaise, myalgia, anosmia, ageusia, and gastrointestinal symptoms including anorexia, nausea, and diarrhea,) [9]

CT Technique

• The study was performed using the Prime Aquilion 80-slice CT machine (Toshiba, US). The patients were placed in a supine position with their arms elevated above their heads. All patients were instructed to hold their breaths with full inspiration for as long as they could, to minimize the breathing motion artifacts.

• CT parameters included tube voltage of 120 kV, mA ranging between 150 and 400 mA according to the patient’s weight, 1.25 mm slice thickness, 0.625 mm slice interval, 512 × 512 matrix, and tube speed 35 mm/rotation (0.5 s rotation time). In pediatric cases, a low -dose CT technique was used via reducing the Kv and mA, and in some pediatric cases with the motion not controlled, oral sedation was used under the supervision of an anesthesiologist.

• Image processing and interpretation: The images were transferred to the workstation (Fuji Synapse workstation or Paxera-Ultima workstation) for image interpretation by three independent readers. (One radiologist with 14 years of experience, the second with 10 years of experience, while the third was a fellow with only 5 years experience in chest radiology.) All radiologists depend on their observation and experience without using any special algorithms or deep learning systems to measure the inter-rater agreement. Each reader was blinded to the results of RT-PCR, the clinical condition of the patients apart from being a suspicious case of COVID-19 infection, and the interpretation of the other radiologists.

• The following radiological points were recorded to estimate the COVID-19 infection probability:

  • Presence of ground-glass opacity (GGO) with or without septal thickening, crazy paving pattern, consolidation, and fibrosis.

  • The distribution of the GGO within the lung was either peripheral or central, unilateral or bilateral, and uni-lobar or multi-lobar.

  • The presence of a reverse halo sign.

  • The presence of other CT features of pneumonia as lobar consolidation, pulmonary cavity/abscess, and pulmonary nodules with a tree in bud appearance.

• Each CT scan was evaluated according to the RSNA recommendations and CO-RADS reporting system as follows:

  • Reporting the case according to the RSNA recommendations into the typical, indeterminate, atypical, or negative probability for COVID-19 infection [7].

  • Reporting the case according to CO-RADS into CO-RADS 1–5 with CO-RADS 1 was considered negative, while CO-RADS 5 was considered a very high probability case [5].

  • The presence of bilateral multi-lobar predominantly peripheral patchy areas of GGO was considered the most typical form of COVID-19 infection consistent with CO-RADS 5.

RT-PCR assessment

Nasopharyngeal or oropharyngeal swab was done for each patient on the same day of the CT study. The RT-PCR tests were repeated twice only for the cases with negative swabs and persistent clinical suspicion.

Sample size calculation

Using PASS program version 11 for sample size calculation (NCSS, LSS Statistical Software, USA), setting power at 80% and alpha error at 0.05, according to the previous literature the expected AUC for diagnostic validity of CO-RADS for diagnosis of COVID-19 pneumonia is about 0.80, assuming a difference of 0.10 in AUC between CO-RADS and RSNA reporting system for diagnosis of COVID-19 pneumonia, a sample size of at least 150 suspected patients were needed (75 cases with positive RT-PCR and 75 cases with negative RT-PCR).

Statistical methods

Data were analyzed using IBM© SPSS© Statistics version 23 (IBM© Corp., Armonk, NY). Continuous numerical variables are presented as mean and SD and categorical variables as numbers and percentages or ratios.

Inter-rater agreement was examined using the intraclass correlation coefficient (ICC). We used a two-way random model for a single measurement to calculate absolute agreement. The ICC is interpreted as follows: ICC values less than 0.5 are indicative of poor agreement, values between 0.5 and 0.75 indicate moderate agreement, values between 0.75 and 0.9 indicate good agreement, and values greater than 0.90 indicate excellent agreement according to Koo and Li [10]

Diagnostic accuracy for RSNA and CO-RADS classification was calculated for each Rater using RT-PCR as the gold standard for diagnosis. The following diagnostic indices were calculated: sensitivity, specificity, positive and negative predictive values, positive and negative likelihood ratios, and correct classification and misclassification rates.

Results

This was a retrospective study conducted on 251 patients who underwent non-contrast CT and RT-PCR. One hundred and thirty-five patients presented with symptoms of suspicion for COVID-19 infection, while the other 116 patients came for hospital admission or preoperative and performed the RT-PCR as well as the CT as screening to avoid the spread of infection inside the non-isolated areas inside the hospital.

Demographic data

The mean age of the patients was 52 ± 19 years ranging from 5 to 98 years. One hundred and thirty-seven patients were males representing 54.6%. Regarding the patients’ comorbidities, cardiovascular diseases were the most common (157/251) (62.5%) followed by DM (79/251) (31.5%) (Fig. 1).

Fig. 1

Illustrating the incidence of different comorbidities between our study population

The number of patients with positive RT-PCR for COVID-19 was 136 (54.2%) patients. The proportion of patients with positive RT-PCR for COVID-19 in typical, indeterminate, atypical, and negative categories according to the RSNA classification system ranged from 73.6–90.4%, 46.7–50.9%, 22.9–34%, and 38.2–48%, respectively. The proportion of patients with positive RT-PCR for COVID-19 in CORADS 1, 2, 3, 4, and 5 categories ranged from 40–48%, 25.6–30.6%, 42.9–45.2%, 50–79.2%, and 73.6–91.1%, respectively (Table 1).

Table 1 Number of patients assigned to each category in both classifications and percentage of confirmed RT-PCR COVID-19 cases

Regarding the RSNA reporting system

The sensitivity and specificity of the CT using the typical criteria established by RSNA recommendations ranged from 47–49% to 80–94%, respectively, for the three raters. The sensitivity and specificity of the typical and indeterminate categories together ranged from 65–68% to 52–71%, respectively, for the three raters. The positive predictive value while using the typical criteria ranged from 81 to 90%, while the negative predictive value ranged from 58 to 61%. The positive predictive value for typical and indeterminate groups ranged from 63 to 73%, while the negative predictive value ranged from 58 to 64%. (Table 2). Reporting using the RSNA reporting system showed good inter-rater agreement with ICC = 0.77 (Table 3) (Figs. 2, 3, 4, 5).

Table 2 Accuracy of RSNA different classifications for diagnosis of COVID-19 as calculated for each of the three raters

Table 3 Inter-rater agreement for RSNA classification

Fig. 2

A and B High-resolution CT scan (axial cuts) for a 60 year-old diabetic male patient presented with fever and cough. CT shows multiple bilateral peripheral patches of ground-glass opacities in both lung lobes with linear consolidations seen in both lower lung lobes interpreted by all readers as a high probability (CO-RADS 5). RT-PCR was positive for COVID-19 and the patient responded to treatment and was discharged 14 days later

Fig. 3

A High-resolution CT scan (axial cuts) for a 55-year-old male patient with recently diagnosed chronic myeloid leukemia presented with fever. The initial CT was interpreted as atypical for COVID (CO-RADS 2) with two patches of subsegmental consolidation in the left lobe and lingual. RT-PCR was positive for COVID-19. B CT scan was done nine days later after the patient developed desaturation shows a picture of non-cardiogenic pulmonary edema (ARDS), and the patient died a few days later

Fig. 4

A and B High-resolution CT chest (axial cuts) for a 74-year-old diabetic hypertensive male patient presented with fever, cough, and dyspnea. CT was interpreted as typical (CO-RADS 5) by all readers. RT-PCR for COVID-19 was negative twice. The patient recovered from antibiotic treatment and was discharged a few days later

Fig. 5

A and B High-resolution CT chest (axial cuts) for a 25-year-old male patient with common variable immune deficiency presented with fever, cough, and dyspnea. CT was interpreted as typical (CO-RADS 5) by all readers. RT-PCR for COVID-19 was positive. The patient improved and was discharged in two weeks

Regarding the CO-RADS reporting system

The sensitivity and specificity of the CO-RADS 5 category ranged from 38–49% to 80–96% respectively for the three raters. The sensitivity and specificity of CO-RADS 4 and 5 categories together ranged from 51–62% to 66–91% respectively for the three raters. The sensitivity and specificity of CO-RADS 3, 4, and 5 categories together ranged from 65–68% to 52–71% respectively for the three raters. The positive predictive value for CO-RADS 5 group ranged from 74 to 91%, while the negative predictive value ranged from 56 to 57%. The positive predictive value for CO-RADS 4 and 5 groups ranged from 72 to 88%, while the negative predictive value ranged from 58 to 61%. The positive predictive value for CO-RADS 3, 4 and 5 groups ranged from 63 to 73%, while the negative predictive value ranged from 58 to 64%. (Table 4). Reporting using the CO-RADS reporting system showed good inter-rater agreement with ICC = 0.75 which was close and similar to the RSNA reporting system (Table 5) (Figs. 2, 3, 4, 5).

Table 4 Accuracy of different CO-RADS classes for diagnosis of COVID-19 as calculated for each of the three raters

Table 5 Inter-rater agreement for CO-RADS classification

The diagnostic accuracy of the two reporting systems, as well as the inter-rater agreement, were almost similar (Tables 2 and 4).

Regarding the use of CT in screening asymptomatic patients

For simplicity and with the base of good inter-rater agreement found between the three raters in the current study, we used the reading of rater 1 who is the more experienced in the analysis of the results among symptomatic and asymptomatic cases. Among 43 asymptomatic cases with positive RT-PCR test for COVID-19 infection, the CT detected changes of high to indeterminate probability for COVID-19 infection in only 17 cases (39.5%). While in 93 symptomatic cases with positive RT-PCR, the CT detected changes of high to indeterminate probability for COVID-19 infection in 72 cases (77.4%). The CT showed a highly significant P-value of 0.000 when used among clinically suspected patients in controversy to a non-significant P-value of 0.95/0.96 when used as screening in asymptomatic patients (Table 6). Negative CT and indeterminate probability were the highest interpretation detected among asymptomatic patients during the use of the RSNA reporting system, while in the CO-RADS reporting system, CO-RADS 1 and 3 were the commonest.

Table 6 Number of patients assigned to each category in both classifications according to presence or absence of symptoms as well as the confirmed RT-PCR

Discussion

In this study, the authors tried to evaluate the inter-observer agreement while reporting the CT using RSNA and CO-RADS reporting systems and showed good inter-observer agreement even with the difference in the radiologists’ experiences. Also, almost similar diagnostic accuracy was found while using the two reporting systems. CT showed non-significant values in the detection of COVID-19 infection while used in the screening of asymptomatic patients.

The American College of Radiology and the Royal College of Radiologists recommended that CT should not be used to screen patients with suspected COVID-19 infection [11, 12]. However, chest CT was used in our hospital as a method of triaging patients with suspected COVID-19 infection, as well as to exclude infection in patients considered for hospitalization for other reasons, especially with the scarcity and time-consuming nature of the RT-PCR test and if there was any clinical suspicion of COVID-19 infection [13]. The RSNA chest CT classification for reporting COVID 19 pneumonia provides a method for conveying to the clinicians the level of suspicion of COVID-19 [7]. In March 2020, the Dutch Radiological Society developed the CO-RADS reporting system to classify pulmonary involvement by COVID-19 [5].

In this study, we found the proportion of patients with RT-PCR confirmed COVID-19 was highest in the typical followed by the indeterminate category of the RSNA system. The proportion of RT-PCR confirmed cases in the negative category (38.2–48%) was higher than that in the atypical category (22.9–34%). This is concurrent with De Jaegere et al. [8] who explained this by the higher prevalence of other lung diseases in the atypical category (e.g., bacterial pneumonia and pulmonary edema) compared to the negative category, which was also the case in our study. Regarding the CO-RADS system, similar proportions were reported for RT-PCR confirmed cases with the CO-RADS 1 category showing a higher proportion of positive cases (40–48%) than the CO-RADS 2 category (25.6–30.6%). This was also in agreement with De Jaegere et al. [8]

Reporting using the RSNA reporting system showed good inter-rater agreement with ICC = 0.77. This is slightly lower than Byrne et al. [14] who reported substantial agreement in the indeterminate category and almost perfect agreement in the other three categories. De Jaegere et al. [8] reported substantial to a moderate overall agreement. An Italian study by Ciccarese et al. [15] reported a moderate interobserver agreement.

We found that reporting using the CO-RADS reporting system showed good inter-rater agreement with ICC = 0.75 which is slightly higher than Prokop et al. [5] who reported moderate reliability with a k value of 0.47. Fujioka et al. [16] reported substantial to an almost perfect interobserver agreement in the CO-RADS system with ICC 0.800–0.874 which is considered slightly higher than that in our study.

The sensitivity of the typical category in the RSNA reporting system ranged from 47 to 49% for the three readers. When using the indeterminate category as the threshold, the sensitivity for typical and indeterminate categories increased to 65–68%. The sensitivity in our study is remarkably less than that reported by Som et al. [17] who reported a sensitivity of typical and indeterminate findings to be 97.5% (range 94–100%). This could likely be explained by the use of CT in our hospital for symptomatic patients suspected to be infected by COVID-19 as well as a screening tool for patients considered for hospitalization for other reasons, while in their study, CT was used as a problem-solving tool in suspected cases and to guide management, which implied that all patients were symptomatic at the time of imaging, which was not the case in our study. Our findings concur with Kim et al. [18] who reported that CT may be normal in asymptomatic and early cases of COVID-19 infection. A meta-analysis done by Kwee et al. [19] reported a pooled sensitivity of 65.2% for the typical category and 90.2% for typical and indeterminate categories, also higher than the values in the current study.

The sensitivity of the CO-RADS 5 category ranged from 38 to 49%. When using CO-RADS 4 and 5 categories as a threshold, the sensitivity ranged from 51 to 62%. The sensitivity of CO-RADS 3, 4, and 5 categories together increased to 65–68% in our study. Our values were lower than those by Kwee et al. [19] who reported a pooled sensitivity of CO-RADS-5 category 70.4%, for CO-RADS 4 and 5 categories increased to 85.8%, and for CO-RADS 3, 4, and 5 categories were reaching 92.5%.

Regarding the specificity of the typical category in our study, it ranged from 80–94% and decreased to 52–71% for the typical and indeterminate categories, concurrent with Som et al. [17] who reported a specificity of 54.7% for the latter categories. The high specificity of the typical category was higher than that reported in a large study in China by Ai et al. [20], who reported a specificity of 25% for chest CT. This was explained in their study by the possibility of RT-PCR being falsely negative in many cases. In these patients, they performed a further analysis based on exposure history, clinical manifestations, and serial CT scans, to better classify patients with positive CT findings and negative RT-PCR tests. Similarly, we reported the specificity of CO-RADS 5 category to range from 80 to 96%, CO-RADS 4 and 5 ranged from 66 to 91% and decreased to 52–71% for CO-RADS 3, 4 and 5 categories, concurrent with the literature stating that at high diagnostic thresholds, specificity increases at the cost of sensitivity [19]

Finally, our results concerning the non-significant value of CT while used in the screening of asymptomatic patients are consistent with the ACR recommendations [11, 12]. Smet et al. [21] found that CO-RADS 3 was the most commonest among positive asymptomatic cases and concluded CT screening for asymptomatic patients is not recommended which is similar to the current study.

Limitations

Our study had several limitations. First the use of RT-PCR as a gold standard for COVID-19 infection with its reported false-negative rate that could reach 63% in nasopharyngeal swabs [22]. Second, RT-PCR was not repeated in all negative cases, yet in patients with persistent clinical suspicion of COVID-19, at least two RT-PCR tests were done. The study was also limited by its retrospective design and the lack of clinical data regarding the duration of symptoms at the time of CT scanning. Among the strengths of the study was the inclusion of symptomatic and asymptomatic cases rendering it more valuable in the assessment of the diagnostic performance of CT without selection bias.

Conclusion

There is a good inter-rater agreement while reporting using either the RSNA recommendations or the CO-RADS reporting systems among radiologists with different levels of experience, rendering them useful in providing standardized reports to convey the level of suspicion of COVID-19 infection to the clinicians. Despite the high specificity of CT in the diagnosis of symptomatic COVID-19, especially with high diagnostic thresholds, our study concluded that its sensitivity is lacking owing to the considerable number of confirmed COVID-19 cases in the negative and atypical categories, limiting its use as a screening test in asymptomatic cases in controversy to what happened while used in symptomatic patients. So, it is not suggested to use CT as screening for COVID-19 infection in asymptomatic patients.