Quantification of pulmonary opacities using artificial intelligence in chest CT scans during SARS-CoV-2 pandemic: validation and prognostic assessment

Abstract

Purpose

To assess whether the analysis of pulmonary opacities on chest CT scans by AI-RAD Companion, an artificial intelligence (AI) software, has any prognostic value.

Background

In December 2019, a new coronavirus named SARS-CoV-2 emerged in Wuhan, China, causing a global pandemic known as COVID-19. The disease initially presents with flu-like symptoms but can progress to severe respiratory distress, organ failure, and high mortality rates. The overwhelming influx of patients strained Emergency Rooms worldwide. To assist in diagnosing and categorizing pneumonia, AI algorithms using deep learning and convolutional neural networks were introduced. However, there is limited research on how applicable these algorithms are in the Emergency Room setting, and their practicality remains uncertain due to most studies focusing on COVID-19-positive patients only.

Methods

Our study has an observational, analytical, and longitudinal design. The sample consisted of patients who visited our emergency room from August 5, 2021, to September 9, 2021, were suspected of having COVID-19 pneumonia, and underwent a chest CT scan. They were categorized into COVID-19 negative and positive groups based on PCR confirmation. Lung opacities were evaluated separately by a team of radiologists and a commercial AI software called AI-Rad Companion (by Siemens Healthineers). After 5 months we gathered clinical data, such as hospital admission, intensive care unit (ICU) admission, death, and hospital stay.

Results

The final sample included 304 patients (144 females, 160 males) with a mean age of 68 ± 19 std. Among them, 129 tested negative for COVID-19 and 175 tested positive. We used AI-generated opacity quantification, compared to radiologists' reports, to create receiver operating characteristic curves. The area under the curve ranged from 0.8 to 0.9 with a 95% confidence interval. We then adjusted opacity tests to a sensitivity cut-off of 95%. We found a significant association between these opacity tests and hospital admission and ICU admission (Chi-Squared, P < 0.05), as well as between the percentage of lung opacities and length of hospital stay (Spearman's rho 0.53–0.54, P < 0.05) in both groups.

Conclusions

During the SARS-CoV-2 pandemic, AI-based opacity tests demonstrated an association with certain prognostic markers in patients with suspected COVID-19 pneumonia, regardless of whether a PCR-confirmed coronavirus infection was ultimately detected.

Background

A newly identified coronavirus called SARS-CoV-2 was first reported in Wuhan, China in December 2019. It caused a worldwide pandemic of respiratory illness, called COVID-19 [1]. The disease frequently starts with flu-like symptoms such as fever, dry cough, or fatigue and can lead to acute respiratory distress syndrome, organ failure, and intensive care unit (ICU) admission with subsequent high mortality rates. The outbreak of the pandemic resulted in an overload of Emergency Rooms all over the world. Consequently, artificial intelligence (AI) algorithms based on deep learning [2] and convolutional neural networks [3] (CNN) began to be used both in the diagnosis of COVID-19 pneumonia and in the classification of other pneumonia and non-pathological findings. However, studies on the predictive value of these algorithms applied to the field of emergency radiology are scarce, and their future utility is uncertain.

Deep learning is a machine learning [4] technique that utilizes neural networks [5] to learn patterns from input data, allowing computers to make informed conclusions. By using extensive databases and training experiences, computers can improve their performance in specific tasks. Convolutional neural networks (CNNs) are a popular type of deep learning architecture, particularly in medical imaging, as they are effective in extracting and classifying patterns [6]. CNNs analyze input images, assigning significance to different features and distinguishing between them.

In the diagnosis of COVID-19 pneumonia by CT, segmentation models based on CNNs such as U-Net [7], V-Net [8], and 3D U-Net++ [9] have been widely used. For example, Ying et al. [10] proposed DeepPneumonia, based on the ResNet-50 [11] system for CT studies to distinguish COVID-19 pneumonia from bacterial pneumonia and healthy patients. On the other hand, Shi et al. [12] applied VB-Net [13] to segment CT images and then used their own CNN model to diagnose COVID-19 pneumonia.

We found a lack of extensive research on the predictive value of artificial intelligence systems when it comes to analyzing pulmonary opacities. The existing studies primarily concentrate on the prognostic usefulness of these systems for patients specifically diagnosed with COVID-19 pneumonia. Notable studies by Zakariaee et al. [14], Gouda W et al. [15], and Mader et al. [16] explored this area and revealed an association between AI-based chest CT opacity quantification and some prognostic markers in these patients.

To fill this knowledge gap, we used an online AI-powered platform to quantify pulmonary opacities in chest CT scans of patients suspected to have COVID-19 pneumonia, irrespective of their subsequent negative PCR test results. We hypothesized that these algorithms could provide valuable prognostic predictions in the field of Emergency radiology. The primary aim of our study was to investigate the prognostic implications of AI quantification in both COVID-19-positive and negative patients. Secondarily, we sought to assess the correlation between AI-based opacity quantification and radiological reports.

Methods

We conducted an observational, analytical, and longitudinal single-center study. The initial sample consisted of patients who consecutively visited our tertiary referral hospital's Emergency Room between August 5, 2021, and September 9, 2021. This sample consisted of patients who were suspected to have COVID-19 pneumonia and underwent a non-contrasted chest CT scan. Lung opacities were assessed independently by both a radiologist from a team of emergency radiologists and a commercial AI software known as AI-Rad Companion. We followed up with these patients for a period of 5 months to observe any negative outcomes. The data collection was carried out in January 2022. To evaluate their prognosis, we divided the patients into two groups based on their COVID-19 status, which was determined through PCR testing. The study was conducted during the SARS-CoV-2 pandemic and received approval from the hospital's ethical committee. Since the study did not involve any interventionist design, the committee deemed obtaining informed consent unnecessary.

Inclusion criteria

• Ambulatory patients aged 16 years or older suspected to have COVID-19 pneumonia in the adult emergency area of our tertiary referral hospital.

• Patients had to undergo their first chest CT scan for this specific reason and should not have been diagnosed with COVID-19 pneumonia previously. This aimed to capture early-stage diagnoses and minimize potential biases.

• Symptoms and laboratory findings consistent with the existing literature were required for eligibility, which included shortness of breath, fatigue, cough, and fever while laboratory findings included elevated C-reactive protein, lymphopenia, and elevated lactate dehydrogenase [17]. This ensured that the sample reflected the expected characteristics of COVID-19 pneumonia cases.

Exclusion criteria

• Cases where the AI platform failed to provide necessary variables due to failed segmentation (e.g., previous lobectomy, motion artifacts, severe pleural effusion). This ensured the reliability of the data obtained from the AI platform by excluding cases where the algorithm might not perform optimally.

• Patients with interstitial lung disease or thoracic tumors impacting the lungs. This maintained the study's focus on COVID-19 pneumonia cases.

• Patients recently hospitalized or with a history of hospitalization within the last month for any reason. This aimed to minimize confounding factors and isolate the effects of COVID-19 pneumonia in the study.

Chest CT scan analysis

For each patient, as soon as a non-contrasted chest CT scan was performed and CT images were available, chest data sets were anonymized and sent by our system to an online AI-powered platform provided by Siemens Healthineers [18] called Teamplay© [19]. Through personal log-in on this platform, we had access to the results of AI-Rad Companion, version VA12A [20], a commercial AI software that processes images from non-contrasted chest CT scans and generates quantitative outputs. The authors did not participate in the development nor testing of any of this technology.

On the other hand, each radiology report was independently created by an experienced emergency radiologist from a group of 23 individuals with at least 8 years of experience. Importantly, these radiologists were blinded to the results generated by AI-Rad Companion, ensuring an unbiased assessment. Additionally, the researchers responsible for retrieving the output from the AI platform were also blind to the radiological reports and laboratory findings.

Variables of the study

We codified radiological variables from the radiological reports of these patients. We also extracted the parameters provided by AI-RAD Companion, expressed in tables for each anonymized chest CT scan. Clinical variables such as hospital stay, hospital admission, ICU admission, and death were obtained from medical records (Table 1).

Table 1 Variables of the study

Chest CT protocol

Chest CT scans were conducted using STOMATOM.go.Up CT model manufactured by Siemens Healthineers. This CT model has 64 detectors, a power of 32 kW, a voltage of up to 130 kV, and a maximum mA of 400. The z-coverage of the CT scans was 32 × 0.7 mm. A slice thickness of 1.5 mm was employed for the chest CT scans.

AI algorithm

The algorithm used by AI-Rad Companion, along with its training and testing datasets, were described by Chaganti et al. [21]. Here is a summary of the algorithm's details:

the AI algorithm begins by generating lung lobe segmentation masks based on chest CT data. It utilizes an advanced deep reinforcement learning algorithm [22] to identify important anatomical landmarks such as the carina bifurcation and sternum tip, which helps determine the region of interest (ROI) for the lungs. To achieve precise lung segmentation, the lung ROI image is resampled to a uniform 2 mm volume and then processed using an adversarial Deep Image-to-Image Network (DI2IN) [23]. This network has been specifically designed to handle lung segmentation tasks. The resulting segmentation mask for the lung ROI is adjusted to match the dimensions and resolution of the input data. The DI2IN was trained using a diverse dataset consisting of over 8000 CT scans from patients in Europe, the USA, and Canada, covering a wide range of diseases. Additionally, the network was fine-tuned using 1000 abnormal patterns, including cases of interstitial lung disease, non-COVID-19 pneumonia, and COVID-19 pneumonia.

The detection and quantification of opaque regions were performed using the DenseUNet convolutional neural network [24]. This algorithm was trained with 900 CT scans from patients with interstitial pneumonia, non-COVID-19 pneumonia, and COVID-19 pneumonia. The algorithm identifies low-opacity regions that resemble ground glass opacities upon visual inspection. Subsequently, a -200 UH cut-off is applied to these regions to obtain high-opacity regions that visually resemble consolidations. However, we could not find a specific rationale for this threshold in the existing literature (Figs. 1, 2, 3).

Fig. 1

Visual output by AI-Rad Companion (1). AI-Rad Companion generates a visual representation that shows calculated and outlined lung opacities found in a dataset of chest CT scans. The output also includes a 3D reconstruction. The opacities are presented in red for low opacities and in fuchsia for high opacities

Fig. 2

Visual output generated by AI-Rad Companion (2). The image represents low opacities as red and certain vessels as fuchsia, as the AI interprets them as high opacities

Fig. 3

Quantitative analysis by AI-Rad Companion. The image shows the complete quantitative output provided by the AI software. Our study specifically focuses on the percentage of low and high opacities in the left lung, right lung, and the combined assessment of both lungs

Study size and potential biases

The study size was limited by resource constraints, specifically the duration of the AI-Rad Companion license and the absence of IT support. These factors influenced the overall scope and duration of the study. As for potential biases, we were unable to assess the impact of vaccination on the outcomes of patients with COVID-19 pneumonia because a significant number of patients did not have complete data regarding the type and number of vaccines they received in their medical records. On the other hand, relying on individual observations for each radiology report and elevating them to gold standard is a potential bias we address in the discussion section.

Statistical analysis

We conducted the statistical analysis using IBM© SPSS© Statistics version 26.0.0.0 (64-bit), owned by IBM Corp.© and was run on the Windows 11© operating system.

All continuous variables were analyzed with the Kolmogorov–Smirnov test to determine their probability of fitting the normal distribution.

We constructed ROC curves to elaborate opacity tests based on a sensitivity cut-off point of 95%, which we considered acceptable at the moment. The AI-Rad Companion's low-opacity percentage (LOP) was used as a predictive condition, with the presence of ground glass opacity from the radiology report (yes, no) as the ground truth. Similarly, the high-opacity percentage (HOP) was used as a predictive condition, with the presence of consolidation (yes, no) as the ground truth. We built separate ROC curves for the right lung, the left lung, and both lungs combined.

Chi-square tests were conducted to establish the statistical association between these opacity tests (LOP and HOP tests) and hospital admission, ICU admission, and death. Spearman correlation was employed as a nonparametric measure to assess the strength and direction of the association between the percentage of each opacity type (low and high opacities) and the length of hospital stay (in days).

Results

More than 500 chest CT scans were performed in the emergency area during the recruitment period, but only 345 patients met the inclusion criteria. Out of this initial sample of consecutive patients, 41 were excluded after applying the exclusion criteria. The final sample consisted of 304 patients (144 females, 160 males) with a mean age of 68 ± 19 std ranging from 22 to 90 years old. Among these, clinical data regarding ICU admission and death were available for 295 patients only, as 9 patients were lost to follow-up after transferring to another hospital. Consequently, the analysis about those variables is solely based on the patients from whom complete data was available (Fig. 4).

Fig. 4

study flow chart

129 of 304 (42.4%) patients tested negative for COVID-19 and 175 (57.5%) patients tested positive for COVID-19, confirmed by a PCR test.

169 of 304 (55.6%) patients were diagnosed with COVID-19 pneumonia, 63 (20.7%) had non-COVID pneumonia, 22 (9.2%) had lung edema, and the rest had miscellaneous conditions such as pulmonary embolism, acute exacerbation of chronic obstructive pulmonary disease, etc. 239 of 304 (78.3%) patients were admitted to our hospital after the initial diagnosis in the Emergency Department. Of the 295 patients from whom we have clinical data, 91 (30.8%) were admitted to the ICU and 37 (12.5%) died after hospitalization (Table 2).

Table 2 Clinical variables of the population of the study

The Kolmogorov–Smirnov test revealed that none of the variables in Table 1 (variables used in the study) exhibited normality (P < 0.05).

The receiver operating characteristic curves (ROC) for LOP in any lung had an area under the curve (AUC) of 0.807. As for HOP in any lung, the AUC was 0.861, both with a 95% confidence interval. ROC curves are depicted in Fig. 5. Quantitative analysis is presented in Table 3.

Fig. 5

ROC curves for low opacity and high opacity percentages. Separated ROC curves are generated for the left lung (blue), the right lung (red), and both lungs combined (green). The value of LOP, provided by the AI software, uses the presence of ground glass opacity, provided by the radiological report, as ground truth, while the value of HOP, provided by the AI software, uses the presence of consolidation, provided by the radiological report, as ground truth. LOP: low opacity percentage; HOP: high opacity percentage

Table 3 AUC for low and high opacities percentages

Opacity tests were adjusted based on the coordinates of these curves to achieve a sensitivity cut-off of 95%. For instance, the cut-off point for LOP in any lung was determined to be 0.76%, while for HOP in any lung, it was 0.35%. This means that, if the AI software detects low opacity regions occupying at least 0.76% of the combined volume of both lungs, it is considered a positive test. These opacity tests demonstrated moderate positive agreement with the radiological report in detecting pulmonary opacities for the right lung, the left lung, and both lungs combined, with κ values ranging from 0.43 to 0.53 (P < 0.05).

The LOP and HOP tests previously generated showed a significant, strong, and positive association with hospital admission in both the COVID-19 positive and negative groups (P < 0.001, Phi > 0.4). For ICU admission, there was a significant, weak, and positive association with the HOP test in both groups and with the LOP test in the COVID-19-positive group only (P < 0.05, Phi > 0.3). In terms of death, there was a significant, weak, and positive association with the HOP test in the COVID-19-positive group only (P < 0.05, Phi = 0.21). Data depicted in Table 4, Figs. 6, and 7.

Table 4 Chi-Square tests for LOP and HOP tests by Hospital admission, ICU admission, and death

Fig. 6

LOP test by Hospital admission. There was a higher frequency of Hospital admission among those patients with a positive LOP test result

Fig. 7

HOP test by Hospital admission. There was a higher frequency of Hospital admission among those patients with a positive HOP test result

There was a statistically significant, positive, and moderate association between both the LOP and HOP values in any lung, provided by AI-Rad Companion, and the duration of hospital stay in both the COVID-19 negative and positive groups. In the COVID-19 negative group, the association had a Spearman's rho of 0.433 for LOP, and 0.438 for HOP (P < 0.001). Similarly, in the COVID-19-positive group, the association had a Spearman's rho of 0.605 for LOP, and 0.596 for HOP (P < 0.001) (Table 5).

Table 5 Spearman’s rho correlations for LOP and HOP in any lung by days of hospital admission

Discussion

Our objective was to assess the prognostic implications of AI-Rad Companion's analysis of lung opacities on chest CT scans, particularly regarding hospital admission, ICU admission, and death. We also aimed to gain insight into COVID-19-negative patients initially suspected to have COVID-19 pneumonia but later diagnosed with non-COVID-19 pneumonia or other pulmonary diseases. The study was driven by the high patient volume in our emergency room at that time and the limited number of studies investigating the predictive value of artificial intelligence systems in analyzing pneumonia on thoracic CT scans in both COVID-19 and non-COVID-19 patients.

Key results

The opacity tests derived from the ROC curves (LOP and HOP tests) with a 95% sensitivity cut-off demonstrated moderate agreement with the radiological opacity quantification and provided valuable prognostic information for both COVID-19-positive and COVID-19-negative patients. We found a significant association between the results of both LOP and HOP tests and hospital admission in both patient groups. The HOP tests also showed an association with ICU admission in both groups and with death specifically in the COVID-19-positive group. However, we did not observe an association between LOP tests and death. This could be explained by the fact that ground glass opacities tend to consolidate as COVID-19 disease progresses and patients' conditions deteriorate [25]. Additionally, the percentage of AI-detected opacities was associated with hospital stay in both groups. This was proven for both types of opacities.

Similar studies

In a similar study, Chaganti et al. [21] used the same AI method for lung segmentation and abnormality quantification. They also reported a strong correlation between their AI predictions and ground truth in COVID-19 patients, with a Pearson correlation coefficient of 0.92 for the percentage of low-opacity (P < 0.001) and 0.97 for the percentage of high-opacity (P < 0.001).

In a study conducted by Fang et al. [26], an AI-based framework utilizing deep neural networks was developed to segment lung lobes and pulmonary opacities. The study revealed a strong association between AI-based severity scores in COVID-19 patients and scores evaluated by radiologists (Spearman's rank = 0.837, P < 0.001). The AI method achieved the highest accuracy in predicting ICU admission with an area under the ROC curve (AUC) of 0.813 (95% CI [0.729, 0.886]), and in estimating mortality with an AUC of 0.741 (95% CI [0.640, 0.837]).

Mader et al. [16] used an AI model to assess pulmonary opacities in COVID-19 patients and investigate their outcomes, including ICU stay and mortality. The study found significant correlations (P < 0.001) between the extent of COVID-19-like opacities on chest CT and the occurrence and duration of ICU stay (R = 0.74 and R = 0.81, respectively), the likelihood of a fatal outcome (R = 0.56), and the length of hospital stay (R = 0.33, P < 0.05).

Gouda et al. [15] used the same software as our study and found that the total lung severity score and the total score for crazy-paving and consolidation, based on the extension of opacities in COVID-19 patients, could effectively differentiate between the severe and critical groups, as well as the mild group (with 90.9% sensitivity, 87.5% specificity, and 93.2% sensitivity, 87.5% specificity, respectively).

Limitations and possible biases

It is important to acknowledge certain limitations and potential biases in our study. Firstly, The AI system used in AI-Rad Companion is currently unable to differentiate between different types of opacities, such as those caused by pneumonia, tumors, atelectasis, or septal thickening. Furthermore, the version of the software that we used cannot specifically classify pneumonia as either COVID-19 or non-COVID-19. Other studies, including those by Ying et al. [10], Zhan et al. [27], and Wang et al. [28], have investigated this issue. The unique circumstances of the SARS-CoV-2 pandemic may have influenced the criteria for hospital and ICU admission, as well as the generalizability of our findings to non-pandemic situations. Additionally, we did not consider the potential impact of vaccination on patient outcomes due to incomplete data in medical records, which could have influenced our results. Moreover, due to resource limitations, we were unable to assess the interobserver and intraobserver variability of radiological reports, relying instead on single human observations as a benchmark for constructing ROC curves, which may introduce variability and subjectivity. Additionally, we could not find the rationale behind the AI software's use of a -200 UH cut-off for classifying high-opacity regions, as this information was not available in the existing literature. Among the patients included in our study who showed lung opacities but tested negative for COVID-19, there was a variety of lung conditions, mainly non-COVID-19 pneumonia and edema. However, it is important to clarify that our study does not focus on analyzing the prognosis for these specific conditions. it is worth noting that our study solely included the initial chest CT scans, and it may be advisable for future research to consider assessing the follow-up scans. These limitations should be taken into consideration when interpreting the results and generalizing the findings of our study.

Generalizability and interpretation

Overall, this study demonstrates the potential value of AI-Rad Companion's analysis of lung opacities in predicting hospital admission, ICU admission, and death in COVID-19 patients, hospitalization in COVID-19-negative patients, and hospital stay in both groups. However, generalization is limited and additional research is necessary outside of a pandemic context to effectively implement this software in Emergency Rooms. We believe that, as these AI algorithms continue to advance, they could be used in the screening of patients undergoing chest CT scans in the emergency area, facilitating risk stratification and predicting the likelihood of hospital admission and adverse outcomes.

Conclusions

AI-based opacity tests developed during the SARS-CoV-2 pandemic showed consistency with the radiological opacity quantification and were associated with some prognostic markers in patients with suspected COVID-19 pneumonia, even if they later tested negative for COVID-19 infection through PCR testing.