Radiomics reproducibility challenge in computed tomography imaging as a nuisance to clinical generalization: a mini-review

Abstract

Background

Radiomics has demonstrated striking potential in accurate cancer diagnosis but still needs strengthening of validity and standardization to achieve reproducible and generalizable results. Despite the advantages of radiomics, inter-scanner and intra-scanner variations of computed tomography (CT) scanning parameters can affect the reproducibility of its results. Accordingly, this article aims to review the impact of CT scanning parameters on the reproducibility of radiomics results.

Main body of the abstract

In general, radiomics results are sensitive to changes in the noise level; therefore, any parameter that affects image noise, such as kilovoltage (kVp), tube current (mAs), slice thickness, spatial resolution, image reconstruction algorithm, etc., can affect radiomics results. Also, region of interest (ROI) segmentation is another fundamental challenge in reducing radiomics reproducibility. Studies showed that almost all scanning parameters affect the reproducibility of radiomics. However, some robust features are reproducible.

Short conclusion

One of the solutions to overcome the radiomics reproducibility challenge is the standardization of imaging protocols according to noise level (not scanning protocols). The second solution is to list reproducible features according to the type of complication and anatomical region. Resampling may also overcome feature instability.

Background

Radiomics has generally defined the process of extracting and analyzing quantitative information from medical images that cannot be recognized by the naked eye [1]. It has become an attractive field of medical research [2, 3]. It can be a suitable auxiliary approach for personalized medicine and correct and appropriate decision-making in diagnosis and treatment. By quantitatively analyzing the images based on the heterogeneity of the lesion, radiomics can overcome the challenges of visual and subjective interpretation of medical images [4, 5]. Studies have shown that radiomics features are significantly related to heterogeneity indices at the cell level, so it can be considered a non-invasive digital biopsy approach [3, 6]. The general flowchart of the radiomics workflow is divided into four main stages (Fig. 1).

Fig. 1

The workflow of typical radiomics

In the first step, different imaging systems, such as computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US), or positron emission tomography (PET), obtain standard medical images [2, 7]. The second step is preprocessing, in which images are homogenized before features are extracted. Homogenization is done concerning pixel spacing, grey-level intensities, bins of the histogram, etc. The third step is segmentation, in which the volume of the region of interest (ROI) is determined. ROIs are delineated on medical images based on clinical demands; they can be the entire tumoral tissue or its subsets. These subsets can also include parts of necrosis and edema (accumulation of fluids). Segmentation may be delineated manually, semi-automatically, and fully automatically. Special toolkits extract radiomics features from the ROI volumes in the fourth step. These features can be divided into two categories: semantic and agnostic. Semantic features or shape features are usually recognized visually by radiologists to describe lesions, such as shape, size, location (lesion location); these features can also be extracted quantitatively and more accurately by the radiomics toolkits [5, 6]. Agnostic features (first and second order) cannot be inferred visually in the medical image. Still, they must be extracted from the voxels of the ROI image in the form of quantitative data based on complex mathematical and statistical methods. First-order features describe the distribution of the values of individual pixels individually without analyzing the spatial relationship of the pixels; actually, the features are based on histogram data, such as mean, median, maximum, and minimum pixel intensity values on the image, as well as skewness, kurtosis, uniformity, and entropy. The second-order features also called textural features are obtained by calculating the statistical inter-relationships between neighboring pixels and show a measure of the spatial arrangement of pixel intensity and, as a result, the heterogeneity within the lesion. Such features can be obtained from the grey-level co-occurrence matrix (GLCM), grey-level run-length matrix (GLRLM), gray-level dependence matrix (GLDM), gray-level size zone matrix (GLSZM), and neighboring gray-tone difference matrix (NGTDM) [4, 8, 9]. Some other features can also be obtained after applying wavelet, Laplacian, and Gaussian transfer functions to the extracted images. The extracted radiomics features can then be analyzed using statistical and machine learning methods [6, 10,11,12,13,14,15,16].

The parameters affecting the results of radiomics in CT scan are illustrated in Fig. 2 [5]. Any variation in scanning parameters may affect image quality and radiomics reproducibility [17]. Reproducibility is the process of minimizing errors while changing the methodology or data at the same time. Specifically, in radiomics, it means finding robust features against variations in equipment, software, image acquisition settings, operators, and even subjects [18, 19]. The parameters responsible for reducing the reproducibility of radiomics features are known as "destructive parameters." Knowing the destructive parameters and the pattern of their impact on radiomics features can be valuable and helpful in standardizing imaging for radiomics purposes. Previous studies about CT scanning parameters' role in radiomics reproducibility have reported different and, in some cases, contradictory results. According to various reports on the results of radiomics in CT scan, the main purpose of this review article is to examine the reports related to the impact of CT imaging parameters on the reproducibility of radiomics features.

Fig. 2

Classification of different CT scanning parameters affecting radiomics

Main text

CT scanning parameters and radiomics reproducibility

Table 1 summarizes several main characteristics of the studies about CT radiomics reproducibility and their main results and conclusions. In general, most CT scanning parameters could be considered destructive. Previous studies have reported that conclusions in radiomics should be made cautiously because radiomics features may undergo significant changes against minor changes in the medical image [5]. Despite all the advantages of radiomics, the most critical challenge is the dependence of radiomics results on scanning parameters, segmentation, image reconstruction algorithm, pre-processing, and post-processing [1, 2, 20, 45]. In general, scanning parameters such as kilovoltage, pitch, and mAs affect the reproducibility of radiomics results. However, there are inconsistent reports regarding the amount of effects and importance of each of these parameters on radiomics features. Also, due to the difference in imaging protocols in different vendors of CT units, it is necessary to conduct more studies on the reproducibility of features under the influence of scanning parameters at different CT brands [19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47].

Table 1 Summary of the characteristics and the results of the studies about CT radiomics reproducibility

A general inference from previous research about the effects of scan parameters on the radiomics features variations is that radiomics features are sensitive to changes in the noise level of images, so any scan parameter that affects image noise can affect the reproducibility of radiomics [20, 21, 33, 41]. Also, pre-processing and post-processing operations in scanners, details of imaging protocols, such as reconstruction algorithms and related noise suppress methods, pixel size, resolution, contrast level, can significantly affect the calculated radiomics features [20].

Suggestion To maximize the valuable information obtained from CT scan images in radiomics and avoid misinterpretation of its results, researchers must understand all noise sources. This understanding can help to develop solutions such as image pre-processing to reduce noise effects. In retrospective studies, knowledge and awareness of noise level can be used as a guide for choosing the appropriate image for radiomics analysis (for example, only images with pixel sizes within a specific range have been selected for radiomics analysis). In prospective studies, noise analysis can be used to optimize imaging protocols. Therefore, it seems better to investigate the impact of noise level on the reproducibility of radiomics in a large cohort study with the participation of different clinics.

Data acquisition

Radiomics features may highly depend on slice thickness, pixel size, resolution, and voxel size changes [19, 23, 26, 27, 34, 38, 39, 44] (Table 1). Therefore, such scanning parameters should be adjusted based on the radiomics feature, imaging parameters, type of lesion, and clinical outcomes. It is necessary to evaluate the strength of the features in terms of their inherent dependence on the slice thickness, pixel size, and FOV.

Suggestion It would be better to obtain the pattern of robust feature changes instead of examining the impact of each scanning parameter on the results of radiomics. Resampling CT images may correct the variability in radiomics features due to inconsistent scanning parameters such as pixel size, FOV, slice thickness.

Image reconstruction and processing

Recent papers also have reported the effect of pre-processing on reproducibility [19, 27, 34, 35, 44] (Table 1). Image pre-processing in radiomics software can change the results of radiomics calculations. Common pre-processing operations are rescaling, resampling, normalizing, gray level range, and quantizing gray values. Accordingly, these image resolution parameters and pre-processing may finally show their impact on noise level and image quality. In general, inconsistent results have been reported regarding the impact of changes in the reconstruction algorithm and kernel on the reproducibility of radiomics features compared to other scanning parameters [21, 22, 26, 36,37,38,39,40, 44] (Table 1). Therefore, it is recommended that the impact of this parameter on the results of radiomics features is evaluated more comprehensively in future studies.

Suggestion For increasing the reproducibility of radiomics is to standardize the imaging method. To generalize radiomics in clinical fields, some studies have suggested that the development of standardized protocols for CT scans may be a solution [5, 6, 45]. However, standardizing CT scanning protocols still has some problems, especially since each CT unit's technical characteristics and settings depend on the vendor. Therefore, even if a standard protocol is designed for radiomics studies workflow, different CT units will not produce similar images. This is due to the difference in detector systems, electronics, and reconstruction kernel between vendors and anatomical, physiological, and patient position changes. Of course, there are efforts to standardize the radiomics workflow in the cancer research community, which is called the process of “Image biomarker standardization initiative (IBSI)” [31]. However, these efforts have not yet provided comprehensive guidelines for practical choices of CT scanning protocols, including the pixel size and the number of gray levels necessary to obtain robust and reliable results. A clear and decisive strategy for the segmentation process should also be provided.

Segmentation

Segmentation of the ROI is mainly conducted manually by medical professionals. Although segmentation is probably the most apparent source of inter-reader (inter-observer) variation and is often identified as a source of potential challenges in radiomics reproducibility, its role in radiomics has not yet been comprehensively investigated (most likely due to difficulties in creating a large dataset of tumors segmented by multiple observers) [1, 32, 37, 42, 43, 46, 47] (Table 1). Radiomics feature calculations can be performed as two dimension and three dimension, which produce different results. 3D radiomics features interpret the diagnostic power and differentiation of the abnormal tissue from the normal tissue better than the 2D radiomics results; however, manipulating 2D radiomics is faster and more accessible, and its results have shown more reproducibility than 3D [24, 32] (Table 1). The high reproducibility of 2D radiomics is perhaps because of the accuracy and feasibility of ROI delineating in the 2D method. Two-dimensional segmentation protocols should be proposed and developed to increase the reproducibility of the radiomics approach. Also, for radiomics analysis using textural features, it would be better to draw the ROI completely inside the lesions and be as far away from the borders as possible.

Suggestion Regarding segmentation, 2D segmentation protocols should be proposed and developed to increase reproducibility in radiomics research, providing those robust features that can distinguish the lesion from normal texture are introduced and extracted. Also, radiomics analysis using textural features should try to delineate the ROI completely inside the mass and be as far away from the anatomical borders as possible. A decision should be made regarding manual or automatic identification [1, 42, 43].

Feature extraction

More than half of the reviewed CT radiomics studies have been performed using specific radiomics software, whereas other studies used in-house software or did not report the software [1, 19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44] (Table 1). MATLAB is the most frequently used tool for radiomics feature calculation, followed by open-access feature extraction toolkits such as Pyradiomics. Several other software has been used to extract radiomics features. Most previous studies have been conducted with LIFEx, IBEX, Pyradiomics, and Mazda. Some features in the software had the same name, but the mathematical calculation methods differed. On the contrary, some features had different names with the same calculation method.

Suggestion Introducing more standardized features in the radiomics workflow seems to be necessary. Several software has been introduced for radiomics analysis [20, 21, 25, 30]. A competent authority should design comprehensive radiomics analysis software to conduct all radiomics research. Also, the features’ names and the features’ mathematical calculation algorithm should be unified and standardized. Other proposed solution to overcome the reproducibility challenge of radiomics is to search for reproducible and robust features according to the type of disease or anatomical region. For example, specific features distinguishing the liver disease from the normal with high reproducibility should be extracted and listed. Then the features should be used in radiomics research for liver diseases. In conclusion shape feature categories have high reproducibility, followed by first-order features (such as pixel intensity, standard deviation, skewness, homogeneity, kurtosis) and second-order textural features such as gray-level co-occurrence matrix (GLCM) [45]

Conclusions

Radiomics has shown great potential in accurate cancer diagnosis, but it is still necessary to strengthen its validity and standardization to achieve reproducible results. It seems that reproducibility is a critical challenge in the way of CT radiomics generalizability. Most CT scanning parameters can impact the reproducibility of radiomics results. However, how each feature is affected by changes in scan parameters is still inconsistent. Also, any scanning parameter which affects the image noise can affect radiomics results. Future studies should focus more on the reproducibility challenge of CT radiomics by introducing standard feature extraction platforms, robust segmentation methods, and standardization of preprocessing parameters.