- Open Access
Radiomics reproducibility challenge in computed tomography imaging as a nuisance to clinical generalization: a mini-review
Egyptian Journal of Radiology and Nuclear Medicine volume 54, Article number: 83 (2023)
Radiomics has demonstrated striking potential in accurate cancer diagnosis but still needs strengthening of validity and standardization to achieve reproducible and generalizable results. Despite the advantages of radiomics, inter-scanner and intra-scanner variations of computed tomography (CT) scanning parameters can affect the reproducibility of its results. Accordingly, this article aims to review the impact of CT scanning parameters on the reproducibility of radiomics results.
Main body of the abstract
In general, radiomics results are sensitive to changes in the noise level; therefore, any parameter that affects image noise, such as kilovoltage (kVp), tube current (mAs), slice thickness, spatial resolution, image reconstruction algorithm, etc., can affect radiomics results. Also, region of interest (ROI) segmentation is another fundamental challenge in reducing radiomics reproducibility. Studies showed that almost all scanning parameters affect the reproducibility of radiomics. However, some robust features are reproducible.
One of the solutions to overcome the radiomics reproducibility challenge is the standardization of imaging protocols according to noise level (not scanning protocols). The second solution is to list reproducible features according to the type of complication and anatomical region. Resampling may also overcome feature instability.
Radiomics has generally defined the process of extracting and analyzing quantitative information from medical images that cannot be recognized by the naked eye . It has become an attractive field of medical research [2, 3]. It can be a suitable auxiliary approach for personalized medicine and correct and appropriate decision-making in diagnosis and treatment. By quantitatively analyzing the images based on the heterogeneity of the lesion, radiomics can overcome the challenges of visual and subjective interpretation of medical images [4, 5]. Studies have shown that radiomics features are significantly related to heterogeneity indices at the cell level, so it can be considered a non-invasive digital biopsy approach [3, 6]. The general flowchart of the radiomics workflow is divided into four main stages (Fig. 1).
In the first step, different imaging systems, such as computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US), or positron emission tomography (PET), obtain standard medical images [2, 7]. The second step is preprocessing, in which images are homogenized before features are extracted. Homogenization is done concerning pixel spacing, grey-level intensities, bins of the histogram, etc. The third step is segmentation, in which the volume of the region of interest (ROI) is determined. ROIs are delineated on medical images based on clinical demands; they can be the entire tumoral tissue or its subsets. These subsets can also include parts of necrosis and edema (accumulation of fluids). Segmentation may be delineated manually, semi-automatically, and fully automatically. Special toolkits extract radiomics features from the ROI volumes in the fourth step. These features can be divided into two categories: semantic and agnostic. Semantic features or shape features are usually recognized visually by radiologists to describe lesions, such as shape, size, location (lesion location); these features can also be extracted quantitatively and more accurately by the radiomics toolkits [5, 6]. Agnostic features (first and second order) cannot be inferred visually in the medical image. Still, they must be extracted from the voxels of the ROI image in the form of quantitative data based on complex mathematical and statistical methods. First-order features describe the distribution of the values of individual pixels individually without analyzing the spatial relationship of the pixels; actually, the features are based on histogram data, such as mean, median, maximum, and minimum pixel intensity values on the image, as well as skewness, kurtosis, uniformity, and entropy. The second-order features also called textural features are obtained by calculating the statistical inter-relationships between neighboring pixels and show a measure of the spatial arrangement of pixel intensity and, as a result, the heterogeneity within the lesion. Such features can be obtained from the grey-level co-occurrence matrix (GLCM), grey-level run-length matrix (GLRLM), gray-level dependence matrix (GLDM), gray-level size zone matrix (GLSZM), and neighboring gray-tone difference matrix (NGTDM) [4, 8, 9]. Some other features can also be obtained after applying wavelet, Laplacian, and Gaussian transfer functions to the extracted images. The extracted radiomics features can then be analyzed using statistical and machine learning methods [6, 10,11,12,13,14,15,16].
The parameters affecting the results of radiomics in CT scan are illustrated in Fig. 2 . Any variation in scanning parameters may affect image quality and radiomics reproducibility . Reproducibility is the process of minimizing errors while changing the methodology or data at the same time. Specifically, in radiomics, it means finding robust features against variations in equipment, software, image acquisition settings, operators, and even subjects [18, 19]. The parameters responsible for reducing the reproducibility of radiomics features are known as "destructive parameters." Knowing the destructive parameters and the pattern of their impact on radiomics features can be valuable and helpful in standardizing imaging for radiomics purposes. Previous studies about CT scanning parameters' role in radiomics reproducibility have reported different and, in some cases, contradictory results. According to various reports on the results of radiomics in CT scan, the main purpose of this review article is to examine the reports related to the impact of CT imaging parameters on the reproducibility of radiomics features.
CT scanning parameters and radiomics reproducibility
Table 1 summarizes several main characteristics of the studies about CT radiomics reproducibility and their main results and conclusions. In general, most CT scanning parameters could be considered destructive. Previous studies have reported that conclusions in radiomics should be made cautiously because radiomics features may undergo significant changes against minor changes in the medical image . Despite all the advantages of radiomics, the most critical challenge is the dependence of radiomics results on scanning parameters, segmentation, image reconstruction algorithm, pre-processing, and post-processing [1, 2, 20, 45]. In general, scanning parameters such as kilovoltage, pitch, and mAs affect the reproducibility of radiomics results. However, there are inconsistent reports regarding the amount of effects and importance of each of these parameters on radiomics features. Also, due to the difference in imaging protocols in different vendors of CT units, it is necessary to conduct more studies on the reproducibility of features under the influence of scanning parameters at different CT brands [19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47].
A general inference from previous research about the effects of scan parameters on the radiomics features variations is that radiomics features are sensitive to changes in the noise level of images, so any scan parameter that affects image noise can affect the reproducibility of radiomics [20, 21, 33, 41]. Also, pre-processing and post-processing operations in scanners, details of imaging protocols, such as reconstruction algorithms and related noise suppress methods, pixel size, resolution, contrast level, can significantly affect the calculated radiomics features .
Suggestion To maximize the valuable information obtained from CT scan images in radiomics and avoid misinterpretation of its results, researchers must understand all noise sources. This understanding can help to develop solutions such as image pre-processing to reduce noise effects. In retrospective studies, knowledge and awareness of noise level can be used as a guide for choosing the appropriate image for radiomics analysis (for example, only images with pixel sizes within a specific range have been selected for radiomics analysis). In prospective studies, noise analysis can be used to optimize imaging protocols. Therefore, it seems better to investigate the impact of noise level on the reproducibility of radiomics in a large cohort study with the participation of different clinics.
Radiomics features may highly depend on slice thickness, pixel size, resolution, and voxel size changes [19, 23, 26, 27, 34, 38, 39, 44] (Table 1). Therefore, such scanning parameters should be adjusted based on the radiomics feature, imaging parameters, type of lesion, and clinical outcomes. It is necessary to evaluate the strength of the features in terms of their inherent dependence on the slice thickness, pixel size, and FOV.
Suggestion It would be better to obtain the pattern of robust feature changes instead of examining the impact of each scanning parameter on the results of radiomics. Resampling CT images may correct the variability in radiomics features due to inconsistent scanning parameters such as pixel size, FOV, slice thickness.
Image reconstruction and processing
Recent papers also have reported the effect of pre-processing on reproducibility [19, 27, 34, 35, 44] (Table 1). Image pre-processing in radiomics software can change the results of radiomics calculations. Common pre-processing operations are rescaling, resampling, normalizing, gray level range, and quantizing gray values. Accordingly, these image resolution parameters and pre-processing may finally show their impact on noise level and image quality. In general, inconsistent results have been reported regarding the impact of changes in the reconstruction algorithm and kernel on the reproducibility of radiomics features compared to other scanning parameters [21, 22, 26, 36,37,38,39,40, 44] (Table 1). Therefore, it is recommended that the impact of this parameter on the results of radiomics features is evaluated more comprehensively in future studies.
Suggestion For increasing the reproducibility of radiomics is to standardize the imaging method. To generalize radiomics in clinical fields, some studies have suggested that the development of standardized protocols for CT scans may be a solution [5, 6, 45]. However, standardizing CT scanning protocols still has some problems, especially since each CT unit's technical characteristics and settings depend on the vendor. Therefore, even if a standard protocol is designed for radiomics studies workflow, different CT units will not produce similar images. This is due to the difference in detector systems, electronics, and reconstruction kernel between vendors and anatomical, physiological, and patient position changes. Of course, there are efforts to standardize the radiomics workflow in the cancer research community, which is called the process of “Image biomarker standardization initiative (IBSI)” . However, these efforts have not yet provided comprehensive guidelines for practical choices of CT scanning protocols, including the pixel size and the number of gray levels necessary to obtain robust and reliable results. A clear and decisive strategy for the segmentation process should also be provided.
Segmentation of the ROI is mainly conducted manually by medical professionals. Although segmentation is probably the most apparent source of inter-reader (inter-observer) variation and is often identified as a source of potential challenges in radiomics reproducibility, its role in radiomics has not yet been comprehensively investigated (most likely due to difficulties in creating a large dataset of tumors segmented by multiple observers) [1, 32, 37, 42, 43, 46, 47] (Table 1). Radiomics feature calculations can be performed as two dimension and three dimension, which produce different results. 3D radiomics features interpret the diagnostic power and differentiation of the abnormal tissue from the normal tissue better than the 2D radiomics results; however, manipulating 2D radiomics is faster and more accessible, and its results have shown more reproducibility than 3D [24, 32] (Table 1). The high reproducibility of 2D radiomics is perhaps because of the accuracy and feasibility of ROI delineating in the 2D method. Two-dimensional segmentation protocols should be proposed and developed to increase the reproducibility of the radiomics approach. Also, for radiomics analysis using textural features, it would be better to draw the ROI completely inside the lesions and be as far away from the borders as possible.
Suggestion Regarding segmentation, 2D segmentation protocols should be proposed and developed to increase reproducibility in radiomics research, providing those robust features that can distinguish the lesion from normal texture are introduced and extracted. Also, radiomics analysis using textural features should try to delineate the ROI completely inside the mass and be as far away from the anatomical borders as possible. A decision should be made regarding manual or automatic identification [1, 42, 43].
More than half of the reviewed CT radiomics studies have been performed using specific radiomics software, whereas other studies used in-house software or did not report the software [1, 19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44] (Table 1). MATLAB is the most frequently used tool for radiomics feature calculation, followed by open-access feature extraction toolkits such as Pyradiomics. Several other software has been used to extract radiomics features. Most previous studies have been conducted with LIFEx, IBEX, Pyradiomics, and Mazda. Some features in the software had the same name, but the mathematical calculation methods differed. On the contrary, some features had different names with the same calculation method.
Suggestion Introducing more standardized features in the radiomics workflow seems to be necessary. Several software has been introduced for radiomics analysis [20, 21, 25, 30]. A competent authority should design comprehensive radiomics analysis software to conduct all radiomics research. Also, the features’ names and the features’ mathematical calculation algorithm should be unified and standardized. Other proposed solution to overcome the reproducibility challenge of radiomics is to search for reproducible and robust features according to the type of disease or anatomical region. For example, specific features distinguishing the liver disease from the normal with high reproducibility should be extracted and listed. Then the features should be used in radiomics research for liver diseases. In conclusion shape feature categories have high reproducibility, followed by first-order features (such as pixel intensity, standard deviation, skewness, homogeneity, kurtosis) and second-order textural features such as gray-level co-occurrence matrix (GLCM) 
Radiomics has shown great potential in accurate cancer diagnosis, but it is still necessary to strengthen its validity and standardization to achieve reproducible results. It seems that reproducibility is a critical challenge in the way of CT radiomics generalizability. Most CT scanning parameters can impact the reproducibility of radiomics results. However, how each feature is affected by changes in scan parameters is still inconsistent. Also, any scanning parameter which affects the image noise can affect radiomics results. Future studies should focus more on the reproducibility challenge of CT radiomics by introducing standard feature extraction platforms, robust segmentation methods, and standardization of preprocessing parameters.
Availability of data and materials
Magnetic resonance imaging
Positron emission tomography
Region of interest
Gray-level co-occurrence matrix
Gray-level run-length matrix
Gray-level dependence matrix
Gray-level size zone matrix
Neighboring gray-tone difference matrix
Image biomarker standardization initiative
Haarburger C et al (2020) Radiomics feature reproducibility under inter-rater variability in segmentations of CT images. Sci Rep 10(1):1–10
Larue RT et al (2017) Quantitative radiomics studies for tissue characterization: a review of technology and methodological procedures. Br J Radiol 90(1070):20160665
Soleymani Y et al (2021) Evaluation of textural-based radiomics features for differentiation of COVID-19 pneumonia from non-COVID pneumonia. Egyptian J Radiol Nuclear Med 52(1):1–7
Avanzo M et al (2020) Machine and deep learning methods for radiomics. Med Phys 47(5):e185–e202
Van Timmeren JE et al (2020) Radiomics in medical imaging—“how-to” guide and critical reflection. Insights Imag 11(1):1–16
Mayerhoefer ME et al (2020) Introduction to radiomics. J Nucl Med 61(4):488–495
Soleymani Y et al (2022) Reproducibility assessment of radiomics features in various ultrasound scan settings and different scanner vendors. J Med Imag Radiat Sci 53(4):664–671
Rizzo S et al (2018) Radiomics: the facts and the challenges of image analysis. European Radiol Exper 2(1):1–8
Van Griethuysen JJ et al (2017) Computational radiomics system to decode the radiographic phenotype. Can Res 77(21):e104–e107
Cook GJ et al (2018) Challenges and promises of PET radiomics. Int J Radiat Oncol Biol Phys 102(4):1083–1089
Reuzé S et al (2018) Radiomics in nuclear medicine applied to radiation therapy: methods, pitfalls, and challenges. Int J Radiat Oncol Biol Phys 102(4):1117–1142
Ha S et al (2019) Radiomics in oncological PET/CT: a methodological overview. Nucl Med Mol Imag 53(1):14–29
Altazi BA et al (2017) Reproducibility of F18-FDG PET radiomic features for different cervical tumor segmentation methods, gray-level discretization, and reconstruction algorithms. J Appl Clin Med Phys 18(6):32–48
Nestle U et al (2005) Comparison of different methods for delineation of 18F-FDG PET–positive tissue for target volume definition in radiotherapy of patients with non–small cell lung cancer. J Nucl Med 46(8):1342–1348
Lu L et al (2016) Robustness of radiomic features in [11C] choline and [18F] FDG PET/CT imaging of nasopharyngeal carcinoma: impact of segmentation and discretization. Mol Imag Biol 18(6):935–945
Zwanenburg A (2019) Radiomics in nuclear medicine: robustness, reproducibility, standardization, and how to avoid data analysis traps and replication crisis. Eur J Nucl Med Mol Imag 46(13):2638–2655
Iranmakani S et al (2022) Image quality and pulmonary nodule detectability at low-dose computed tomography (low kVp and mAs): a phantom study. J Med Signals Sens 12(1):64–68
Park JE et al (2019) Reproducibility and generalizability in radiomics modeling: possible strategies in radiologic and statistical perspectives. Korean J Radiol 20(7):1124–1137
Escudero Sanchez L et al (2021) Robustness of radiomic features in CT images with different slice thickness, comparing liver tumour and muscle. Sci Rep 11(1):1–15
Mackin D et al (2018) Effect of tube current on computed tomography radiomic features. Sci Rep 8(1):1–10
Midya A et al (2018) Influence of CT acquisition and reconstruction parameters on radiomic feature reproducibility. J Med Imag 5(1):011020
Berenguer R et al (2018) Radiomics of CT features may be nonreproducible and redundant: influence of CT acquisition parameters. Radiology 288(2):407–415
Buch K et al (2018) Quantitative variations in texture analysis features dependent on MRI scanning parameters: a phantom model. J Appl Clin Med Phys 19(6):253–264
Fave X et al (2015) Preliminary investigation into sources of uncertainty in quantitative imaging features. Comput Med Imag Graph 44:54–61
Gao Y et al (2022) Reproducibility of radiomic features of pulmonary nodules between low-dose CT and conventional-dose CT. Quant Imag Med Surg 12(4):2368
Li Y et al (2022) The impact of phantom design and material-dependence on repeatability and reproducibility of CT-based radiomics features. Med Phys 49(3):1648–1659
Larue RT et al (2017) Influence of gray level discretization on radiomic feature stability for different CT scanners, tube currents and slice thicknesses: a comprehensive phantom study. Acta Oncol 56(11):1544–1553
Mackin D et al (2015) Measuring CT scanner variability of radiomics features. Invest Radiol 50(11):757
Ibrahim A et al (2021) Reproducibility of CT-based hepatocellular carcinoma radiomic features across different contrast imaging phases: a proof of concept on SORAMIC trial data. Cancers 13(18):4638
Caramella C et al (2018) Can we trust the calculation of texture indices of CT images? A phantom study. Med Phys 45(4):1529–1536
Zwanenburg A et al (2020) The image biomarker standardization initiative: standardized quantitative radiomics for high-throughput image-based phenotyping. Radiology 295(2):328
Balagurunathan Y et al (2014) Reproducibility and prognosis of quantitative features extracted from CT images. Transl Oncol 7(1):72–87
Fave X et al (2015) Can radiomics features be reproducibly measured from CBCT images for patients with non-small cell lung cancer? Med Phys 42(12):6784–6797
Shafiq-Ul-Hassan M et al (2017) Intrinsic dependencies of CT radiomic features on voxel size and number of gray levels. Med Phys 44(3):1050–1062
Mackin D et al (2017) Harmonizing the pixel size in retrospective computed tomography radiomics studies. PLoS ONE 12(9):e0178524
Solomon J et al (2016) Quantitative features of liver lesions, lung nodules, and renal stones at multi–detector row CT examinations: dependency on radiation dose and reconstruction algorithm. Radiology 279(1):185–194
Kim H et al (2016) Impact of reconstruction algorithms on CT radiomic features of pulmonary tumors: analysis of intra-and inter-reader variability and inter-reconstruction algorithm variability. PLoS ONE 11(10):e0164924
Meyer M et al (2019) Reproducibility of CT radiomic features within the same patient: influence of radiation dose and CT reconstruction settings. Radiology 293(3):583–591
He L et al (2016) Effects of contrast-enhancement, reconstruction slice thickness and convolution kernel on the diagnostic performance of radiomics signature in solitary pulmonary nodule. Sci Rep 6:34921
Muenzfeld H et al (2021) Intra-scanner repeatability of quantitative imaging features in a 3D printed semi-anthropomorphic CT phantom. Eur J Radiol 141:109818
Zwanenburg A et al (2019) Assessing robustness of radiomic features by image perturbation. Sci Rep 9(1):1–10
Kalpathy-Cramer J et al (2016) Radiomics of lung nodules: a multi-institutional study of robustness and agreement of quantitative imaging features. Tomography 2(4):430–437
Kelahan LC et al (2022) Role of hepatic metastatic lesion size on inter-reader reproducibility of CT-based radiomics features. Eur Radiol 32(6):4025–4033
Li Y et al (2020) Influence of feature calculating parameters on the reproducibility of CT radiomic features: a thoracic phantom study. Quant Imaging Med Surg 10(9):1775
Reiazi R et al (2021) The impact of the variation of imaging parameters on the robustness of computed tomography radiomic features: a review. Comput Biol Med 133:104400
Jensen LJ et al (2021) Stability of radiomic features across different region of interest sizes—A CT and MR phantom study. Tomography 7(2):238–252
Jensen LJ et al (2022) Enhancing the stability of CT radiomics across different volume of interest sizes using parametric feature maps: a phantom study. European Radiol Exper 6(1):43
This article has been approved and implemented from a grant allocated by the Honorable Deputy for Research of Tabriz University of Medical Sciences. Therefore, we consider it necessary to express our thanks and appreciation to the esteemed Vice President for Research.
This article has been funded by a grant allocated by the Honorable Deputy for Research of Tabriz University of Medical Sciences. The funder had no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.
Ethics approval and consent to participate
This article has been approved and implemented from a grant allocated by the Honorable Deputy for Research of Tabriz University of Medical Sciences with tracking code 68265 and code of ethics IR.TBZMED.VCR.REC.1400.346. This was a review article and there was no need for consent to participate.
Consent for publication
Authors declare that there is no conflict of interest to declare.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Jahanshahi, A., Soleymani, Y., Fazel Ghaziani, M. et al. Radiomics reproducibility challenge in computed tomography imaging as a nuisance to clinical generalization: a mini-review. Egypt J Radiol Nucl Med 54, 83 (2023). https://doi.org/10.1186/s43055-023-01029-6
- Computed tomography
- Radiomics reproducibility
- Quantitative medical imaging