Interrater reliability of the ovarian–adnexal reporting and data system magnetic resonance imaging (O‑RADS MRI)

Abstract

Aim

The current study aims to evaluate interrater reliability of ovarian–adnexal reporting and data system (O-RADS) magnetic resonance imaging (MRI) in interpretation of adnexal and ovarian lesions.

Material and methods

Retrospective analysis of 131 ovarian lesions was as done for 106 consecutive female patients with adnexal and/or ovarian lesions that underwent MR imaging of the pelvis. Images interpretation was accomplished by two-blinded independent raters for cystic and solid parts of ovarian lesions. The score was 5 types classified pursuant to O-RADS.

Results

A perfect interrater agreement regarding overall O-RADS [Kappa: 0.874, P: 0.001]. There was a perfect interrater agreement of the solid component (Kappa: 0.979, 95% confidence interval (CI) 0.938–1.0, P: 0.001), enhancement degree relative to myometrium (Kappa: 0.876, 95% CI 0.781–0.971, P: 0.001) and entirely solid lesions (Kappa: 1.0, 95% CI 1.0–1.0, P: 0.001). A perfect interrater agreement for ORADS 1 (Kappa: 0.937, P: 0.001), ORADS 2 (Kappa: 0.983, P: 0.001), ORADS 3 (Kappa: 0.834, P: 0.001), ORADS 4 (Kappa: 0.827, P: 0.001) and ORADS 5 (Kappa: 0.963, P: 0.001) was present.

Conclusions

The O-RADS MRI scoring system has better characterization of adnexal masses with high interrater agreement. Overcoming limitations of this study, O-RADS, may be suggested as a basic system in assessment of adnexal masses.

Introduction

Ovarian and adnexal lesions are common and remain the first indication for female genital system surgeries worldwide [1]. The main purpose of imaging modalities is to decrease the number of unnecessary gynecologic operations, to maintain fertility in females in the childbearing period (by allowing minor operations as laparoscopic ones), and if required, facilitate the patients’ guidance to a gynecology oncologist in specialized center to ensure perfect outcome. The most important prognostic factor is complete resection of the lesion with no residue. Moreover, preoperative characterization of adnexal lesion is crucial, as diagnoses on the basis of histopathologic findings through frozen sections are uncertain [2,3,4,5].

Ultrasound (US), contrast-enhanced computed tomography (CT) and MRI are used for diagnosis of ovarian and adnexal lesions [6,7,8,9]; however, MRI is superior to US in the interpretation of lesions. Theoretically, the accuracy of MRI for differentiating complex benign and malignant adnexal lesions is about 80–90% compared with 60–90% by US, as it mainly depends on the operator experience [10,11,12,13]. Biopsy is the gold standard but must be handled cautiously [2,3,4,5].

The International Ovarian Tumor Analysis has built US-based risk classification systems by using basic evidence-based terms and abbreviations to make a simple rule for characterizing benign from malignant masses [14]. However, these rules lacked the ability to categorize complex adnexal lesions whether benign or malignant in about 20% of the situations, decreasing their importance. However, its features were used to measure the likelihood of malignancy [15].

Further multiple other systems have been proposed, but still lack standardized terminology and definitions of objective criteria for all lesions [16,17,18,19,20,21].

Furthermore, an MRI scoring system (ADNEX MR scoring system) was published by Thomassin-Naggara et al. in 2013 to interpret and characterize sonographically indeterminate adnexal masses. This system classified them into five subgroups based on appearance, T2-weighted image (WI) and diffusion-weighted imaging (DWI) signal, and dynamic contrast-enhanced MRI criteria of the solid component, by interpreting the time-intensity curve with high specificity and sensitivity in detecting malignancy (96.6% and 93.5%, respectively) [1]. Later, Thomassin-Naggara et al. upgraded this scoring system and released the O-RADS MRI pursuant to MRI findings. It showed a similarly high specificity and sensitivity (78% and 99%, respectively) [22, 23].

Anyhow, the main difficulty in applying both scoring systems is that time-intensity curve is not currently commonly used in clinical practice as there is no easy access for all radiologists to software package to evaluate.

Therefore, the current study aims to evaluate interrater reliability of O-RADS MRI in interpretation of ovarian and adnexal lesions.

Methods

Patients and inclusion criteria

This is a retrospective study that was approved by the institutional research board [code number: R.23.08.2316], and the informed consent was waived.

This study included 145 patients who underwent MRI of the pelvis for further assessment of adnexal masses after US examination, these patients were complaining of either pelvic pain or vaginal bleeding. Women evaluated during neoadjuvant chemotherapy (n = 5), and studies with technical errors such as artifacts (n = 4) were excluded. Thirty patients who lost follow-up (i.e., with neither obtainable histopathology nor follow-up imaging) were also excluded, so the final involved patients were 106 ladies with 131 suspected adnexal lesions, their age ranged from 18 to 84 years old (mean ± standard deviation (SD): 39.41 ± 14.48). 94 (71.8%) of the lesions were in pre-menopausal patients, while 37(28.2%) were in post-menopausal patients (Table 1). All patients underwent MR examination during the period from January 2019 to March 2020.

Table 1 Demographic and clinical data of the patients

Magnetic resonance imaging technique

Every patient involved in the research had an MRI through a 1.5-T scanner (Ingenia^®, Philips Healthcare) using abdominal phased-array coil. The patients were asked to fast for 4–6 h and avoid abdominal breathing and movement through examination. Pre-contrast sequences included T1 WI, T2 WI (axial, sagittal and coronal) with and without fat suppression (TE 80–100 ms, TR 2500-3000 ms), axial DWI with variable b values (0, 500 & 1000 s/mm²), FOV = 240 × 220, matrix a = 124 × 100, slice thickness = 6 mm, and inter-slice gap = 1 mm, apparent diffusion coefficient (ADC) maps were generated for all patients. For 70 patients (who had 73 lesions), intravenous contrast injection was done with a 0.1 mmol/kg dose by automatic injector, 36 patients did not have post-contrast study as they had findings coping with O-RADS 1 and 2 that means either normal or certainly benign lesions, so there was no need for contrast administration. Also, this study was a retrospective study. Post-contrast T1 gradient echo sequences (axial, sagittal and coronal) were done. The parameters were: flip angle 10°, TR/TE 3.3–4.5/1.4–1.9 ms, matrix size 172 × 135, number of excitations 2, slice thickness 2–3 mm, and field of view 300–400 mm.

Images interpretation

A secondary workstation was used (Phillips Advantage windows workstation). The examinations were studied by two independent radiologists who were blinded to clinical data, laboratory findings and previous radiological findings (with 9- and 12-years of gynecological imaging experience). Based on O-RADS criteria, the images were analyzed for laterality, locularity, whether purely cystic, fatty, and purely endometriotic lesion, absence of wall enhancement, grouped thickened septae, thickened regular and irregular septae, vegetations, solid tissue, purely solid, pattern of enhancement of solid lesions relative to myometrium, low T2 signal of solid part, low b 1000 signal, peritoneal fluid, and implants. The maximum dimensions of the lesions were measured on contrast enhanced and T2 images.

A purely cystic lesion (O-RADS 2) was diagnosed when no solid tissue or mural enhancement was detected and signal following that of fluid and corresponding to a unilocular cyst or hydrosalpinx [1, 14, 23]. A purely endometriotic lesion (O-RADS 2) was diagnosed when the lesion displayed high T1 SI, higher than or like that of subcutaneous fat, T2 shading, without solid component [1, 14, 23]. A purely fatty lesion (O-RADS 2) was determined when a lesion showed hyper-intense SI on T1 WI that was suppressed on fat saturation images with no associated solid component [1, 14, 23]. The raters also evaluated cyst wall enhancement, locularity, and the existence or absence of thick regular, grouped, or irregular septae. Solid tissue was defined as a solid component that shows post contrast enhancement with morphologic characteristics of papillary projections, mural nodules, irregular septations or walls [23, 24]. The solid tissue was evaluated for T2 SI (in comparison with the outer myometrium) by both readers. Lastly, interpretation of the presence or absence of peritoneal free fluid and nodules was done.

The pattern of enhancement of solid tissue was visually evaluated in comparison to the outer myometrium thirty and sixty seconds after contrast injection. There were three patterns of enhancement pursuant to visual assessment: low risk that appears as SI lower, than that of myometrium at thirty and sixty seconds after contrast administration; intermediate risk that appears as SI lower than that of myometrium at thirty seconds and higher at sixty seconds; and high risk, that appear as SI higher than that of myometrium at thirty seconds [24]. Each lesion with solid tissue was then categorized using the O-RADS MRI score: ORADS 2 was assigned for purely solid tissue if exhibited low T2 and low DWI SI. ORADS 3 was assigned for visual assessment low risk; ORADS 4 for visual assessment intermediate risk and ORADS 5 for visual assessment high risk or in cases with peritoneal involvement, according to the O-RADS MRI lexicon [23, 24].

The major and supplementary criteria were also studied for accurate O-RADS grouping, anyhow in this research, we considered the main criteria and the eventual O-RADS group for interrater agreement.

The two raters independently studied the MR examinations and categorized the findings conforming to O-RADS into: ORADS-1 (normal); ORADS-2 (definitely benign); ORADS-3 (probably benign); ORADS-4 (Indeterminate); and ORADS-5 (Probably malignant) [22,23,24].

Reference standard

The standard reference used for adnexal masses was post-operative histopathological diagnosis (n = 40 [benign = 10, borderline = 12, malignant = 16]) or imaging follow-up at 3- and 6-months intervals using either MRI and/or US. The lesions were assessed mainly for size (whether stationary, resolved, decreased size or even increased size) and other imaging criteria as development of solid tissue or newly appearing lesions (n = 60 [high possibility of malignancy = 3, mostly of benign nature = 57]).

Statistical analysis

Analysis of data was done by IBM SPSS Corp. Released 2013. Statistics for Windows (22 Armonk, NY: IBM Corp) version.

Qualitative variables were interpreted as numbers and percentage, while quantitative variables were interpreted as mean and SD for parametric data after normality testing using Kolmogo rov-Smirnov test. P value of (0.05) and less was considered statistically significant.

Interrater reliability was analyzed by measuring the 95% CI, intraclass correlation coefficient, and Cohen kappa statistics. A kappa value of 0.00–0.20 set slight agreement, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, and 0.81–1.00 almost perfect agreement [25].

Results

Interrater agreement for pelvic lesion

Interrater agreement of the imaging features is shown in Table 2. Perfect agreement was found regarding; laterality (Kappa: 0.918, 95% CI 0.854–0.982, P: 0.001), locularity (Kappa: 0.936, 95% CI 0.874–0.998, P = 0.001), pure cystic lesion (Kappa: 0.92, 95% CI 0.852–0.989, P: 0.001), and absence of wall enhancement (Kappa: 0.876, 95% CI 0.781–0.971, P: 0.001). The percentage of agreement between both raters was 98.5% for pure endometriotic lesion, 99.24% for vegetations, 99.24% for solid tissue and 95.4% for low T2 signal, 94.57% for low b 1000 SI and 91.7% for enhancement degree relative to the myometrium. The agreement between both raters was perfect for thickened regular (Kappa: 0.922, 95% CI 0.854–0.989, P: 0.001) and irregular septae (Kappa: 0.911, 95% CI 0.834–0.987, P: 0.001), and purely solid lesions (Kappa: 1.0, 95% CI 1.0–1.0, P 0.001). There was moderate agreement between both raters as regard purely fatty lesions (Kappa: 0.663, 95% CI 0.044–1.0, P 0.001) with 99.24% agreement percentage.

Table 2 Inter-rater agreement of findings of pelvic lesions of O-RADS

Interrater agreement regarding peritoneal fluid and nodules

A perfect interrater agreement regarding peritoneal fluid (Kappa: 0.985, 95% CI 0.955–1.0, P 0.001), and peritoneal nodules (Kappa: 1.0, 95% CI 1.0, P 0.001) was present. The percentage of agreement between both raters was 99.24% and 100%, respectively Table 2.

Interrater agreement for O-RADS categorization

Interrater agreement of O-RADS groups is shown in Table 3. The O-RADS of both readers were O-RADS 1 (n = 9 & 8), O-RADS 2 (n = 43 & 44) (Fig. 1), O-RADS 3 (n = 47 & 47) (Fig. 2), ORADS 4 (n = 17 & 16) (Fig. 3), O-RADS 5 (n = 15 & 16) (Fig. 4).

Table 3 Inter-rater agreement of different categories of O-RADS

Fig. 1

O-RADS MRI score 2 in a 49-year-old female with pelvic pain A Axial T1WI, B axial T2WI, C, D DWI; Right simple ovarian cyst of fluid like SI, no mural nodules, vegetations or soft tissue component

Fig. 2

O-RADS MRI score 3 in 53-year-old female with left pelvic pain A Axial T1WI, B axial T2WI, C, D DWI and ADC images: left adnexal bi-locular cystic lesion with thick wall and septae (black arrows) that displays low T1 and high T2 SI with thickening and stranding of surrounding fat planes (likely edema, white arrows), E Axial T1 post-contrast image shows thick marginal and septal enhancement but no associated solid component. This was diagnosed as left tubo-ovarian abscess

Fig. 3

O-RADS MRI score 4 in a 29-year-old female with pelvic pain A Axial T1WI, B, C axial, sagittal T2WI: Right adnexal cystic lesion (black arrow) shows few small mural nodules (white arrows) of low T1 and intermediate to high T2 SI. D Coronal T1 post-contrast image shows relatively thick marginal contrast enhancement and enhancement of the mural nodules (white arrows) [with enhancement pattern categorized as intermediate risk]. This was pathologically proven borderline serous ovarian tumor with intact capsule

Fig. 4

O-RADS MRI score 5 in a 56-year-old-female complaining form progressive abdominal enlargement, weight loss and malaise A Axial T1WI, B axial T2WI: Bilateral prominently solid adnexal lesions of low T1 and intermediate T2 SI (black arrows) with moderate ascites (white arrows) and few peritoneal nodules [arrow heads] C DWI: Restricted diffusion of the solid components D coronal postcontrast T1 WI showing heterogeneous contrast enhancement (with enhancement pattern categorized as high risk). This was pathologically proven bilateral high grade serous carcinoma involving both tubes and ovaries with detected lympho-vascular emboli

A perfect interrater agreement was present for O-RADS 1 (Kappa: 0.937, 95% CI 0.815–1.0, P: 0.001), O-RADS 2 (Kappa: 0.983, 95% CI 0.949–1.0, P: 0.001), O-RADS 3 (Kappa: 0.834, 95% CI 0.735–0.933, P: 0.001), O-RADS 4 (Kappa: 0.827, 95% CI 0.679–0.974, P: 0.001) and O-RADS 5 (Kappa: 0.963, 95% CI 0.892–1.0, P: 0.001) overall agreement was (Kappa: 0.874, 95% CI 0.806-0.942, P: 0.001). The agreement percentage of both raters were 99.24%, 99.2%, 92.37%, 96.18%, 99.24% for O-RADS one, O-RADS two, O-RADS three, O-RADS four, and O-RADS five, respectively. It was 90.84% for all cases.

Discussion

Ovarian–adnexal RADS is continuously updated, which is developed as the result of combination of expert radiological and clinical accordance, it will be continuously modified according to the developing experience, collected data, multi-disciplinary experts input, and actively updated feedback [22].

In the current study, there was overall perfect interrater agreement of both radiologists evaluating O-RADS. There was also excellent interrater agreement of the major criteria of O-RADS including laterality, locularity, purely cystic, endometriotic lesions, low T2 and b1000 signal, absence of wall enhancement, enhancement degree relative to the myometrium, vegetations, grouped septae, thickened regular and irregular septae (Kappa: 0.918, 0.936, 0.920, 0.915, 0.893, 0.874, 0.876, 0.876, 0.979, 0.903, 0.922, 0.911), respectively. There was moderate agreement for purely fatty lesion (Kappa: 0.663) this could be attributed to very small number of purely fatty lesions (only 2 by one observer and one by the second observer), so minor variation was magnified. These findings indicate that perfect interrater reliability could be achieved by well-trained raters who newly adopt the O-RADS and multicentric studies are warranted to increase the numbers of each category for more accurate results.

In this study, there was an excellent interrater agreement regarding enhancement degree of the solid lesion or part relative to the myometrium (Kappa: 0.979). This is one of the main criteria through which the lesion could be of visual high risk enhancement pattern, so those lesions can be categorized as O-RADS 5 [22].

Inspirited by the breast imaging-RADS, this scoring system depends on the probability of malignancy, so it may have several applications in clinical practice: O-RADS 4 or 5 is associated with higher possibility of malignancy. Thus, patients should be guided to the specialized center [22, 23]. For O-RADS 3, the possibility of malignancy is minimal, so the patients can avail themselves for follow up, more imaging, or conservative management. For O-RADS 2, the mass is benign, and no more investigations were required for interpretation [22].

In this research, it was observed that these MRI criteria are extremely important for exclusion of malignant nature in most benign cases. These criteria are mainly purely cystic, fatty or endometriotic lesion, low T2 and b = 1000 s/mm² SI within the solid part, and absent mural enhancement.

It is already documented that low T2 and low b = 1000 s/ mm² SI of the solid tissue are beneficial in the interpretation of complex adnexal lesions [9]. This was applied in our study with excellent results. Also, in this study, it was observed that existence of solid tissue is not enough to predict malignancy, while the lack of solid tissue is highly suggestive of benign lesion.

In this study, we unfortunately used visual assessment for interpretation of contrast enhancement of ovarian lesions compared to myometrium. Future study with time intensity curves application would be beneficial especially because the O-RADS MRI score was shown to perform better with time intensity curve than with visual assessment [24].

There was excellent agreement between both raters for peritoneal fluid and nodules, this high agreement could be explained by the high sensitivity of DWI (particularly b1000) images in the detection of peritoneal nodules and could be also attributed to the small number of patients with peritoneal fluid and nodules in this study. However, the presence of peritoneal fluid was not definitely associated with malignancy in all cases, as in some cases, it was a small amount that could be physiologically seen in the pelvis of females in the childbearing period, in other cases attributed to other co-associated morbidities. Peritoneal fluid was associated with malignancy in 15 lesions. In this study, it was observed that co-existence of both peritoneal fluid and nodules (15 lesions) together was highly associated with of malignant lesions.

A few studies discussed the reliability of O-RADS (based on both ultrasound and MRI) [26]. One of them reported that the use of an updated O-RADS version for 3–5 categories results in higher interrater reliability and more accurate categorization [22]. Some studies discussed the reliability of US O-RADS in categorization of adnexal and ovarian lesions [27,28,29]. Another study reported that interrater agreement is higher for O-RADS MRI that is based on O-RADS ultrasound [23].

Recent retrospective study demonstrated that pelvic MRI interpreted with the O-RADS MRI had more accurate diagnostic performance for the characterization of US-indeterminate lesions [30].

In accordance with other recent studies [31,32,33], this study showed excellent interrater agreement for all groups of ORADS. The relatively different interrater agreement in this study might be attributed to the different radiologists’ experience and the subjective manner in the assessment of the septae, thickness, and b1000 and enhancement of solid portion. This is in accordance with other recent studies.

The O-RADS is better to be incorporated with recommendations of the American association for studying ovarian diseases to manage ovarian lesions. O-RADS display choices and time intervals for the ovarian imaging and suggest reasonable alternatives of imaging for proper handling [34].

Reporting according to O-RADS has several merits in comparison to conventional reporting. Firstly, reporting of O-RADS provides a mutual language among radiologists, and clinicians to promote lucidity in communication and reporting. Secondly, structured O-RADS template report is more comprehensive and consistent with major features of ovarian lesions. Lastly, O-RADS provides recommendations for management of ovarian lesions. Recently, the acceptance of the new O-RADS by both clinicians and radiologists has improved compared with standard reporting [23, 24, 34].

This study had some limitations. First, this was a retrospective study. Thus, O-RADS scores were not used for patient management, and therefore, the impact of this scoring system on management and outcomes was unknown. Second, not all patients underwent post contrast MRI examinations due to the retrospective nature of the study, in these patients, the clinical outcomes were based on follow-up imaging. Third, only visual assessment of contrast enhancement of the solid portion was applied without time intensity curve as these cases are collected from a single-institution and thus likely affected by its own imaging standards and techniques, Fourth, patients with previous gynecologic surgeries were excluded from this study, future studies with inclusion of patients with previously operated lesions and pregnant ladies are recommended. Fifth, lack of the use of advanced MRI techniques. Finally, the readers were blinded to the patients’ medical histories and any prior imaging. However, this information may add some help during imaging interpretation and modify guideline recommendations based on patient status.

Future studies with advanced functional MRI imaging modalities as acquiring time intensity curve, applying diffusion tensor imaging, arterial spin labeling, histogram and machine learning would add in the accuracy, reliability, and update of O-RADS and can create another subdivisions and categories as previously mentioned reporting systems. Also, future multicentric studies are warranted to increase numbers of each category for more accurate results.

Conclusions

The O-RADS MRI scoring system has better characterization of adnexal masses with high interrater agreement. Overcoming limitations of this study, O-RADS, may be suggested as a basic system in assessment of adnexal masses.