Efficiency assessment of a two-stage diagnostic strategy combining CT angiography and fractional flow reserve derived from coronary CT angiography for the detection of myocardial ischemia: a simulation study

Abstract

Background

The importance of a diagnostic strategy combining coronary computed tomography angiography (CCTA) with fractional flow reserve derived from CCTA (FFRCT) for detecting myocardial ischemia is increasing. However, sensitivity and specificity alone may be insufficient to understand the efficiency characteristics of a diagnostic strategy combining CCTA and FFRCT (DSCCF). Our study aimed to evaluate the overall efficiency of DSCCF in detecting myocardial ischemia and compare it with other diagnostic strategies to determine whether evaluation by DSCCF is currently appropriate.

Results

This simulation study included 1000 patients with stable chest pain and suspected myocardial ischemia. Using a decision tree analysis, assuming a diagnostic strategy of adding FFRCT to CCTA-positive patients, we calculated the following efficiency parameters of DSCCF: (1) true positive (TP), false positive (FP), net false negative (FN), and net true negative (TN) test results; (2) net sensitivity; (3) net specificity; (4) positive predictive value; (5) negative predictive value; (6) post-test probability; (7) diagnostic accuracy; (8) diagnostic odds ratio; and (9) number needed to diagnose. We also calculated the efficiency parameters of other diagnostic strategies and compared them with those of DSCCF. In the basic setting, regarding efficiency parameters (1), the number of TPs, FPs, net FNs, and net TNs were 254, 69, 46, and 631, respectively. Efficiency parameters (2)–(9) were 0.85 (95% confidence interval [CI], 0.80–0.89), 0.90 (95% CI 0.88–0.92), 0.79 (95% CI 0.74–0.83), 0.93 (95% CI 0.91–0.95), 0.07 (95% CI 0.05–0.09), 0.89 (95% CI 0.86–0.90), 50.50 (95% CI 33.83–75.37), and 1.34 (95% CI 1.24–1.48), respectively. Compared with other diagnostic strategies, DSCCF had good efficiency parameters. Moreover, the sensitivity analysis did not reveal any evidence to contradict the findings in the basic setting.

Conclusions

This study demonstrated the diagnostic ability characteristics of DSCCF by assessing various efficiency parameters. Compared with other diagnostic strategies, DSCCF had good efficiency. In terms of efficiency, evaluation using DSCCF for detecting myocardial ischemia appears to be appropriate.

Background

Non-invasive diagnostic imaging for detecting myocardial ischemia in patients with stable chest pain is essential for deciding whether to perform invasive coronary angiography and revascularization [1,2,3]. Recently, with technological improvements in computed tomography (CT), coronary CT angiography (CCTA) has been widely performed to detect coronary artery disease (CAD) [3, 4]. The latest guidelines recommend using CCTA as an anatomical examination for future risk assessment of patients with intermediate-to-high risk for major cardiac events [5, 6]. However, since CCTA only provides information on coronary artery morphology, it may not be possible to determine the presence or absence of myocardial ischemia based on the presence or absence of coronary artery stenosis alone [7]. Therefore, in such cases, assessment of myocardial perfusion using other methods is required to accurately detect myocardial ischemia [5, 6].

In recent years, the use of fractional flow reserve (FFR) derived from CCTA (FFRCT) has become widespread in clinical practice for assessing myocardial perfusion using CT. This method evaluates myocardial blood flow by calculating FFR values through computational fluid dynamics analysis based on image data from CCTA [8,9,10]. Consequently, no additional imaging examinations are required, and the presence of myocardial ischemia can be evaluated by reanalyzing the image data obtained from CCTA. Therefore, a diagnostic strategy combining CCTA and FFRCT (DSCCF), which can obtain both coronary artery morphological information and myocardial perfusion in a single examination, has been widely used for detecting myocardial ischemia [9, 10]. Diagnostic tests with non-invasive functional imaging modalities, including FFRCT, are performed to select patients who require invasive coronary angiography and revascularization procedures [11,12,13]. Recent guidelines recommend additional evaluation with FFRCT as class 2a recommendations if 40–90% of stenotic lesions are detected using CCTA [6]. Most studies that have investigated the ability of cardiac imaging, including CCTA and FFRCT, in detecting myocardial ischemia have reported results using mainly sensitivity and specificity as indicators [1, 11]. However, it is difficult to determine the characteristics relevant to the diagnostic performance of DSCCF based solely on the sensitivity and specificity of CCTA and FFRCT, respectively. To properly incorporate DSCCF into diagnostic strategies aimed at detecting myocardial ischemia, it may be necessary to evaluate its efficiency, namely, the properties relevant to its diagnostic performance based on actual clinical situations. One example would be to obtain diagnostic performance indicators calculated from the combination of the pre-test probability (PTP) and the sensitivity and specificity of the DSCCF. If the effectiveness of DSCCF needs to be evaluated, the efficiency of diagnostic strategies combining CCTA and imaging modalities other than FFRCT should be evaluated and analyzed in comparison with DSCCF. Although the efficiency of FFRCT alone has been previously evaluated [14], the efficiency of DSCCF as a whole has not yet been assessed. Clarifying the efficiency of DSCCF using various indexes is considered to potentially yield advantages not only for the physician but also for other medical practitioners involved in the examination by enhancing their understanding of the capabilities of DSCCF and leading them to conduct precise and suitable diagnostic examinations.

Therefore, in this study, we aimed (1) to obtain the integrated sensitivity and specificity of DSCCF for detecting myocardial ischemia using information from the literature, (2) to evaluate the efficiency of DSCCF and PTP, and (3) to consider whether the evaluation using DSCCF is currently appropriate by comparing its efficiency with that of diagnostic strategies that add other modalities to CCTA.

Methods

Study design

This was a simulation study. This study was conducted using only published literature data, without including individual patient data. Therefore, institutional ethics approval was not obtained. In the analysis, 1,000 patients who satisfied both of the following clinical conditions were included [6]:

• Stable chest pain with no known CAD

• Intermediate-to-high risk for major CAD events based on the results of the initial evaluation

In these patients, the following clinical course was assumed for the simulation analysis:

• As a further examination, CCTA was performed first.

• Although a significant stenotic lesion was detected using CCTA, it was difficult to determine the presence or absence of myocardial ischemia based on CCTA results and symptoms [5, 6].

• FFRCT was performed to confirm the presence of myocardial ischemia using the same imaging data as CCTA.

DSCCF for the above clinical course was defined as a two-stage strategy (TS).

Literature search

We performed a literature search to collect data on the diagnostic ability for analyses. The literature search was as broad as possible to minimize potential bias and ensure transparency in the selection of data for analysis. Meta-analysis articles that evaluated the diagnostic ability of non-invasive imaging modalities to detect myocardial ischemia caused by CAD on a patient basis were searched. The reference standard of each searched article was exclusively invasive FFR because many studies have used this as the reference standard for assessing the diagnostic ability of non-invasive diagnostic imaging modalities for detecting CAD. Furthermore, it is a reference standard for the assessment of the severity of CAD and an important parameter when considering coronary revascularization [15]. A literature search was performed using the PubMed database to identify articles published between January 1, 2017 and October 31, 2022 (search date: November 17, 2022). The search terms were as follows:

• (1) “diagnostic accuracy” and “coronary artery disease”

• (2) “diagnostic accuracy” and “myocardial ischemia”

• (3) “diagnostic performance” and “coronary artery disease

• (4) “diagnostic performance” and “myocardial ischemia”

In case of multiple results, as a candidate article, the top three articles with the highest number of target patients were extracted. Subsequently, a qualitative assessment was performed using a previously published method [14]. Consequently, the article with the highest total score was selected for analysis. The article with the highest number of patients was selected in cases with the same scores. Finally, from the selected articles, data on the sensitivity and specificity of the diagnostic ability of each modality were extracted.

Definition of efficiencies for detecting myocardial ischemia

The efficiencies for detecting myocardial ischemia in this study were as follows:

(1) The following indicators calculated per 1,000 patients in the TS (Fig. 1):

(a) Number of true positives (TPs) and false positives (FPs),

(b) Number of false-negative (FN) results in CCTA (FN1),

(c) Number of true-negative (TN) results in CCTA (TN1),

(d) Number of false-negative (FN) results in FFRCT (FN2),

(e) Number of true-negative (TN) results in FFRCT (TN2),

(f) Number of FN (net FN) = FN1 + FN2,

(g) Number of TN (net TN) = TN1 + TN2,

(2) Net sensitivity and net specificity (net SEN and net SP) [16],

(3) Positive predictive value (PPV) = post-test probability (positive results),

(4) Negative predictive value (NPV),

(5) Post-test probability (post-TP [negative results]) [17],

(6) Diagnostic accuracy (DA),

(7) Diagnostic odds ratio (DOR),

(8) Number needed to diagnose (NND) [18].

Fig. 1

Decision tree model (two-stage and the other five strategies). CCTA: coronary CT angiography; FFRCT: fractional flow reserve derived from CCTA; CMRI: cardiac MRI; SE: stress echocardiography; CTP: CT perfusion

Definition of the other diagnostic strategies and comparison of the efficiencies using the TS

To assess the acceptability of TS, we defined the following diagnostic strategies and evaluated their efficiencies. Subsequently, the obtained efficiency parameters were compared with those of the TS.

(1) Simultaneous strategy

As DSCCF differs from TS, a diagnostic strategy was used in performing FFRCT in all patients undergoing CCTA. In this strategy, the test result was defined as follows: if any result of CCTA and FFRCT, or both, was positive, the final result was considered positive, and if both CCTA and FFRCT were negative, the final result was considered negative.

(2) CCTA-only strategy

This strategy was defined as the performance of only CCTA.

(3) Diagnostic strategies combining CCTA with other modalities

We proposed a diagnostic strategy combining CCTA with the existing non-invasive functional imaging modality. The subject of the evaluation was a diagnostic strategy that combined CCTA with the following imaging modalities currently used to detect myocardial ischemia:

(a) Cardiac magnetic resonance imaging (CMRI: stress perfusion CMRI).

(b) Single-photon emission computed tomography (SPECT).

(c) Positron emission computed tomography (PET).

(d) Stress echocardiography (SE).

(e) CT perfusion (CTP).

It was assumed that each modality was performed after CCTA in patients with the same conditions as TS. The literature on diagnostic ability was investigated and selected similarly to TS. A comparison of each diagnostic strategy defined to evaluate the efficiencies is shown in Fig. 2.

Fig.2

Comparison of each diagnostic strategy. CAD: coronary artery disease. CCTA: coronary CT angiography. FFRCT: fractional flow reserve derived from CCTA

Calculation and comparison of efficiency parameters

A decision tree analysis simulation [19] was performed to assess the efficiency of the patient group. Based on the PTP and the sensitivity and specificity of CCTA and FFRCT in the literature, we calculated the final probability of reaching the end point of each branch in the decision tree (Fig. 1). Based on previous studies [6, 20], using each probability, the number of TP, FP, FN1, TN1, FN2, TN2, net FN, and net TN per 1,000 patients was calculated (Tables S1–S2). To calculate the number of patients, we followed the method published by Hsu et al. [21]. Subsequently, using TP, FP, net FN, and net TN, efficiency parameters (3–8) and 95% confidence intervals (CIs) were calculated (Tables S1–S2). In the CCTA-only strategy, the efficiency parameters were calculated similarly to those of the TS (Fig. S1 and Table S1). Using the TP, FP, FN1, and TN1, efficiency parameters (3–8) and 95% CIs were calculated (Tables S1–S2). In the simultaneous strategy, we calculated the net SEN and net SP using methods described in the literature (Table S3) [16]. Subsequently, the efficiency parameters (2–8) and 95% CIs of the simultaneous strategy were calculated using net SEN, net SP, and PTP. In the diagnostic strategy combining FFRCT with the other modalities, its efficiency parameters were calculated in the same manner as TS. Finally, we compared the efficiency parameters of TS with those of the other strategies.

Sensitivity analyses

In the basic analysis, the PTP was set to 30%. However, the PTP of CAD depends on patient background factors (such as sex, age, lifestyle habit, and the presence or absence of risk factors) [5, 6]. Therefore, we performed a sensitivity analysis to assess the efficiency of various PTPs. Changes in the efficiency of the TS were assessed at various PTPs (10–90%) centered on an intermediate PTP. In intermediate PTP, additional diagnostic tests are useful for detecting myocardial ischemia due to CAD [22]. The subject of the sensitivity analysis was exclusively the efficiency that changed in response to changes in the PTP. The change in efficiency parameters of the strategies (a–e) was also assessed for various PTPs; subsequently, these were compared with those of the TS.

Calculation of each efficiency and statistical analysis

In the comparison of efficiency parameters (2–8), the difference in the point estimated value was considered significant if there was no overlap in the 95% CIs. For statistical analysis and calculation of each efficiency parameter, R version 3.6.1 (R Foundation for Statistical Computing, Vienna, Austria, package: epiR) was used. For decision analysis, Microsoft Excel for Mac 2021 version 16.54 (Microsoft Corp., Redmond, WA, USA) was used.

Results

Literature search and articles selected for analysis

In the initial selection, we extracted eight, nine, six, four, one, two, and three articles on CCTA, FFRCT, CMRI, SPECT, PET, SE, and CTP, respectively (Table S4). To select articles for the analysis, three articles for each modality were extracted [23,24,25,26,27,28,29,30,31,32]. Among them, the articles by Celeng et al. regarding CCTA [25], Zhou et al. regarding FFRCT [23], Pontone et al. regarding CMRI and SE [28], Knuuti et al. regarding SPECT and PET [31], and Celeng et al. regarding CTP [25] satisfied the inclusion criteria. The patients’ characteristics are presented in Table 1.

Table 1 List of candidate articles and their characteristics

Efficiency of the TS, comparison of efficiency parameters between the TS and the simultaneous strategy, and the TS and the CCTA-only strategy

The efficiency parameters of TS and each strategy in the basic setting (PTP = 30%) are listed in Table 2. Comparing TS with the simultaneous strategy, the TP and FP of TS were lower, whereas the FN, TN, and DOR of TS were higher than those of the simultaneous strategy. The net SP, PPV, post-TP (negative result), and DA of TS were significantly higher, whereas the net SEN, NPV, and NND of TS were significantly lower. Comparing TS with the CCTA-only strategy, the TP and FP of the TS were lower, whereas the FN and TN were higher than in the CCTA-only strategy. The net SP, PPV, DA, and DOR of TS were considerably higher, whereas the net SEN and NND of TS were significantly lower.

Table 2 Comparison of efficiencies of the three types of strategies in the basic setting (PTP = 30%)

Comparison of efficiency parameters between the TS and other diagnostic strategies

The efficiency parameters of TS and other diagnostic strategies in the basic setting (PTP = 30%) are listed in Table 3. The calculated numbers of TP and FP ranged from 180 (SE strategy) to 254 (TS) and from 44 (CMRI strategy) to 76 (CTP strategy). FN2 and TN2 ranged from 28 (TS) to 102 (SE strategy) and 288 (CTP strategy) to 320 (CMRI strategy), respectively. Net FN and TN ranged from 46 (TS) to 120 (SE strategy) and 624 (CTP strategy) to 656 (CMRI strategy), respectively. Regarding other efficiency parameters and comparisons, TS had the highest net SEN (0.85), while its net SP (0.90) was comparable to the other strategies. The PPV and DOR of the TS (0.79 and 50.50, respectively) were higher than those of the SPECT, SE, and CTP strategies and lower than those of the CMRI and PET strategies. The NPV, post-TP (negative result), DA, and NND of TS (0.93, 0.07, 0.89, and 1.34, respectively) were comparable to those of the CMRI and PET strategies. We also confirmed significant differences in the net SEN, NPV, post-TP (negative results), DOR, and NND between the TS, SPECT, and SE strategies and in the DA between the TS and SE strategies.

Table 3 Comparison of efficiencies between TS and other diagnostic strategies in the basic setting (PTP = 30%)

Sensitivity analyses

The changes in the efficiency of each diagnostic strategy in the sensitivity analysis with various PTPs are shown in Figs. 3, 4 and Figure S2. The estimates of TPs, NPVs, and DAs for the TS, CMRI, and PET strategies were high for all PTPs, with no change in the sequence of the six strategies (TS > PET > CMRI > CTP > SPECT > SE). The FPs of TS were the second highest after CTP, with no change in the sequence (CTP > TS > SPECT > SE > PET > CMRI). TS had the lowest FN2s and net FNs among all PTPs, with no change in the sequence of the six strategies (TS < PET < CMRI < CTP < SPECT < SE). The TN2s and net TNs of the TS were the second lowest, with no change in the sequence (CTP < TS < SPECT < SE < PET < CMRI). The PPVs of TS were the third highest, although the difference between the six strategies was eliminated as the PTP increased (CMRI > PET > TS > SPECT > SE > CTP). TS had the third-highest DOR value. Although there was no change in the order of the six strategies (CMRI > PET > TS > CTP > SPECT > SE), there was no consistent trend of increasing or decreasing DOR with changes in PTP. The post-TP (negative results) of TS was the lowest. Additionally, at all PTPs, there was almost no difference among the TS, CMRI, and PET strategies.

Fig. 3

Sensitivity analysis 1. Changes in the numbers of TP (a), FP (b), net FN (c), and net TN results (d) for various CAD pre-test probabilities. CAD: coronary artery disease; TS: two-stage strategy; CMRI: cardiac MRI; SE: stress echocardiography; CTP: CT perfusion; TP: true positive; FP: false positive; net FN: net false negative; net TN: net true negative

Fig. 4

Sensitivity analysis 2. Changes in PPV (a), NPV (b), DA (c), DOR (d), and post-test probability (e) for various pre-test probabilities of CAD. Note: in e, upper: post-test probability (for positive results), under: post-test probability (for negative results). CAD: coronary artery disease; TS: two-stage strategy; CMRI: cardiac MRI; SE: stress echocardiography; CTP: CT perfusion; PPV: positive predictive value; NPV: negative predictive value; DA: diagnostic accuracy; DOR: diagnostic odds ratio

Discussion

We evaluated the efficiencies of the TS DSCCF, which performs from CCTA to FFRCT, for the detection of myocardial ischemia, and compared them with those obtained using other diagnostic strategies. Sensitivity analyses at various PTPs were also performed. Our results indicated that the TS has good efficiencies. In the basic setting, compared with the simultaneous and CCTA-only strategies, the TS showed a considerable reduction in FP, a large increase in TN, and significant increases in net SP, PPV, and DA. Compared with the simultaneous strategy, although the TS showed a moderate increase in FN and a significant decrease in net SEN, there was also a considerable reduction in FP, a large increase in TN, a significant decrease in NND, and significant increases in net SP, PPV, and DA. Therefore, if the clinical situation is consistent with the recommendations of guidelines for appropriate use, it is considered reasonable to add FFRCT only for patients who are CCTA positive, and it is considered inappropriate to add FFRCT for all patients undergoing CCTA. The exception to this is that it might be acceptable to add FFRCT to all patients undergoing CCTA if the physician determines that a higher sensitivity and NPV are necessary to minimize the risk of missed myocardial ischemia. Compared with the CMR and PET strategies, the TS had a slightly lower net SP, PPV, and DOR due to a slightly higher FP. However, although we did not obtain evidence of statistical equivalence, point estimates of the TP, net SEN, NPV, post-TP, and DA of the TS were almost the same as for the other two strategies. Furthermore, there was no significant difference in NND. Except for FP and TN, each efficiency of the TS was almost the same or more superior than those of the SPECT, SE, and CTP strategies. Moreover, the results of each sensitivity analysis did not reveal any evidence to deny the findings in the basic setting. Therefore, regarding efficiency, it is conceivable that adding FFRCT is appropriate for patients with significant coronary artery stenosis using CCTA.

In the non-invasive diagnostic imaging modalities, past studies reported the efficiency of detecting myocardial ischemia with stable chest pain using economic analyses such as cost-effectiveness and cost-utility analyses [33,34,35,36]. However, interpreting the efficiency indicators derived from these results requires some expertise. In this study, we obtained the integrated sensitivity and specificity of TS in detecting myocardial ischemia, and based on this, we were able to clarify the efficiency of TS using various indexes. These efficiency parameters may be more useful than the sensitivity specificity of CCTA and FFRCT alone in understanding the characteristics of the diagnostic performance of TS. Moreover, based on the PTP estimated from the interview results and basic tests, using our findings, the physician can determine the degree of accuracy or inaccuracy in detecting myocardial ischemia in the TS and each diagnostic strategy in terms of percentage or number of patients. Furthermore, for physicians and medical professionals associated with the examination, our findings may be useful from the perspective of understanding the characteristics of the diagnostic performance of TS.

In a previous meta-analysis, Tan et al. [37] assessed the diagnostic performance of strategies involving the combination of CCTA and FFRCT, reporting that the pooled sensitivity and specificity were 0.99 and 0.16, respectively, when either CCTA or FFRCT was positive and 0.8 and 0.81, respectively, when both were positive. Pooled DOR and DA were 12.6 and 0.54 (either) and 17.6 and 0.81 (both), respectively. Our values were the same as or greater than those reported by Tan et al. However, such a comparison should consider that the calculation models differed between studies in terms of treating positive and negative test results when calculating each efficiency. As reported by Tan et al., the TS may potentially reduce the transition of non-diseased patients to invasive testing, decrease medical resource wastage, and reduce the prognostic risk of invasive procedures.

Considering our findings from another perspective, each calculated efficiency parameter may be easier to understand for patients without medical expertise than indicators such as sensitivity and specificity. In particular, the number of TP, FP, net FN, and net TN per 1,000 patients, as well as PPV, NPV, and DA, is considered more indicative of the test’s strengths, weaknesses, and ability to distinguish the presence instead of the absence of lesions. If physicians or medical staff conducting the test can use these efficiency parameters to provide patients with information on diagnostic ability, our results may also contribute to the smoother implementation of informed consent [14]. Additionally, regarding diagnostic ability, these may provide the evidence needed to evaluate the validity of using DSCCF to detect myocardial ischemia. Furthermore, this research methodology can potentially be extended to other areas of research that assess the efficiency and effectiveness of diagnostic techniques using a combination of multiple tests.

Our study has several limitations. First, each index calculated as the efficiency was calculated by a simulation based on data obtained from each article. Therefore, our results may not be appropriate in some situations. Second, the diagnostic ability of each modality for calculating efficiency was obtained from meta-analysis studies. Owing to the nature of the data obtained from the literature, patient background characteristics for each modality in the calculations and efficiency comparisons were not similar. Furthermore, differences in diagnostic ability based on sex [10] were not considered. Therefore, comparisons of efficiency parameters were confined to point estimates and their 95% CIs without statistical significance tests. To cope with these problems, we conducted a comprehensive literature search and restricted our inclusion criteria when selecting literature for analysis. However, some bias may have occurred. Thus, to overcome these problems, further evaluations based on large-scale real-world data with more consideration of patient background are needed.

The TS is considered a good diagnostic strategy that can obtain information on the coronary artery and myocardial perfusion within one test without additional contrast agent administration, radiation exposure, and additional tests [10]. Furthermore, by adding FFRCT to CCTA, TS can reduce the FP results to almost equal and increase the TN results equivalently compared with other strategies. Moreover, it is also expected to reduce the unnecessary psychological burden caused by FP results [38] as well as the physical and economic burden caused by the reduction of additional examinations such as invasive coronary angiography. Previous studies have shown a reduction in invasive evaluations in most patients with negative FFRCT results. Furthermore, a positive impact was noted in decision-making on patient management following the FFRCT implementation [1, 39]. Although limited to testing and diagnostic opportunities, our study clarified the efficiency of TS in detecting myocardial ischemia. By comparing the efficiency of TS with other diagnostic strategies using existing diagnostic modalities, we also clarified that TS has good efficiency. To further validate the usefulness of TS, our findings should be applied to actual clinical practice, and the results should be evaluated. Finally, the following points should be considered when considering the implementation of TS: (1) The overall efficiency of TS depends on the diagnostic ability of CCTA; thus, patients who are actually ischemia-negative will be diagnosed positive on CCTA and undergo FFRCT (Table 2); and (2) Patient factors such as obesity, heart rate variability, and vascular calcification significantly affect the image quality of CCTA. Furthermore, FFRCT cannot currently be performed in patients with a history of coronary stenting or coronary artery bypass surgery [40].

Conclusions

TS is more efficient in detecting myocardial ischemia than other diagnostic strategies. Therefore, DSCCF with TS evaluation is considered appropriate. In addition, elucidating the diagnostic ability of a test with various efficiency indexes might help physicians, patients, and medical professionals conducting the examination understand its characteristics.