Uncertainty in evidence underpinning health technology assessment (HTA) submissions was a key theme at ISPOR Europe 2024, clear already from the first plenary session at the conference looking at the “Evidence-Price Conundrum”.1 The panel discussion covered increasing reliance on evidence other than the gold-standard randomised control trial (RCT), such as single-arm trials, in the context of reduced regulatory thresholds where there is a specific unmet need, or pressure to reduce time to access. However, another key theme of the conference, the upcoming EU Joint Clinical Assessment (JCA), created a new context for evidence requirements, where earlier and more comprehensive consideration of evidence synthesis is likely needed. Given increasing uncertainty in evidence at the point of HTA and a perception of increasing importance of robust comparative efficacy evidence, many sessions at ISPOR focused on advancement of indirect treatment comparison (ITC) methods to address uncertainty and bias in comparative effectiveness analysis.
An ever-present requirement of ITCs for HTA is to meet (or account for not meeting) the underlying assumptions of similarity, homogeneity and consistency.2, 3 Advanced methods to address this presented at the conference included multi-level network meta-regression (ML-NMR) specifically for time-to-event (TTE) outcomes, as well as external control arms (ECA) for single-arm trials and accompanying methods used for sensitivity analysis.4
ML-NMR extends the established population adjustment methods in ITCs, such as matching adjusted indirect comparison (MAIC) and simulated treatment comparison (STC), used in the case of the similarity assumption being broken in a network of studies.5 A neat method to combine trial individual patient data (IPD) with aggregate data in a connected network of studies, ML-NMR can address bias arising from imbalances in treatment effect modifiers between studies. It was noted that case studies to date have applied this method only to binomial outcomes, but a more recently proposed method allows for application of ML-NMR to TTE outcomes (including R and Stan code, using the multinma R package).6, 7 It was discussed that ML-NMR was successful in reducing bias compared to pairwise MAICs in a case study using a TTE outcome.4 There is large potential for the use of this method, given the common situation where several treatments are approved but patient populations differ in terms of effect modifying characteristics, particularly in oncology where overall and progression-free survival are key endpoints.
Several sessions also explored ECA studies for single-arm trials using real world evidence (RWE). One forum at the conference highlighted a 13-fold increase in the use of single-arm trials in HTA submissions between 2011–2019, and the challenges associated with these;8, 9 while ECA studies for single arm trials are not substitutes for RCTs (where pragmatic RCT designs such as adaptive trial design and platform trials should typically also be considered first), methods to ensure ECA studies can be robustly conducted for HTA purposes are important. It was discussed that these can be integrated at an early stage in the evidence generation plan for a product, which will now also be shaped by JCA requirements including the eventual need for multiple PICOs.8
Another presentation dug deeper into quantifying bias in ECA studies, centred around a review of National Institute of Health and Care Excellence (NICE) submissions that used ECA studies using RWE as the source of comparative effectiveness and that explored sensitivity analyses as proposed in the NICE RWE framework.4, 10 Sensitivity analyses included varying the data curation method (e.g. with alternative eligibility criteria) and adjustment method, or conducting quantitative bias analysis (QBA).10 Notably, the review found that QBA was not commonly used, and no clear trend on the impact of the use of sensitivity analyses on reimbursement was identified, perhaps reflecting the inherent barriers to reimbursement arising from unanchored comparisons of clinical trial data to real-world data. Nevertheless, given the numerous circumstances where single-arm trials and ECA studies are the only real option, well-established sensitivity analyses to directly address uncertainty in ECA studies will help to explicitly identify and discuss sources of bias including unmeasured confounding, and ultimately ensure comparative effectiveness evidence is sufficiently robust for HTA decision making.
An increased focus on comparative effectiveness during evidence generation planning is a welcome development for those working in HTA. The advancements in ITC methods such as the increased use of ML-NMR, including for TTE outcomes, and arsenal of sensitivity analyses available for ECA studies should serve as useful options of addressing uncertainty and navigating the challenges anticipated in HTA submissions.
Footnotes
Similarity: For included studies, potential treatment effect modifiers should be similar across studies; for a robust approach, potential treatment effect modifiers should be identified from a literature search, input from healthcare professionals and, with caution, subgroup results (given potential issues with power and preserving randomisation).
Homogeneity: There should be no meaningful heterogeneity between the effect estimates of each possible comparisons (which may be caused by unknown treatment effect modifiers), as shown by I^2 values for pairwise comparisons.
Consistency: Indirect and direct evidence should concur, which can be tested via methods including the Bucher two-step method or fitting inconsistency models.
QBA: Can be used to examine the extent to which bias would have to be present to change results, or to estimate the direction, magnitude and uncertainty of bias associated with measures of effect.
References
Podium Session 109. Addressing Uncertainty and Bias in Comparative Effectiveness Analysis., In ISPOR Europe, Barcelona, Spain, 2024.
If you would like any further information on the themes presented above, please do not hesitate to contact Andrei Karlsson, Senior Statistician (LinkedIn) or Alex Porteous, Deputy Head of HTA (LinkedIn). Andrei Karlsson and Alex Porteous are employees at Costello Medical. The views/opinions expressed are their own and do not necessarily reflect those of Costello Medical’s clients/affiliated partners.
