Artificial Intelligence in Network Meta-Analysis for HTA

While the approach described by Reason et al. automates large parts of the NMA workflow, there are significant aspects which it does not capture. For example, the study did not include a comprehensive feasibility assessment to determine which trials should be included, whether trials were of suitable similarity to compare, and which analyses should be conducted. The analyses conducted assumed that studies to be included were already identified and sufficiently homogenous for analysis via NMAs. However, this feasibility process usually consists of strategic, statistically and clinically informed decision-making which often requires collaboration between multiple stakeholders, such as statisticians, clinicians and other healthcare or HEOR professionals.

Any implicit ‘decision-making’ done by genAI is in a ‘black box’, so without transparency to the researcher. This contrasts with the standard HTA process, where each decision made needs to be clearly and strongly justified. For example, AI can be instructed to use specific NMA code (e.g. National Institute for Health and Care Excellence [NICE] recommended NMA model code). But the extent of precision in which AI could follow an analysis plan, and to what extent this would require human oversight is unclear. There may be efficiencies in AI interpretation of an analysis plan, but there is no guarantee (without oversight) that this would follow the intended instructions.

While genAI prompting may be an efficient way of completing key NMA steps, it is unclear how much time (if any) would be saved in practice for the analysis stage. Since existing NMA processes already incorporate a large degree of automation via standardized and quality-controlled template code, genAI prompting may not actually save time for this step. In contrast, data extraction and report writing are more promising ways for efficiency gains with genAI, even if we’d still fully recommend independent human modification and verification at these stages. That said, and at least initially – substantial time investments would be required to either manually adapt NMA reports generated by AI or develop detailed and case-specific prompts which ensure accuracy as well as adherence to specific style and content requirements in reporting.

Where available, national-level criteria for use of AI in HTA should be fulfilled. For many countries, there is currently a lack of guidance in HTA guidelines as to how AI fits within the analysis and review process. However, in the UK specifically, the NICE position statement regarding use of AI for evidence synthesis states that (amongst other criteria): there must be engagement with NICE regarding plans to use AI; AI should be used to augment rather than replace human input; its usage must be transparent and justified; considerations must be given towards compliance with copyright; and risks in use of AI must be reported.² More specifically focusing on NMAs, NICE has noted that the use of AI is NMAs is less established than the automation of stages in the literature review process.

Formal guidance on best practice of the use of AI in evidence synthesis should be developed and built upon. For example, the Guidelines International Network (GIN) have published a set of key principles to abide by when using AI in evidence synthesis, and Responsible AI in Evidence SynthEsis (RAISE) guidance and recommendations are being developed by a collaboration of agencies, including the Cochrane collaboration, to provide targeted guidance on the use of AI in particular evidence synthesis roles.^{3, 4}