The conference emphasised the growing trend of adopting artificial intelligence (AI) and machine learning (ML) tools, like GPT-4, across various Health Economics and Outcomes Research (HEOR) activities, such as systematic literature reviews (SLRs), economic model development, and data analysis. It was evident that these tools are gaining traction; in an early session at the conference, an audience poll found approximately 55% felt that generative AI is a useful tool and about 35% believed it would completely transform HEOR.1
Various groups reported having experimented with AI tools, with promising results.1, 2 The availability of AI/ML tools for systematic reviews is increasing. These have been applied at each stage of the review (search string generation, article screening, risk of bias assessment, and data extraction), with the use of AI/ML at the abstract screening step being most common.3-5 Additionally, case studies were presented on using GPT-4 to:
Nonetheless, multiple studies found lower accuracy in AI-generated outputs compared to human outputs,8, 9 emphasising the need for careful prompt engineering to successfully use generative AI and the importance of maintaining human review of outputs.
The National Institute for Health and Care Excellence (NICE) provided various updates on its exploration into the impact of AI and machine learning (ML) technologies for health technology assessment, including work being done through the Next Generation Health Technology Assessment (HTx) project.10 NICE has been exploring the use of AI and ML in three main categories:11
Internally, NICE has used AI and ML to automate various processes like deriving search strategies, randomised controlled trial (RCT) classification, and guidance surveillance. For example, they’ve piloted an algorithm to match recommendations from NICE and the Office for National Statistics to accelerate guidance surveillance for breast cancer screening.
In relation to company submissions, NICE is contemplating guidance updates to identify areas of high and low risk for AI use. While the use of AI and ML in submissions to NICE is currently limited, it is anticipated to increase significantly in the future.
Lastly, NICE is reviewing how health technologies incorporating AI or ML components should be appraised. Updates to the CHEERS checklist are also expected in the near future, to better suit such technologies.
While the potential of AI to transform the HEOR field is exciting, numerous challenges were also raised over the course of the conference. Among these are concerns around the accuracy and reliability of output from generative AI tools, reproducibility and transparency, and acceptance of AI outputs by health technology assessment agencies. A recurring theme at the conference was the need for retaining human review checkpoints throughout any process using AI to address some of these challenges, and there exists a need for upskilling health economists and related professionals in AI methodologies to leverage AI’s full potential and to foster trust in its outputs.
IT security and data confidentiality is another concern, particularly in relation to proprietary data inputs. Mitigation suggestions offered during the conference include avoiding AI/ML model training with publicy available models, using ring-fenced third-party models, and establishing a consistent company-wide policy on the use of AI tools. There was notably little coverage of the legal and copyright considerations around the use of AI, despite their importance, particularly in the context of processing published articles or other copyright material.
Despite these challenges, there was optimism that AI has the potential to disrupt the HEOR field in a positive way. Ongoing work is needed to identify exactly where AI provides a consistent improvement over conventional approaches and to ensure AI is implemented in a way that does not compromise on quality.
For more on the ISPOR learnings on the use of generative AI for health economic modelling specifically, please see our commentary “The Promise and Pitfalls of Generative AI for Health Economics”.
References
If you would like any further information on the themes presented above, please do not hesitate to contact , Head of Technical Innovation and Development or Hannah Luedke, Consultant (LinkedIn). Sara Steeves and Hannah Luedke are employees at Costello Medical. The views/opinions expressed are their own and do not necessarily reflect those of Costello Medical’s clients/affiliated partners.
