In today’s fast-evolving landscape of systematic literature reviews (SLRs), the promise for innovative methods to enhance efficiency and accuracy is huge. We want to harness this potential in a way that maximises the benefits without compromising the quality and integrity of our work. In our Literature Reviews team at Costello Medical, following on from our previous thought piece on the evolving industry landscape, we are preparing by conducting rigorous, evidence-based testing to evaluate where AI will have the biggest impact and identify the likely pitfalls.
One of our ongoing initiatives focuses on developing sophisticated AI prompts for LLM (large language model)-assisted data extraction from studies underpinning the robust evidence base for HTA (Health Technology Assessment) submissions, like randomised controlled trials (RCTs), economic evaluations and health-related cost and resource use studies. We are putting the technology through its paces and aiming to use it to extract all of the typical data required in SLR extractions. This includes complex concepts with heterogeneously defined outcomes, e.g. subgroup analyses and supplementary materials, alongside simpler information like sample size and study design. In a much-needed approach, the prompts we are developing are being validated on a separate set of new, unseen data, to test and increase their generalisability. While not equal to that of a human, we are seeing some cases where performance is “good” or even “excellent” in terms of F1 score (a balance of precision and recall). Concomitantly, it is helping us pinpoint areas where the models are simply not there yet. The results are promising and a more distant future may see AI replacing a human reviewer for dual extractions, however we can confidently say that we will not be giving the robots free reign any time soon.
In parallel, we are comparing the use of a machine learning classifier against a traditional RCT search filter in a real-life SLR context. We want to test whether the classifier performs better or worse in metrics like sensitivity and specificity, time and user experience.
The results from both of these research projects will be coming out soon, so watch this space!
Looking ahead, our efforts will expand into integrating AI technologies into our in-house SLR platform, which will become a one-stop-shop for the full literature review lifecycle, from protocol development through record review and data extraction, all the way to analysis and reporting of results.
Stay tuned for more updates on our website and LinkedIn pages, and collaborative opportunities for revolutionising the literature review processes.
If you would like any further information on the themes presented above, please get in touch, or visit our Literature Reviews page to learn how our expertise can benefit you. Molly Murton created this article on behalf of Costello Medical. The views/opinions expressed are her own and do not necessarily reflect those of Costello Medical’s clients or affiliated partners.
