| アブストラクト | BACKGROUND: Causality assessment is central to pharmacovigilance but remains resource-intensive and subjective. The applicability of large language models (LLMs) to formal World Health Organization-Uppsala Monitoring Centre (WHO-UMC) drug-adverse event causality assessment has not been well established. OBJECTIVE: This study aims to evaluate the performance of LLMs in WHO-UMC causality assessment. METHODS: A curated set of 55 cases derived from the US Food and Drug Administration Adverse Event Reporting System, comprising 337 drug-level assessments, was constructed. Cases involving 2 to 11 suspected drugs were stratified by drug count, and 5 cases were sampled from each stratum. To ensure representation of rare but clinically important categories, 5 additional cases containing at least 1 "Certain" drug-adverse event pair were included. Case data were reorganized into a standardized semistructured format that preserved key elements required for WHO-UMC causality assessment. Domain experts conducted a pilot evaluation to align interpretation criteria prior to independently assessing the final dataset, yielding an interexpert agreement (Fleiss kappa) of 0.762 across 337 drug-level assessments. Multiple prompting strategies, including standard prompting, chain-of-thought (CoT), CoT with self-consistency, few-shot, reasoning and acting, and tree-of-thought prompting, were applied across multiple LLMs, including GPT-5.4 and its mini variant and Gemini 2.5 Flash and Pro, via their respective application programming interfaces. Agreement with expert assessments was quantified using Cohen kappa, weighted kappa, and accuracy metrics. Internal consistency across repeated inferences was evaluated using Fleiss kappa. RESULTS: Performance varied across models and prompting strategies. Cohen kappa ranged from 0.368 to 0.641, weighted kappa ranged from 0.641 to 0.821, accuracy ranged from 0.583 to 0.804, and balanced accuracy ranged from 0.513 to 0.735. Fleiss kappa ranged from 0.730 to 0.915, corresponding to substantial to almost perfect agreement. The highest Cohen kappa was observed for Gemini 2.5 Flash with CoT prompting (0.641). Gemini 2.5 Flash with CoT-self-consistency prompting showed a Cohen kappa of 0.640 and achieved the highest observed point estimates for weighted kappa (0.821), accuracy (0.804), and Fleiss kappa (0.915), although the gains over other prompting strategies were modest. Category-level performance for this model showed higher performance for "Certain" (F(1)-score=0.793), "Probable/Likely" (F(1)-score=0.794), and "Unlikely" (F(1)-score=0.898), whereas performance for "Possible" remained substantially lower (F(1)-score=0.293), reflecting the difficulty of intermediate causality assessment. CONCLUSIONS: LLMs demonstrated moderate to substantial agreement in WHO-UMC causality assessment, indicating meaningful but still limited performance relative to expert judgment. Although LLMs are not suitable for independent decision-making, they may serve as supportive tools in pharmacovigilance workflows, particularly for preliminary case triage. Further studies using larger and more diverse datasets and evaluating performance on raw narrative reports are warranted. |
|---|
| ジャーナル名 | Journal of medical Internet research |
|---|
| Pubmed追加日 | 2026/7/8 |
|---|
| 投稿者 | Ha, Young Mi; Kim, Minjung; Bang, YoungIn; Choi, Daejin; Kim, Jae Hyun; Rhie, Sandy Jeong; Noguchi, Yoshihiro; Kim, Myeong Gyu |
|---|
| 組織名 | Graduate School of Pharmaceutical Sciences, Ewha Womans University, Seoul,;Republic of Korea.;College of Pharmacy, Ewha Womans University, Seoul, Republic of Korea.;College of Artificial Intelligence, Ewha Womans University, Seoul, Republic of;Korea.;Human-Centered Artificial Intelligence Research Institute, Ewha Womans;University, Seoul, Republic of Korea.;School of Pharmacy and Institute of New Drug Development, Jeonbuk National;University, Jeonju, Republic of Korea.;Laboratory of Clinical Pharmacy, Gifu Pharmaceutical University, Gifu, Japan. |
|---|
| Pubmed リンク | https://www.ncbi.nlm.nih.gov/pubmed/42418253/ |
|---|
