By Kae Jansen van Vuuren
There is nothing more arduous than manual data entry, but it is unfortunately a key process in death classifications and recording overdose deaths. Across the world, there is a rising rate of drug-related deaths, which need swift and accurate surveillance to inform public health decisions, but overdose data is buried in free-text coroner reports, which means logging these deaths and coding them with ICD-10 classifications (the international codes for every disease) slow and labour-intensive – resulting in delays and information loss. So now, we turn to computers and natural language processors (NLPs)
Natural language processing is a field of artificial intelligence that enables computers to understand, interpret, generate, and manipulate human language in a meaningful way. Right now, we have many decoder-only large language models (such as everyone’s favourite: ChatGPT) but implementing this in personnel and resource limited settings is difficult because of the high compute and fiscal cost. But we also have encoder-only models, like Bidirectional Encoder Representations from Transformers (BERT) models, which may be more suitable for these settings, considering their high accuracy, lower complexity, and lower cost.
The researchers looking into this solution, found that fine-tuned BioClinicalBERT models achieved near-perfect performance, outperforming conventional machine learning, general-domain BERT models, and various decoder-only LLMs.
How did they do this?
The researchers collected death certificate data from various states in the USA between Jan and Dec 2020 and created 9 categories of drugs that resulted in overdose. This was the main dataset. They also got a second external validation dataset from oct 2023 to April 2024, using different states. They went through each case and labelled it in a binary method (yes or no) for each drug category and if it contributed to the person’s death.
Then they tried four different modelling methods:
- Single-label classification: A type of AI that chooses one category from a list for each piece of text
- Multi-label classification: A type of AI that can assign multiple categories to the same piece of text if more than one type applies
- Fine-tuned encoder-only Bert models: An AI that has been specially trained to become better at understanding text and sorting it into the correct categories
- Fine-tuned decoder-only LLMs: An LLM, like ChatGPT, that has been further trained to perform a specific task while still being able to understand and generate natural language.
What did they find?
Fine-tuned BioClinicalBERT was the most accurate model for identifying drugs involved in overdose deaths from free-text medical examiner reports, outperforming most other MLMs and LLMs, achieving near-perfect accuracy. It was also very effective at recognising cases involving multiple drugs, making it great for real-world overdose investigations. It was also able to capture rare or emerging substances, providing both precision and completeness,
What does this mean
These models could enable coroner offices to clear the backlog of cases and enable up-to-date reporting on overdose mortality, overcoming delays associated with manual ICD-10 coding and enhancing timely response to emerging drug trends.
Newer LLMs (such as ChatGPT) are most likely to perform better in the future, but currently the cost associated with running them are too great – making a lower cost model such as BioClinicalBERT a valuable tool to assist coroner offices, offering an efficient and scalable solution for accelerating overdose surveillance efforts across jurisdictions
Things needed for the future
The current model may not capture variation in language across different regions in the USA or in other countries, or in places that may use different reporting systems, so for the future, more diverse datasets for training and evaluation of the model need to be implemented study evaluated retrospective data and needs to be battle-tested.
Funnell AJ, Petousis P, Harel-Canada F, Romero R, Bui AAT, Koncsol A, et al. Improving drug identification in overdose death surveillance by using clinical natural language processing models. J Forensic Sci. 2026; 71: 1329–1337. https://doi.org/10.1111/1556-4029.70281
Leave a Reply