The U.S. Food and Drug Administration (FDA) is responsible for protecting the public health by assuring the safety, efficacy, and security of all FDA-regulated products, including human and veterinary drugs, prescription and over-the-counter pharmaceutical drugs, vaccines, biopharmaceuticals, blood transfusions, biological products, and others. FDA and the U.S. National Library of Medicine (NLM) have been working together on transforming the content of Structured Product Labeling (SPL) documents for prescription drugs into discrete, coded, computer-readable data that will be made available to the public in individual SPL index documents. Transforming the narrative text to structured information encoded in national standard terminologies is a prerequisite to the effective deployment of drug safety information. Being able to electronically access labeling information and to search and sort that information is an important step toward the creation of a fully automated health information exchange system. TAC 2017 addressed one of the important drug safety issues: automated extraction of adverse drug reactions (ADRs) reported in SPLs. An equally important and complex task is automated extraction of drug-drug interaction information. Drug-drug interactions can lead to a variety of adverse events, and it has been suggested that preventable adverse events are the eighth leading cause of death in the United States.
The results of this TAC track will inform future FDA efforts at automating important safety processes, and could potentially lead to future FDA collaboration with interested researchers in this area.
The purpose of this TAC track is to test various natural language processing (NLP) approaches for their information extraction (IE) performance on drug-drug interactions in Structured Product Labeling (SPL) documents. SPL is a document markup standard approved by Health Level Seven (HL7) and adopted by the FDA as a mechanism for exchanging product and facility information about drugs. For more information about TAC, please visit https://tac.nist.gov/about/index.html.
The participants may choose any one specific task described below or approach the tasks as each one building upon the previous tasks. Some tasks do necessarily require the output of previous tasks, e.g., Task 2 requires Task 1.
Any resources, e.g., the UMLS© Terminology Services, may be used to aid with the normalization process.
The participants are provided with the following data for training:
These labels contain gold standard annotations created by NLM and FDA. An additional set of at least 50 drug labels will be provided as the official test set in the same format. The annotations in the training set were generated semi-automatically and might be missing some interactions. The automatically extracted entities and relations in these sentences were manually corrected by the FDA experts and NLM volunteers using these guidelines (schematic presentation courtesy of Mark Sharp.)
The ultimate aim is to know which interactions are in the labels, not the precise offsets or relations, such that the interactions may be linked to structured knowledge sources. Further, an interaction mentioned several times should not necessarily carry more weight than an interaction mentioned once. As such, the gold standard contains a list of unique interaction aggregated at the document level.
These interactions are mapped as follows:
The following datasets may be used for training and are available for immediate download:
By contrast, the test data will become available in August, 2019.
The evaluation measures are:
|Task 1||Precision/Recall/F1-measure on entity-level annotations, using both partial and exact matching.||Micro-averaged F1 on exact matches.|
|Task 2||Precision/Recall/F1-measure on relations.||Micro-averaged F1.|
|Task 3||Precision/Recall/F1-measure on unique Interactions.||Macro-averaged F1 (by label)|
|Task 4||Precision/Recall/F1-measure on unique Interactions.||Macro-averaged F1 (by label)|
The official evaluation script will be used to calculate these scores, and will be provided soon.
Participants are allowed three separate submissions. Submissions that do not conform to the provided XML standards will be rejected without consideration.
We are keeping the ADR mailing list for all drug (label) related evaluations: