| DOI | Resolve DOI: https://doi.org/10.18653/v1/2020.bionlp-1.19 |
|---|
| Author | Search for: Nejadgholi, Isar1; Search for: Fraser, Kathleen C.1; Search for: De Bruijn, Berry1 |
|---|
| Affiliation | - National Research Council Canada. Digital Technologies
|
|---|
| Format | Text, Article |
|---|
| Conference | 19th SIGBioMed Workshop on Biomedical Language Processing, July 9, 2020, Online |
|---|
| Abstract | When comparing entities extracted by a medical entity recognition system with gold standard annotations over a test set, two types of mismatches might occur, label mismatch or span mismatch. Here we focus on span mismatch and show that its severity can vary from a serious error to a fully acceptable entity extraction due to the subjectivity of span annotations. For a domain-specific BERT-based NER system, we showed that 25% of the errors have the same labels and overlapping span with gold standard entities. We collected expert judgement which shows more than 90% of these mismatches are accepted or partially accepted by the user. Using the training set of the NER system, we built a fast and lightweight entity classifier to approximate the user experience of such mismatches through accepting or rejecting them. The decisions made by this classifier are used to calculate a learning-based F-score which is shown to be a better approximation of a forgiving user's experience than the relaxed F-score. We demonstrated the results of applying the proposed evaluation metric for a variety of deep learning medical entity recognition models trained with two datasets. |
|---|
| Publication date | 2020-07 |
|---|
| Publisher | Association for Computational Linguistics |
|---|
| In | |
|---|
| Language | English |
|---|
| Peer reviewed | Yes |
|---|
| Export citation | Export as RIS |
|---|
| Report a correction | Report a correction (opens in a new tab) |
|---|
| Record identifier | 1fb0858f-4b47-4b40-ae47-8d92941dfe48 |
|---|
| Record created | 2020-11-13 |
|---|
| Record modified | 2022-02-21 |
|---|
