| DOI | Trouver le DOI : https://doi.org/10.18653/v1/2020.bionlp-1.19 |
|---|
| Auteur | Rechercher : Nejadgholi, Isar1; Rechercher : Fraser, Kathleen C.1; Rechercher : De Bruijn, Berry1 |
|---|
| Affiliation | - Conseil national de recherches du Canada. Technologies numériques
|
|---|
| Format | Texte, Article |
|---|
| Conférence | 19th SIGBioMed Workshop on Biomedical Language Processing, July 9, 2020, Online |
|---|
| Résumé | When comparing entities extracted by a medical entity recognition system with gold standard annotations over a test set, two types of mismatches might occur, label mismatch or span mismatch. Here we focus on span mismatch and show that its severity can vary from a serious error to a fully acceptable entity extraction due to the subjectivity of span annotations. For a domain-specific BERT-based NER system, we showed that 25% of the errors have the same labels and overlapping span with gold standard entities. We collected expert judgement which shows more than 90% of these mismatches are accepted or partially accepted by the user. Using the training set of the NER system, we built a fast and lightweight entity classifier to approximate the user experience of such mismatches through accepting or rejecting them. The decisions made by this classifier are used to calculate a learning-based F-score which is shown to be a better approximation of a forgiving user's experience than the relaxed F-score. We demonstrated the results of applying the proposed evaluation metric for a variety of deep learning medical entity recognition models trained with two datasets. |
|---|
| Date de publication | 2020-07 |
|---|
| Maison d’édition | Association for Computational Linguistics |
|---|
| Dans | |
|---|
| Langue | anglais |
|---|
| Publications évaluées par des pairs | Oui |
|---|
| Exporter la notice | Exporter en format RIS |
|---|
| Signaler une correction | Signaler une correction (s'ouvre dans un nouvel onglet) |
|---|
| Identificateur de l’enregistrement | 1fb0858f-4b47-4b40-ae47-8d92941dfe48 |
|---|
| Enregistrement créé | 2020-11-13 |
|---|
| Enregistrement modifié | 2022-02-21 |
|---|
