| Téléchargement | - Voir la version finale : HardEval: focusing on challenging tokens to assess robustness of NER (PDF, 333 Kio)
|
|---|
| Lien | https://www.aclweb.org/anthology/2020.lrec-1.211/ |
|---|
| Auteur | Rechercher : Bernier-Colborne, Gabriel1; Rechercher : Langlais, Philippe1 |
|---|
| Affiliation | - Conseil national de recherches du Canada. Technologies numériques
|
|---|
| Format | Texte, Article |
|---|
| Conférence | The 12th Language Resources and Evaluation Conference, LREC 2020, 11–16 May 2020, Marseille, France |
|---|
| Sujet | named entity recognition; natural language processing; evaluation; robustness |
|---|
| Résumé | To assess the robustness of NER systems, we propose an evaluation method that focuses on subsets of tokens that represent specific sources of errors: unknown words and label shift or ambiguity. These subsets provide a system-agnostic basis for evaluating specific sources of NER errors and assessing room for improvement in terms of robustness. We analyze these subsets of challenging tokens in two widely-used NER benchmarks, then exploit them to evaluate NER systems in both in-domain and out-of-domain settings. Results show that these challenging tokens explain the majority of errors made by modern NER systems, although they represent only a small fraction of test tokens. They also indicate that label shift is harder to deal with than unknown words, and that there is much more room for improvement than the standard NER evaluation procedure would suggest. We hope this work will encourage NLP researchers to adopt rigorous and meaningful evaluation methods, and will help them develop more robust models. |
|---|
| Date de publication | 2020-05 |
|---|
| Maison d’édition | European Language Resources Association |
|---|
| Licence | |
|---|
| Dans | |
|---|
| Langue | anglais |
|---|
| Publications évaluées par des pairs | Oui |
|---|
| Exporter la notice | Exporter en format RIS |
|---|
| Signaler une correction | Signaler une correction (s'ouvre dans un nouvel onglet) |
|---|
| Identificateur de l’enregistrement | fd2cf5ab-a739-4b7d-bd16-4edbf3666d42 |
|---|
| Enregistrement créé | 2020-11-02 |
|---|
| Enregistrement modifié | 2021-09-17 |
|---|
