| Download | - View final version: HardEval: focusing on challenging tokens to assess robustness of NER (PDF, 333 KiB)
|
|---|
| Link | https://aclanthology.org/2020.lrec-1.211/ |
|---|
| Author | Search for: Bernier-Colborne, Gabriel1; Search for: Langlais, Philippe1 |
|---|
| Affiliation | - National Research Council Canada. Digital Technologies
|
|---|
| Format | Text, Article |
|---|
| Conference | The 12th Language Resources and Evaluation Conference, LREC 2020, May 11–16, 2020, Marseille, France |
|---|
| Subject | named entity recognition; natural language processing; evaluation; robustness |
|---|
| Abstract | To assess the robustness of NER systems, we propose an evaluation method that focuses on subsets of tokens that represent specific sources of errors: unknown words and label shift or ambiguity. These subsets provide a system-agnostic basis for evaluating specific sources of NER errors and assessing room for improvement in terms of robustness. We analyze these subsets of challenging tokens in two widely-used NER benchmarks, then exploit them to evaluate NER systems in both in-domain and out-of-domain settings. Results show that these challenging tokens explain the majority of errors made by modern NER systems, although they represent only a small fraction of test tokens. They also indicate that label shift is harder to deal with than unknown words, and that there is much more room for improvement than the standard NER evaluation procedure would suggest. We hope this work will encourage NLP researchers to adopt rigorous and meaningful evaluation methods, and will help them develop more robust models. |
|---|
| Publication date | 2020-05 |
|---|
| Publisher | European Language Resources Association |
|---|
| Licence | |
|---|
| In | |
|---|
| Language | English |
|---|
| Peer reviewed | Yes |
|---|
| Export citation | Export as RIS |
|---|
| Report a correction | Report a correction (opens in a new tab) |
|---|
| Record identifier | fd2cf5ab-a739-4b7d-bd16-4edbf3666d42 |
|---|
| Record created | 2020-11-02 |
|---|
| Record modified | 2026-01-26 |
|---|
