Language selection

Government of Canada / Gouvernement du Canada

Search

MSLC24: further challenges for metrics on a wide landscape of translation quality

From National Research Council Canada

Download
  1. (PDF, 3.9 MiB)
Linkhttps://aclanthology.org/2024.wmt-1.34
AuthorSearch for: 1ORCID identifier: https://orcid.org/0000-0002-1647-584X; Search for: 1ORCID identifier: https://orcid.org/0009-0000-6147-9631; Search for: 1ORCID identifier: https://orcid.org/0000-0001-8714-7846
Affiliation
  1. National Research Council Canada. Digital Technologies
FormatText, Article
ConferenceNinth Conference on Machine Translation (WMT24), November 15-16, 2024, Miami, Florida, USA
Abstract
In this second edition of the Metric Score Landscape Challenge (MSLC), we examine how automatic metrics for machine translation perform on a wide variety of machine translation output, ranging from very low quality systems to the types of high-quality systems submitted to the General MT shared task at WMT. We also explore metric results on specific types of data, such as empty strings, wrong- or mixed-language text, and more. We raise several alarms about inconsistencies in metric scores, some of which can be resolved by increasingly explicit instructions for metric use, while others highlight technical flaws.
Publication date
PublisherAssociation for Computational Linguistics
Licence
In
  • Proceedings of the Ninth Conference on Machine Translation (WMT24) (15 November 2024): 475–491.
LanguageEnglish
Peer reviewedYes
Export citationExport as RIS
Report a correctionReport a correction (opens in a new tab)
Record identifier888f2024-592a-4b55-a841-52f62072f3f3
Record created2024-11-14
Record modified2024-11-15
Date modified: