Measuring Semantic Preservation in Machine Translation with HCOMET: Human Cognitive Metric for Evaluating Translation

75 Pages Posted: 29 Apr 2021

See all articles by João Marinotti

João Marinotti

Indiana University Maurer School of Law; Center for Intellectual Property Research, Indiana University Maurer School of Law; Information Society Project, Yale Law School

Date Written: August 20, 2014

Abstract

Human ranking of machine translation output is a commonly used method for comparing different innovations in machine translation research. Theoretically simple, the comparison of multiple translations is, in effect, cognitively complex, requiring judges to balance the weight of different types of translation errors in the context of the whole sentence. This cognitive complexity is made evident through low intra- and inter- annotator agreements, which call into question the reliability of such ranking metrics. HMEANT (Lo and Wu, 2011) attempted to decrease the complexity of ranking by dividing sentences into smaller semantic units whose translation alignments were more objective, rendering the task cognitively simpler. However, HMEANT does not discern how these semantic units are related and relies heavily on language-dependent verb frames – a significant problem for a translation metric. This project defines a new set of human metrics focusing on HCOMET (Human COgnitive Metric for Evaluating Translation). HCOMET, attempting to overcome the limitations of HMEANT, employed a new cognitively-informed annotation scheme and new scoring guidelines. While the inter-annotator agreement did not surpass that of HMEANT, the conceptual framework of HCOMET allows for a much more detailed analysis of semantic adequacy in machine translation.

Keywords: Machine Translation, NLP, Metric, Computational Linguistics

Suggested Citation

Marinotti, João, Measuring Semantic Preservation in Machine Translation with HCOMET: Human Cognitive Metric for Evaluating Translation (August 20, 2014). Available at SSRN: https://ssrn.com/abstract=3830666 or http://dx.doi.org/10.2139/ssrn.3830666

João Marinotti (Contact Author)

Indiana University Maurer School of Law ( email )

211 S. Indiana Avenue
Office #247
Bloomington, IN 47405
United States

HOME PAGE: http://law.indiana.edu/about/people/details/marinotti-joão.html

Center for Intellectual Property Research, Indiana University Maurer School of Law ( email )

211 South Indiana Avenue
Bloomington, IN 47405
United States

Information Society Project, Yale Law School ( email )

P.O. Box 208215
New Haven, CT 06520-8215
United States

HOME PAGE: http://law.yale.edu/joao-marinotti

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
27
Abstract Views
349
PlumX Metrics