Uses of MQM

Can MQM be used for evaluating language service providers or evaluators?

Yes, but its focus is on measuring the quality of translation products. It cannot address aspects that go beyond the scope of errors in content, such as professionalism or timeliness of delivery. It also cannot evaluate processes directly, but the results can suggest ways to improve them. For example, error and root cause analyses can help TSPs identify areas that require additional attention, such as source quality, localization readiness, or challenges specific to a given domain, locale or service provider. For evaluators, results can be used for training purposes to fine-tune criteria to improve inter-rater reliability.

Can implementers use MQM to provide feedback on the source context?

Yes. There are two ways to do this. One is to apply MQM to the source content directly. All top level error dimensions except Accuracy – which requires comparison of source and target content – can be used monolingually to evaluate source content. Another is to point out problems in the source discovered through root cause analysis during the evaluation of the target text. Root cause analysis is a feature introduced in MQM 2.0. It was developed from the “fairness principle” used in MQM 1.0.

Can you use MQM for spoken language services?

MQM assumes the existence of a written representation of content. If you convert spoken language to written form, you can use MQM to evaluate that the resulting text. Some implementers use MQM to evaluate the quality of speech-to-text services such as auto-transcription.

Can you use MQM for multimedia translation?

You can use MQM for evaluating multimedia content, but, as with spoken language, doing so requires the language portion of the multimedia product to be reproduced as written text. The MQM working group is in the process of adding features specific to multimedia applications.

Can MQM be used to evaluate the quality of transcreation?

It can, although applying MQM becomes less straightforward the higher the degree of content adaptation involved, e.g. when transcreation becomes copywriting in the target language.

What is the difference between MQM and BLEU?

MQM is a manual, reference-free approach that identifies specific translation errors (unlike BLEU and other automatic evaluation metrics that require reference translations). Unlike BLEU, MQM does not require a reference translation, but reference translations are rarely available for real-world projects.

Who uses MQM?

Because the broad applicability that goes far beyond quality evaluation of translation products, MQM is widely implemented in the language industry, in institutional translation, in translation tools, and in research projects.

 

See further at Why Use MQM? and MQM History.