Translation Quality Evaluation (TQE) provides guidance and procedures for the inspection of a translation to measure and determine whether it meets specifications agreed on by translation stakeholders. A translation is often called a target text. The source text is a text in a starting (source) language, and the target text is a corresponding text in the target language.
Part of preparing a translation for an evaluation using MQM is to divide the source text and target text into segments, usually sentences or similar size chunks, and align the segments. Together, a source text segment and target text segment that contain corresponding content are called a segment pair or a translation unit (TU). MQM TQE focuses on TUs during the evaluation process.
The Concrete Example sections of this document use a short French text translated into English that discusses several aspects of European Union law, referred here as the Elections text. The Elections text is based on an evaluation that was assigned to an actual human evaluator, but was slightly modified to provide some additional errors for illustrative purposes.
MQM TQE is based on the assumption that evaluators are well trained and familiar with evaluation and tools, and that they have translation and editing skills comparable to those of a bilingual editor. It is also assumed that the system has been shown to be reliable through training and trial, meaning that different evaluators independently produce very similar evaluations.
MQM TQE provides an analytic method for measuring the quality of a translation. An analytic evaluation associates errors with specific words and phrases in the text, whereas a holistic evaluation considers the text as a whole. MQM TQE can be thought of as consisting of three stages. In practice, the second and third stages could be combined, but each element of the TQE method needs to go somewhere.
Preliminary Stage: the following tasks do not need to be implemented in a specific order
- reviewing and ensuring access to the agreed-on translation specifications
- verifying (or selecting/creating, if necessary) the metric for performing the evaluation based on the translation specifications
- assigning the Threshold Value for pass/fail acceptance of the evaluation
- preparing the source text and target text for evaluation
- determining the Evaluation Word Count, usually by means of a software app such as a CAT (computer assisted translation) tool
In an established TQE system, the first three elements set forth the model that can be used in later projects as a template to be used over and over again. In such instances, the three elements regarding translation specifications, metrics, and Threshold Values are more like checkpoints that the implementer verifies before continuing the TQE.
Error Annotation Stage: during this stage, the evaluator examines the translated text against the source text and specifications, and annotates (meaning identifying, marking, and assigning error type and penalty points) errors in accordance with the metric. The 3 annotated errors generate the Absolute Penalty Total.
Automatic Calculation & Follow-Up Stage: during this stage, the Overall Quality Score is calculated according to the selected scoring model using the Evaluation Word Count from the Preliminary Stage and the Absolute Penalty Total from the Error Annotation Stage, then compared to the Threshold Value to assign a pass/fail rating. Other actionable items are determined as a result of the evaluation.