ISO 5060, ASKM WK46396, and these webpages all stress the need for translation quality evaluators to be appropriately qualified and well trained in order to assure consistent and reliable translation quality evaluation outcomes.
ISO 5060 outlines a comprehensive list of competences that are recommended for translation quality evaluators. In brief, evaluators first shall possess the skills requirements stated in ISO 17100 for translators, revisers, bilingual editors, and reviewers. In addition to customary translation-oriented linguistic competences, the list includes organizational, cultural, technical, and domain-specific competences. These factors are likely to be fundamental “job-description” requirements for potential evaluators. Ideally, individuals chosen to become translation evaluators should be “known” persons who have been working in an environment in roles described here, and are chosen because of their familiar skills.
The most critical of the competence descriptions in ISO 5060 states that evaluators who are qualified to function under the specifications of these standards shall be capable of carefully comparing paired source and target language segments and judging translation quality based on the criteria defined in the MQM Error Typology. The ability to objectively classify and annotate any errors found during the evaluation process by assigning them to their relevant error type and severity level comprises the competence most likely to require focused training. Even well-qualified, experienced evaluators shall require introduction to any unfamiliar annotation metric and scoring system.
In any given environment, and even for these kinds of well-trained, experienced individuals, it shall be essential to provide examples of previous texts, sample error annotations, severity assignments, and proposed corrections. When onboarding new evaluators, training should provide adequate exposure to existing resources, approaches, and procedures in order to ensure consistent inter-rater reliability across language pairs, text types, and subject fields.
Attention to specifications involving desired translation grade, subject-field and terminology expertise, as well as evaluation approaches (light analysis of MT texts, for instance, as opposed to more demanding evaluation of premium quality texts) all play a role in assigning evaluators to specific evaluation projects. Avoiding “over-correction” or even the introduction of undesirable requests for change can be a major challenge for evaluators who think that they must register many comments or error tallies in order to justify their roles in the production process.
Testing new people by giving them previously evaluated translations to annotate and then comparing their results with previous ratings provides a good tool for training and for judging performance. Periodic training and mutual discussion among evaluators with different experience levels can improve evaluator reproducibility and reliability, particularly when considering detailed error sub-types. Conscious agreement on the interpretation of various error types and weights and severity levels supports more reliable performance across a team of evaluators.
As noted in WK46396, the maintenance of long-term statistics reflecting evaluator patterns can provide valuable trend data designed to support evaluator consistency, identify undesirable variations, and support continued improvement. Individual evaluators should demonstrate inter-rater reliability scores above 75%. This performance level can be illustrated either by comparing scores to a known “reference rater” who has demonstrated high quality skills in the past or via comparison of scores across a team. Evaluators who vary from this norm should engage in further training or not be used in this capacity in future. The desirability of engaging highly competent evaluators is further affected by the fact that many highly effective translators prefer to be translating and refuse to take on assignments involving revision or evaluation.
Resource materials
ISO 17100:2015, Translation services — Requirements for translation services
ISO DIS 5060:2022, Translation services — Evaluation of translation output — General guidance
WK46396: Standard Practice for Analytic Translation Quality Evaluation
https://themqm.org/