MQM-Core and Scorecard Design
Download the sample Excel Scorecard Calculator File
Figure 1 shows a scorecard modeled on the complete MQM-Core Error Typology. This selection mirrors the complete MQM-Core set of translation error types that is revealed if all sub-dimension links are clicked for the high-level dimensions.
It may be unlikely that any evaluation group would want to include the entire MQM-Core set of dimensions because doing so would entail a very high level of effort and potential cost. MQM descriptive documents suggest keeping the number of error types in the scorecard to around twelve. However, the full scorecard is extremely useful for selecting a customized subset of error types. Figure 2 shows a sample customized score card created by a working group that had identified a short set of error types they wanted for evaluating student work. In this case, for instance, layout and localization issues were not necessarily relevant.
Without reference to any standardized error typologies, the group had decided on the following values, originally listed in alphabetical order, but here ordered logically:
- Inaccurate or inconsistent term; wrong word
- Addition, omission, mistranslation,
- Grammar, capitalization, spelling, punctuation, wrong tense
- Rhythm, register
- Excellent solution
In order to configure the list in an MQM scorecard, the first step was to map the error types in the list to those in MQM Core.
- Terminology: The “terminology” issues fit obviously under Terminology, retaining wrong term, although it is not entirely clear if the usage is identical with the MQM definition.
- Accuracy: Addition, omission, mistranslation clearly belong here
- Linguistic Conventions: Grammar, capitalization, spelling, are in the list, but punctuation and wrong tense don’t show up in MQM-Core. A quick consultation of MQM-Full, however, shows them at lower levels under Linguistic Conventions, as shown by their three digit ID numbers,
- Style: Rhythm is confusing – the topics treated do not necessarily involve poetry or song, where rhythm might be an issue. The potentially arbitrary decision was made to map this item to Style, bringing Register
- Excellent solution is tucked in as a custom solution requiring special treatment in that it takes positive values, or in a context where positive values represent errors, it is expressed as a negative value subtracted from the Absolute Penalty Total. Another option for this value is to subtract the value of H23 from the sum for the Absolute Penalty Total.
The resulting analysis yields the scorecard in Figure 2. The actual scorecard is produced by making a copy of MQM-Full, simply and efficiently deleting rows for un-wanted error types, and adding rows for the items that are not in the MQM-Full scorecard. The process is efficient and speedy.
Elections Example Scorecard
The Detailed Description section of the MQM-TQE Procedure document discusses a test implementation used to produce the BYU MQM Scorecard App, which works with an expanded scorecard that displays the totals for intermediate products of the Error Counts (EC) multiplied by the Severity Multipliers. Note that this view also calculates totals for the Error Counts in each severity level, as well as the penalty totals for each severity level. The Penalty Total for Critical Errors reveals the Critical Error Count, which is important because any value other than “0” signals an automatic “Fail” evaluation, although the error can of course be corrected to achieve a Pass rating.
Figure 4 shows a similar scorecard, configured as an Excel spreadsheet (without displaying the intermediate products).The same procedure is followed as for the previous example: starting from the Full-MQM scorecard, the desired error types are preserved, and the unneeded or unwanted error types are deleted to arrive at a short, concise scorecard.
