Criterion Contamination: When Performance Reviews Measure Things Employees Can't Control

Two account managers. Maya gets the Kathmandu enterprise territory with three renewing clients. Dev inherits a region where the largest customer just went bankrupt. Maya finishes at 130% of quota and is rated 'exceptional'. Dev finishes at 70% after saving two accounts nobody thought were savable, and is rated 'needs improvement'. The review system worked exactly as designed. It just measured the territory, not the people.

The technical idea

Industrial-organisational psychologists distinguish between the 'ultimate criterion' — the true, complete concept of job performance — and the 'actual criterion' — the measure you can collect. The gap produces two errors. Criterion deficiency is when your measure misses important parts of the job (for example, rating engineers only on tickets closed and ignoring code reviews). Criterion contamination is when your measure includes things that are not job performance — luck, resources, the rater's mood, or factors outside the employee's control. Austin and Villanova's history of 'the criterion problem' (1992) shows this has troubled the field for a century.

The five biggest contaminants

The controllability test

The simplest practical tool is to ask, for every metric in a review: 'To what extent could this person have changed this result through their own choices?' Rate each metric high, medium, or low on controllability. Low-controllability metrics can still be tracked — they matter to the business — but they should carry less weight in individual evaluation, or be adjusted for context.

How to decontaminate reviews

1. Separate 'what happened' (outcomes) from 'what the person did' (behaviours and decisions).
2. Normalise outcomes for context: territory potential, team size, inherited problems.
3. Use multiple raters and calibration so one manager's idiosyncrasies don't dominate.
4. Train raters on specific behavioural anchors rather than general impressions.
5. Evaluate decision quality: given what they knew at the time, was the call sound?
6. Audit rating outcomes for patterns by team, location, work arrangement, and demographic group.

Limits and caveats

• Some roles are legitimately paid for outcomes, luck included — for example, commission sales. The question is whether that is a conscious choice.
• Over-adjusting can create endless excuses. Context should inform ratings, not replace accountability.
• Exact rater-effect sizes vary by study and instrument; the consistent finding is that they are large.

Contamination and deficiency: the two ways a measure fails

In industrial-organisational psychology, the 'criterion' is the measure you use to judge performance. A criterion can fail in two ways. It is contaminated when it includes things that are not really performance — luck, territory quality, the rater's mood. It is deficient when it leaves out things that are part of performance, such as helping colleagues or preventing problems. Most review systems suffer from both at once.

How big is the problem?

A widely cited study by Scullen, Mount and Goff (2000) analysed multi-rater data for thousands of managers. They found that the largest share of variance in ratings — around 62% — was explained by the rater's own tendencies, while actual performance explained far less. In other words, a rating often tells you as much about the person giving it as the person receiving it. This 'idiosyncratic rater effect' is one of the clearest forms of contamination.

Common contaminants in real reviews

A cleaner review design

Questions to ask in calibration

• What would this person's result have been with an average territory, team, or tool set?
• Is this rating based on evidence from the whole period or the last few weeks?
• Would another manager with the same evidence give the same rating?
• What did this person do that does not appear in the numbers?

• Review forms separate behaviours from results.
• Constraints are documented during the year.
• Rater distributions are compared across managers.
• Pay decisions are not based on a single rating number alone.

Worked case: the rating that measured visibility

Two engineers deliver similar work. One works in the office, presents at every demo and replies to messages within minutes. The other works remotely, ships quietly and writes the documentation everyone relies on. At calibration, the first is rated 'exceeds', the second 'meets'. When the manager is asked for evidence, most of what he lists for the first engineer is visible activity: demos, meetings, quick responses.

That is criterion contamination: the rating has absorbed things that are not the job's real outcomes. Visibility is not worthless — communication matters — but if it is not in the written expectations, it should not quietly decide the rating.

A contamination check for every review

• Every rating comment points to an outcome or behaviour listed in the role's expectations.
• Evidence comes from the whole period, not only the last six weeks.
• The reviewer has asked: would I rate this the same if this person worked in another location?
• Results outside the person's control (a lost client, a frozen budget) are noted and set aside.
• Peer input is from people who saw the work, not only people who sit nearby.

Contamination vs deficiency

Most review systems have both problems at once. Fixing contamination without fixing deficiency just leaves a narrower, still unfair measure.

What each role can do

• HR: rewrite the rating form so each rating requires a linked piece of evidence.
• Calibration leaders: ask 'what is the evidence?' every time a rating is defended with a general impression.
• Managers: keep a short monthly note per person so the review is not built from memory.
• Employees: keep your own record of outcomes and share it before the review.