Think about the last time you had a performance review. How accurately do you think you were assessed by your peers? And how accurately do you believe you assessed them? More importantly, why are performance reviews universally associated with feelings of tension, disdain, and the disappointment of being unfairly judged?

Let’s take 360 performance reviews, for example—the very idea behind one is that a greater number of relevant people reviewing the same person provides for a more accurate assessment of an individual. 

But this idea may be inherently flawed. 

Some of the best books ever written—think A Wrinkle in Time, Life of Pi, The Wonderful Wizard of Oz, and of course, the Harry Potter series—all had to go through repeated rejections before they saw the light of day. The second most popular book in America, Margaret Mitchell’s Gone With The Wind, was rejected by almost 40 publishers before going on to sell almost 30 million copies!

What does this say about human beings as reviewers? 

In a nutshell—we suck at assessments. Our rating of something is subject to several factors outside the scope of the subject of assessment, making us unreliable indicators of how good (or bad) it is.

Now what does this mean for performance reviews—a system of appraisals that is dependent entirely on peer rating? It’s an uncomfortable topic for organizations, but one that is very real.

The first step to acknowledging any problem is naming it—so let’s talk about the idiosyncratic rater effect.

The Idiosyncratic Rater Effect and its dire consequences

The idiosyncratic rater effect (IRE) is a psychological phenomenon that occurs when multiple evaluators perceive the same piece of work or individual differently. Each evaluator approaches their analysis from different angles and has different biases and expectations, thus forming differential views on the quality of performance or outcomes. 

Remember the saying, “What you see in others is a reflection of yourself”? That’s precisely what the idiosyncratic rater effect is trying to tell us—our rating of someone else says more about us than it does about them. And this isn’t some neomodern ideology based merely on assumptions. Research has proven its existence.

This phenomenon was extensively studied and confirmed in 1998 and 2010 in the scientific journal ‘Personnel Psychology’ and in the ‘Journal of Applied Psychology’ in 2000.

Needless to say, this effect points us to question the accuracy of ratings given by multiple individuals that may be influenced by biases, misunderstandings, and lack of shared criteria in evaluating gathered evidence. One can only imagine the dire consequences this brings about—unfair performance reviews, disturbed interpersonal relationships, reduced career prospects and growth, flawed recruitment processes, faulty promotions—the list goes on and on.

The Idiosyncratic Rater Effect is rooted in our biases

If there’s one thing that’s innate to the nature of all human beings, it’s bias. It runs in your blood, and it isn’t something you can always shake off in time for performance reviews. Biases are the product of months, or sometimes, years of conditioning. And this antagonist in our performance reviews presents itself in a variety of ways:

Each of these biases plays a role in sabotaging your performance reviews as well as your reviews of others. There’s no need to play the blame game—we’re all guilty of it. But it isn’t our fault. The problem is with the system—a flawed system of performance reviews that feeds off a weakness that is innate to our nature.

So what does this mean for HR? Are performance reviews doomed? What will replace 360 performance reviews and the 9-box performance grid? 

Let’s not get ahead of ourselves yet.

The problem isn’t the concept of peer reviews or 360 surveys. It’s bad data.

Granted, the 360 review system has its own faults. But the malefactor that has the potential to make it such an ardent purveyor of inaccurate reviews is bad data—which, in our case, means inaccurate data that is rooted in bias. 

In contrast to the idea behind the 360 survey, sourcing data from multiple sources may not always solve the problem. In fact, biases have the power to accelerate it. Your peers are as susceptible to bias as your manager, and at the end of the day, what you receive in the name of a performance review could very well be a jumbled mess of contradictory opinions that tell you absolutely nothing.

And bad data is no joke. It has dangerously far-reaching implications. 

The idiosyncratic rater effect is evidence that an astonishing bulk of the data that we use to assess employees may be erroneous. 

Erroneous assessments are followed by erroneous promotions, which are in turn, followed by resentful sentiments, stunted growth, accelerated attrition, and more. Scary!

But how can we fix this innate issue with peer assessments? How do we go about sourcing better data to fuel crucial business decisions? 

The way out of biased performance reviews

There are no clear guidelines on how we can completely eradicate the idiosyncratic rater effect from our performance reviews—mostly because an entire system redesign of talent management & performance review practices would be required to truly eliminate it. 

This may sound scary, but organizations can still take certain steps to minimize the idiosyncratic rater effect and make peer performance reviews more reliable. For example:

• Managers, peers, and other supervisors can, to an extent, be trained on how to assess progress over time using objective criteria such as specific goals, targets, and measures. This can help make their evaluations more objective and minimize the effects of bias in their evaluations.
• Using calibrated ratings where raters are trained to use a consistent rating scale and provided with examples of what different ratings mean will help serve as a system of reference for raters.
• Additionally, encouraging open communication between management and reportees would result in more transparent discussions about expectations and performances during the review process. 
• Using an automated performance enablement system can help in streamlining the process of data collection, analysis, and reporting of performance to limit the influence of bias.
• Establishing well-defined protocols for recruiting raters, and creating fair evaluation methodologies with built-in controls such as blinding procedures or standardization scoring rubrics will further help eliminate this effect.

Ultimately, organizations must maintain regular communication regarding standards set by the company in order to eliminate any bias when evaluating employees' performances.

Better data all the way

Data is key in building your business strategy and unreliable data can derail any organization far from its goals. So, a better understanding of data guidelines is paramount in reducing the risk of the idiosyncratic rater effect. By setting clear expectations for stakeholder involvement in data collection, organizations can ensure that ratings are as reliable and unbiased as possible.

Most importantly, having consistent and open communication with team members to help them understand why certain choices were made is the key to forging trust among people. An environment built upon trust is an environment where people from all backgrounds are motivated to bring forth their best.