Evaluating a novel method of scoring multirater assessments of leader competencies

By R. Michael Furr
Wake Forest University

Stephen Jeong
Center for Creative Leadership

John W. Fleenor
Center for Creative Leadership

Sirish Shrestha
Center for Creative Leadership

Summary

In leadership development, 360-degree assessments serve as multirater instruments in which leaders evaluate their own competencies and receive assessments from various raters, such as peers, direct reports, and superiors. These assessment methods provide significant opportunities for both leaders and practitioners, but they also present complex challenges, particularly related to scoring. This article examines a new method for scoring 360-degree feedback assessments. The researchers compared a novel scoring approach based on advanced statistical (i.e., bi-factor) models to the traditional scoring method (using mean score comparisons).

This study proposed a Latent Approximation Method (LAM) for scoring these assessments, a method grounded in recent advances in multirater data modeling. The researchers applied this LAM scoring method to a 360-degree measure of various leadership competencies, using an international sample of approximately 17,000 leaders and nearly 150,000 raters across more than 120 countries. They examined the associations among LAM scores and between LAM scores and standard averaged scores, and evaluated the incremental predictive utility of LAM scores beyond standard scores. The results, replicated across subsamples, revealed a considerable overlap between LAM and standard scores. While LAM scores offered a significant but minor incremental predictive power over Standard Averaging Method scores, the overall value of LAM scoring for practitioners appeared limited.

While the new method showed some potential benefits, the researchers found that it didn’t offer significant advantages over the simpler traditional approach in most cases. The study concludes that the more complex scoring method may not be necessary for interpreting 360-degree assessments, but suggests areas for future research to potentially improve leadership development practices.

Citation

Furr, R. M., Jeong, S. B., Fleenor, J. W., & Shrestha, S. (2024). Evaluating a novel method of scoring multirater assessments of leader competencies. Consulting Psychology Journal. Advance online publication.

LINK

Leave a Comment