Updates to the JVR Safety Profile scales

By Andrew Morris

There is no definitive scientific evidence to suggest that a 4-point is better than a 5-point report scale or vice versa. As such many times the choice is based on personal preferences or consideration of the (dis)advantages of a particular choice in context e.g. selection vs development, and/or the audience e.g. trained individual’s vs managers.

JVR Psychometrics made the informed decision to move to a 5-point scale to display psychometric results in the Safety Profile reports for the following reasoning:

1 - Increased sensitivity to individual differences: In this article we unpack the value of individual differences as a framework for predicting safety performance (https://lnkd.in/dN79rV8u). As such we have come to realise that 5-point scales have offered clients more differentiation between candidates which has proved useful in both selection and development contexts when distinguishing between candidates can offer significant gains. 4-point scales are useful for the uninitiated as they are intuitive and quick to interpret but there is a loss of information that is not optimal in high-stakes decision making. It is our view that providing a more nuanced view of individual’s results is also a fairer way of treating candidates who can then be better compared to one another against job requirements rather than diluting differences when compressing results.

2 - Reduced potential for biased decisions: The inclusion of average categories inherent in 5-point scales serves as valuable reference points for interpretation, offering report readers a clearer understanding of individuals' positions relative to the average. In selection contexts, especially for panels lacking extensive experience, this aids in mitigating the occurrence of false negatives and false positives. The additional granularity provided by a 5-point scale can help decision-makers avoid oversimplification and make more accurate distinctions between candidates' competencies.

3 - Optimising broader talent analytics initiatives: In talent analytics, the use of competency scores presented on a 4-point scale can lead to skewed distributions when viewing results. Such scales often result in distributions that are either positively skewed (pushed to the left) or negatively skewed (pushed to the right), potentially distorting the perception of overall talent potential as either poor or excessively good. However, by incorporating a mid-level/average scoring category, such as a 3 on a 5-point scale, there is a greater likelihood of approximating a normal distribution where there is a clear average range. This allows for a more accurate understanding of how the sample is positioned, enabling organisations to make more informed decisions and effectively address talent management challenges. For example, looking at the overall distribution of a group on a 4-point scale which appears positively skewed (pushed to the left) may give the impression of a poor performing group or highlight that the assessments were too difficult which would possibly be a false conclusion based on a forced distribution.

4 - Statistical advantages: A 5-point scale offers distinct advantages from a statistical standpoint compared to a 4-point scale. With more data points available, the 5-point scale enhances variability in the distribution of responses, thereby increasing the precision and reliability of further statistical analyses e.g. validation studies. Additionally, the finer gradations provided by the 5-point scale enable the application of more advanced analytical techniques that may be required in broader talent analytics and techniques that assume a normal distribution. This facilitates more robust uses of the data beyond the immediate selection need.

5 - Protects construct validity: The conversion to a 5-point competency scale preserves the construct validity of the various assessments used by maintaining the underlying relationships between psychometric variables and the overall competencies which are usually a combination of scales from different assessments. By anchoring psychometric scores to more varied competency categories, the scales are able to retain their ability to accurately represent the constructs of interest. This ensures that the integrated reports provide meaningful and valid information about candidates' competencies, thereby supporting informed decision-making.

6 - Improved sensitivity to change: A 5-point scale offers greater sensitivity to detect changes over time or in response to interventions. The additional response options provide a more nuanced measurement of shifts in behaviours or competencies, which is particularly valuable in longitudinal studies or performance interventions. By capturing subtle changes that may not be discernible with a 4-point scale, organisations can track progress more effectively and tailor interventions accordingly.

7 - Enhanced interpretability and actionability: A 5-point competency scale offers greater interpretability and actionability in some contexts compared to a 4-point scale. The additional point allows for a more nuanced representation of individuals' competencies, facilitating clearer communication of assessment results to stakeholders. It is our experience that when stakeholders cannot easily see the distinction between candidates that productive discussions come to a halt. This increased clarity also enables more targeted interventions and development plans tailored to individuals' specific competency levels, ultimately leading to more effective talent management practices. For example, in development feedback where people are rated on a 4-point scale and forced into a particular category (see forced distribution above), so a development category other than ‘average’ or ‘meets requirements’, this lack of nuance and loss of detail may mean reduced buy-in from the feedback recipient and/or lead to a focus on areas that are not a top priority.

There is not a standard rule-of-thumb and an optimal number of categories. In surveys and other organisational listening processes, this may matter more and where answers are clearer cut but for displaying integrated psychometric results the choice comes down to preferences on many of the points noted above. It is our view that increasing the number of scale points can enhance the validity and reliability of the competency results. By expanding from a 4-point to a 5-point scale, the results provide greater precision in measuring overall competency. This increased precision contributes to less chance of biased decisions and improved validity, ensuring that the assessment accurately captures individuals' potential performance. Additionally, the reliability of the assessment process is bolstered as the additional scale point provides more variability, reducing measurement error, increasing predictive potential and overall being a fairer way of treating candidates.