D3 A5 (15min pres)
Tracks
Track A | Ball Room 1 (recorded for In-person & digital)
| Saturday, October 26, 2024 |
| 12:15 PM - 12:30 PM |
| Stream A | Ballroom 1 |
Overview
A tale of two scales: applying fuzzy graphic ratings to referee scores.
(Julie West)
Presenter
Dr Julie West
Principal
Workplace Research Associates
A Tale of Two Scales: Applying Fuzzy Graphic Ratings to Referee Scores
12:15 PM - 12:30 PMAuthor(s)
Pearson, Jessica
West, Julie
West, Julie
Abstract
Referee reports are frequently used in recruitment, including in the context of medical selection (e.g. into postgraduate medical specialist programs). Referees can be asked to verify details of past employment, comment on behavioural concerns, and give insights into job-relevant capabilities. Drawing on direct experience of candidates in the workplace, their perspective can be valuable and can prevent poor selection decisions.
Although referees can provide unique insights, they tend to be lenient in their assessments. Referee scores may also demonstrate low inter-rater reliability and predictive validity. The challenge for organisations lies in using referee reports to tap into the valuable insights they can provide without inviting an overly positive and undifferentiated assessment that does not help in discriminating effectively between candidates or in accurately predicting their future performance.
Improved methods of data collection can increase the quality of referee ratings, such as the use of standardised approaches that require referees to score candidates on the same well-defined scale. There is potential to identify other methods that may increase the reliability and validity of information provided. This is especially important in the context of high-stakes selection.
This research explores the relative merits of two different types of standardised rating scales used over successive years to obtain referee scores for candidates applying to a postgraduate medical specialist program in Australia. In 2023, using Referee Scale #1, referees put each applicant’s performance in one of four percentile bands (e.g. in the lower 75%, in the top 5% etc.). Candidates were explicitly rated relative to their peers. On this kind of scale, most candidates were rated as more competent than their peers but, by definition, most candidates cannot fall above the average in reality.
Referee Scale #2, introduced in 2024, used a Fuzzy Graphic Rating Technique (FGRT). Referees made three ratings to describe each candidates’ typical, best and poorest performance on each rated attribute. In this approach, ratings are more rigorous, and referees are able to represent the variability of real world performance.
It is hypothesised that ratings from Referee Scale 1 would be negatively skewed, representing a lenient portrayal of candidate performance, while ‘Typical’ ratings on Referee Scale 2 would follow a more normal distribution. Findings, methodological limitations, conclusions and recommendations for the use of referee ratings in selection are discussed.
Although referees can provide unique insights, they tend to be lenient in their assessments. Referee scores may also demonstrate low inter-rater reliability and predictive validity. The challenge for organisations lies in using referee reports to tap into the valuable insights they can provide without inviting an overly positive and undifferentiated assessment that does not help in discriminating effectively between candidates or in accurately predicting their future performance.
Improved methods of data collection can increase the quality of referee ratings, such as the use of standardised approaches that require referees to score candidates on the same well-defined scale. There is potential to identify other methods that may increase the reliability and validity of information provided. This is especially important in the context of high-stakes selection.
This research explores the relative merits of two different types of standardised rating scales used over successive years to obtain referee scores for candidates applying to a postgraduate medical specialist program in Australia. In 2023, using Referee Scale #1, referees put each applicant’s performance in one of four percentile bands (e.g. in the lower 75%, in the top 5% etc.). Candidates were explicitly rated relative to their peers. On this kind of scale, most candidates were rated as more competent than their peers but, by definition, most candidates cannot fall above the average in reality.
Referee Scale #2, introduced in 2024, used a Fuzzy Graphic Rating Technique (FGRT). Referees made three ratings to describe each candidates’ typical, best and poorest performance on each rated attribute. In this approach, ratings are more rigorous, and referees are able to represent the variability of real world performance.
It is hypothesised that ratings from Referee Scale 1 would be negatively skewed, representing a lenient portrayal of candidate performance, while ‘Typical’ ratings on Referee Scale 2 would follow a more normal distribution. Findings, methodological limitations, conclusions and recommendations for the use of referee ratings in selection are discussed.
Learning outcomes
Insight into how to improve the validity and reliability of referee ratings in selection.
An improved understanding of the utility of the Fuzzy Graphic Rating Technique (FGRT) to gather referee ratings.
An improved understanding of the utility of the Fuzzy Graphic Rating Technique (FGRT) to gather referee ratings.
.....
Dr Julie West is a senior organisational psychologist with over 30 years' experience working as a specialist consultant and, prior to this, in the Australian Public Service. She is the Principal and founder of Workplace Research Associates, a management consultancy established in 1996, which specialises in the delivery of organisational psychology services.
Julie has considerable applied experience in organisational structural and functional reviews, job analysis and design, role evaluation, senior executive and graduate recruitment, assessment centres, training development and delivery, psychological testing and assessment, program evaluation, survey design and analysis, and the design of appraisal and feedback systems.
She has lectured and tutored in psychology in both Australia and the United States, and has held academic and research positions. Julie maintains a keen interest in promoting research-led best practice in all aspects of her applied work.
Ms Jessica Pearson
Workplace Research Associates
A Tale of Two Scales: Applying Fuzzy Graphic Ratings to Referee Scores
12:15 PM - 12:30 PM.....
Jessica is a Registered Psychologist and Senior Consultant with Workplace Research Associates in Canberra. She has worked in both public and private settings as a Psychologist, providing evidence-based therapeutic services to adolescents and adults, and has contracted to multiple tertiary institutions in the provision of psychoeducational lectures and workshops.
Jessica also spent several years working and contracting for a regionally-based Human Resources (HR) organisation. She initially worked as an HR Project Officer, conducting research for, and coordinating, various recruitment projects, and delivering legislation-informed policies, procedures, and frameworks for small-to-medium enterprises. Following her psychology registration, Jessica transitioned into a Consultant Psychologist role, delivering psychoeducational webinars to rural and regional clients to support mental health improvements in the workplace.
Jessica joined Workplace Research in 2022 and has worked on numerous consulting projects, assisting and enhancing organisations to optimise their performance.
