Can we get away with using lo-fi assessment to recruit advanced positions?

In recruitment, the promise of comparable results for less effort is understandably tempting. It's offered by the offsetting of costly assessments with alternative measures that use pencils, screens and standardised questions instead of expert assessors. However, as some sources suggest a bad hire can cost twice or more that position's annual salary, the stakes are high. A new study kicks some assessment tyres to see whether that bargain is actually a banger.

Researchers Filip Lievens and Fiona Patterson looked at recruitment into advanced roles which typically seek the skills and knowledge to hit the ground running. They took their sample of 196 successful candidates from the UK selection process for General Practitioners in medicine (GPs). To get here, you've completed two years of basic training and up to six years of prior education, by which stage you're after someone ready to go, not a future 'bright star'. Lievens and Patterson were specifically interested in how much assessment fidelity matters, meaning the extent to which assessment task and context mirror that in the actual job.

Three types of assessment were involved, all designed by experienced doctors with assistance from assessment psychologists. Written tests assessed declarative knowledge through diagnostic dilemmas such as “a 75-year-old man, who is a heavy smoker, with a blood pressure of 170/105, complains of floaters in the left eye”. Assessment centre (AC) simulations meanwhile probe skills and behaviours in an open-ended, live situation such as emulating a patient consultation; these tend to be more powerful predictors of job performance, but are costly.

The third was the situational judgement test (SJT), a pencil and paper assessment where candidates select actions in response to situations, such as a senior colleague making a non-ideal prescription. SJTs are considered by many to be “low-fidelity simulations”, losing their open-endedness and embodied qualities, but hanging on to the what-would-you-do-if? focus. The authors were interested in whether its predictive power would be in the same class as the AC simulations, or mirror the more modest validity of its pencil and paper counterpart.

The data showed that all assessments were useful predictors of job performance, as measured by supervisors after a year spent in role. Both types of simulation - AC and SJT - provided additional insight over and above that given by the rather disembodied knowledge test – each explaining about a further 6% of the variance. But in comparison with each other, the simulations were difficult to tell apart, with no significant difference in how well they predicted performance.

It should be noted that the AC simulations did capture some variance over and above the SJT, notably relating to non-cognitive aspects of job performance, such as empathy, which is important as such areas are less trainable than clinical expertise. However, this extra insight was fairly modest, just a few percentage points of variance. More expensive AC assessments can provide additional value, but the study suggests that at least in this specific recruitment domain, you can get away with a loss of fidelity if the assessments are appropriately designed.

Lievens, F., & Patterson, F. (2011). The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection. Journal of Applied Psychology, 96 (5), 927-940 DOI: 10.1037/a0023496

Positive pollyannas more frustrated by unmet expectations

A tendency to view the world positively yields many benefits, such as higher wellbeing and a sense of personal effectiveness that gets you ahead in life. However, according to a recent article in the Journal of Occupational and Organizational Psychology, this sunny view of life brings with it certain workplace expectations – and when they aren't fulfilled, it can spell trouble. Behold the “disaffected pollyannas”.

Researchers Olivia O'Neill, Laura Stanley, and Charles O'Reilly were interested in the career expectations and choices made by individuals with high “trait positive affect” (PA). They recruited 132 participants from a full-time MBA course, who sat an assessment centre where they completed a Positive and Negative Affect Scale and reported what they expected their highest lifetime salary to be. The data showed that PA results in confidence about salary, with every one-unit increase in PA associated with a $100,000 increase in expected earnings. High hopes, but what if they aren't met?

Four years later, 105 of the participants responded to a follow-up survey probing salary levels and frequency of moving organisation. High PA individuals turned out to be far more responsive to low salary; at a standard deviation below average, they shifted through an average of four jobs, compared to two for their low-PA counterparts. This higher turnover was expected, as frustration of their higher salary expectations is more likely to lead to doubting their fit at the organisation. Moreover, they are more willing to believe the grass is greener elsewhere, and more able to make that step successfully, due to better social networks and an interview advantage due to infectious positive affect at interview (similar to the rapport advantage described here).

Measures of job and life satisfaction at a final follow-up eight years on showed that for low PA individuals, more frequent job shifts led to more satisfaction. The notion is that shopping around means that over time you get a sense of what is realistic and make your peace with what a reasonable job constitutes. But for high PA, the reverse was true, with more job shifts making them more frustrated, bemoaning the absence of the perfect job they are destined for.

High PA individuals can be a positive feature in organisations, but this research shows that they can be open to disappointment, affecting their prospects in the organisation and their feelings on life. The authors conclude that “the key to finding long-term satisfaction, then, may be managing expectations, rather than pursuing unrealistic ideals.”


O'Neill, O., Stanley, L., & O'Reilly, C. (2011). Disaffected Pollyannas: The influence of positive affect on salary expectations, turnover, and long-term satisfaction Journal of Occupational and Organizational Psychology, 84 (3), 599-617 DOI: 10.1348/096317910X500801

How much should we trust job applicant ratings of their own emotional intelligence?

Self-rating is a popular way to measure emotional intelligence in the workplace. Under lab conditions it's been shown that these ratings vary depending on what your (imaginary) objective is: to give a 'true' picture or to successfully win a job. A new study translates this lab finding to the workplace, finding that applicants for jobs really do rate themselves higher on EI than counterparts already working in that organisation.

The study compared scores for 109 job applicants with 239 volunteers, matched by department and managerial level. They rated themselves on four classic components of EI: self emotion appraisal, others emotion appraisal, use of emotion, and regulation of emotion. Applicants significantly outscored incumbents in all areas, on average rating themselves more than a standard deviation better. The areas of greatest divergence were in use of emotions and regulation of emotions, which have much in common with the Big Five personality traits conscientiousness and emotional stability, which we know job applicants have a higher tendency to inflate.

On all but one of the components, applicant scores were significantly more bunched together than incumbent scores, which could be seen as additional support that they were manufactured, with candidates homing in on scores that were solidly good, avoiding suspicious high or unhelpful low scores.

The study is important because in other areas of research, score discrepancies can be found in the lab, due to different explicit instructions, that don't seem to surface in the real world, suggesting the overt nature of lab conditions can exaggerate or even manufacture differences. Yet here the effect is found again, suggesting that if we do want to rely on self-report to assess EI we should recognise that this inflation may take place, and that relying on the normative data that accompanies these tests may lead us to unrealistically high appraisals of candidates.


Lievens, F., Klehe, U., & Libbrecht, N. (2011). Applicant Versus Employee Scores on Self-Report Emotional Intelligence Measures Journal of Personnel Psychology, 10 (2), 89-95 DOI: 10.1027/1866-5888/a000036

Are job selection methods actually measuring 'ability to identify criteria'?

While we know that modern selection procedures such as ability tests and structured interviews are successful in predicting job performance, it's much less clear how they pull off those predictions. The occupational psychology process – and thus our belief system of how things work - is essentially a) identify what the job needs b) distil this to measurable dimensions c) assess performance on your dimensions. But a recent review article by Martin Kleinman and colleagues suggests that in some cases, we may largely be assessing something else: the “ability to identify criteria”.

The review unpacks a field of research that recognises that people aren't passive when being assessed. Candidates try to squirrel out what they are being asked to do, or even who they are being asked to be, and funnel their energies towards that. When the situation is ambiguous, a so-called “weak” situation, those better at squirrelling – those with high “ability to identify criteria” (ATIC) - will put on the right performance, and those that are worse will put on Peer Gynt for the panto crowd.

Some people are better at guessing what an assessment is measuring than others, so in itself ATIC is a real phenomenon. And the research shows that higher ATIC scores are associated with higher overall assessment performance, and better scores specifically on the dimensions they correctly guess. ATIC clearly has a 'figuring-out' element, so we might suspect its effects are an artefact of it being strongly associated with cognitive ability, itself associated with better performance in many types of assessment. But if anything the evidence works the other way. ATIC has an effect over and above cognitive ability, and it seems possible that cognitive ability buffs assessment scores mainly due to its contribution to the ATIC effect.

In a recent study, ATIC, assessment performance, and candidate job performance were examined within a single selection scenario. Remarkably it found that job performance correlated better with ATIC than it did with the assessment scores themselves. In fact, the relationship between assessment scores and job performance became insignificant after controlling for ATIC. This offers the provocative possibility that the main reason assessments are useful is as a window into ATIC, which the authors consider “the cognitive component of social competence in selection situations”. After all, many modern jobs, particularly managerial ones, depend upon figuring out what a social situation demands of you.

So what to make of this, especially if you are an assessment practitioner? We must be realistic about what we are really assessing, which in no small part is 'figuring out the rules of the game'. If you're unhappy about that, there's a simple way to wipe out the ATIC effect: making the assessed dimensions transparent, turning the weak situation into a strong, unambiguous one. Losing the contamination of ATIC leads to more accurate measures of the individual dimensions you decided were important. But overall your prediction of job performance measures will be weaker, because you've lost the ATIC factor which does genuinely seem to matter. And while no-one is suggesting that it is all that matters in the job, it may be the aspect of work that assessments are best positioned to pick up.

Kleinmann, M., Ingold, P., Lievens, F., Jansen, A., Melchers, K., & Konig, C. (2011). A different look at why selection procedures work: The role of candidates' ability to identify criteria Organizational Psychology Review, 1 (2), 128-146 DOI: 10.1177/2041386610387000

Using work technology at home hinders our ability to detach from work

We know that psychologically detaching from work is important, leading to less fatigue, more positive working-week experiences, and higher overall life satisfaction. How you fill your leisure time has a big impact on psychological detachment - for instance, we've reported on the beneficial effects of volunteering on detachment. A recent study confirms what many suspect – it's harder to switch off when technology keeps you plugged in.

In their study, YoungAh Park, Charlotte Fritz and Steve Jex looked at work-home segmentation: how much we partition our domains of leisure and work. Some of this is preference – for example, you might choose not to take a job likely to intrude into your home life. And some is about surrounding norms: if it's typical to take work home, or to call a colleague on a work issue in an evening, it's difficult not to be drawn into these activities.

But the authors suspected that a major factor was technology use at home, and investigated this through a survey completed by 431 university alumni now in full-time employment. As well as measures of detachment from work, segmentation preference eg “I prefer to keep work life at work”; and perceived segmentation norm, they also looked at frequency of use of different technologies (email, internet, phone, pda) for workplace purposes when at home.

As expected, both a preference for and a culture of less segmentation led to less psychological detachment. People who used technologies for work purposes while at home struggled to detach from work, and the analysis showed that this was a major route through which weak segmentation had its effect on detachment. In part, weak segmentation manifests as work-technology behaviours at home.

It's important to note that technology did not explain all of the variance, which means that setting strict rules about technology use is not the only way to help psychological detachment, nor necessarily sufficient; you may want to develop habits that deal with ruminations, develop end-of-day rituals, or establish clearer boundaries with colleagues. But technology certainly plays a part, and so it's worth considering the practices of your own workplace: for instance,are the trends towards shedding work desktops for laptops, and “bring your own computer” programs, helping or hurting us?


Park, Y., Fritz, C., & Jex, S. (2011). Relationships between work-home segmentation and psychological detachment from work: The role of communication technology use at home. Journal of Occupational Health Psychology DOI: 10.1037/a0023594

Some of us experience bigger 'emotional hangovers', whether from fun activities or hurricanes

While some of us may be generally happier than others, all of us experience different emotions from day to day. A fascinating new study suggests that these fluctuations are due to two factors: a cycling of emotion levels across the working week, and our unique personal sensitivity to both good and bad daily events. The study even has hurricanes.

Daniel J. Beal and Louma Ghandour from Rice University set out to track the daily affect patterns of participants from an IT services company. They were particularly interested in how intrinsic task motivation – how fulfilling the participants found their work that day – influenced emotion or affect. Ten days in, Hurricane Ike struck the region. Recommencing some weeks later, the study also took the chance to examine how this negative one-off event influenced matters.

The 65 participants completed 21 end-of-day surveys (prompted by an email reminder), rating intrinsic task motivation, together with how much they felt emotional states like frustrated, discouraged, happy and proud. As per other recent research, the negative emotions showed a cyclical pattern, peaking at Wednesday with a projected bottoming out on Saturday; positive emotions showed the inverse pattern. There were also individual differences in average scores: some people are generally more frustrated than others.

The authors also calculated each participant’s ‘affect spin’, a measure of day-to-day emotional volatility, a high score meaning that person experienced a wide range of different affect states from day to day. The authors found that having a motivating day's work affected that day’s positive mood for everyone, but individuals with high affect spin saw a kind of positive hangover into the next day as well.

After Hurricane Ike, everyone experienced lower levels of positive affect. This began to recover as the event receded into the past, but not for those with high affect spin, who seemed to be suffering a longer hangover again, but this time with negative consequences.

Individual differences in emotional state matter, and this study reminds us that we don't just differ on average, but also in how dynamically our mood responds to events. It's possible that offering a fascinating problem to your reactive employee on a Monday will generate benefits that carry forward, and battle the mid-week dip. The authors conclude that “mapping the terrain of positive and negative affective events and their implications for worker well-being can help to ground the field of organizational psychology in a truly experiential understanding of work life.”

Beal, D., & Ghandour, L. (2011). Stability, change, and the stability of change in daily workplace affect Journal of Organizational Behavior, 32 (4), 526-546 DOI: 10.1002/job.713

Buying into the idea of 'free choice' makes us less likely to see discrimination

Illustration: Emily Wilkinson, www.mindfulmaps.com

To all our women readers: it's great to be living in a post-discrimination world, right? Right? Is this thing on?

Whatever your view – and regardless of facts such as woman's earnings standing at under 80% of men's - many people seem to feel that way, such as the 53% of Americans in a recent Gallup poll. Nicole Stephens and Cynthia Levine of Northwestern University identify one reason for this: the 'choice framework', a view of the world particularly popular in the US that treats all actions as freely chosen based on our preferences. Seeing life purely in terms of choices can empower individuals, and studies show we can benefit psychologically. But in a new paper to be published in Psychological Science, these researchers explore how it can make us reluctant to see discrimination as a cause of mothers leaving the workforce.

In a first study, 171 stay-at-home mothers revealed through questionnaire ratings that they saw their departure from the workforces as a choice rather than something imposed on them. The more they endorsed the choice explanation, the less likely they were to interpret genuine gender inequality in a range of industries as due to discrimination or structural challenges to women working (such as a default model of work that doesn't adequately account for childcare).

The second study adopted experimental methods to manipulate exposure to the choice framework. While waiting to begin the experiment, 46 undergraduates were unwittingly exposed to a poster on the wall about “women's experiences in the workforce”; in one condition, the title began with the phrase “Choosing to leave”. Participants then completed a questionnaire, and those who had had this slight level of exposure to the choice framework were somewhat more likely to rate gender discrimination as non-existent.

Stephens and Levine note that culture propagates such messages at higher frequencies than those manipulated in their study, thus the baseline influence might be substantial. The consequences are twofold: although people feel happier when they see themselves as an active agent in their own life, this can turn against them when they meet genuine structural challenges, where it “could undermine their sense of competence or deter them from seeking help”. And on a societal level, this tendency may prevent the correction of genuine inequity. We may need cultural and political actors to reframe the debate. And an individual level, it might be enlightening to reflect once in a while on the limits to choice.

Stephens, Nicole, & Levine, Cynthia (2011). Opting Out or Denying Discrimination? How the Framework of Free Choice in American Society Influences Perceptions of Gender Equality Psychological Science