Do test cheats matter if you test enough people?

Over the past decade, the cheapness and convenience of online testing has seen its usage grow tremendously. Its critics raise the openings it makes for cheaters, who might take a test many times under different identities, conspire with past users to identify answers, or even employ a proxy candidate with superior levels of the desired trait. Its defenders point to countertactics, from data forensics to follow-up tests taken in person. But the statistical models employed by researchers Richard Landers and Paul Sackett suggest that in recruitment situations, the loss of validity due to online cheating can be recovered simply due to the greater numbers of applicants able to take the test.

Landers and Sackett point out that test administrators normally intend to select a certain volume of candidates through testing, such as the ten final interviewees. The accessibility factor of online testing could allow you to grow your candidate pool, say from 20 to 50. Considering these numbers, its possible to now select those that scored better than 80% of the other candidates, rather than merely those in the top half. And if some of your candidates cheat, oomphing their scores to the 82nd percentile when they only deserve the 62nd, that's still a better calibre than the 50-or-better you would have been prepared to accept from your smaller face-to-face pool.

Landers and Sackett moved from these first principles to modelling out some realistic large data sets containing a range of true ability scores. They considered sets where cheating gave a small (.5 SD improvement) or large (1 SD) bonus to your test score; against this was another factor, how much your natural ability influenced your likely to cheat, from no relationship, r=0, into increasingly strong negative relationships, from -.25 to -.75, modelling the idea that weaker performers are more likely to cheat. And finally, they varied the prevalence of cheating in increments from zero up to 100%.

The researchers ran simulations in each data set by picking a random subset - the 'candidate pool' - and selecting the half of the pool with better test scores. In the totally honest datasets, the mean genuine ability score of selected candidates was .24. but that value was lower for sets that contained cheaters, as some individuals passed without deserving it. Landers and Sackett then added more candidates into each pool, allowing pickier selection, and reran the process to see what true abilities were obtained. In many data sets the loss of validity due to cheating was easily compensated by growth of applicant pool. For instance, if cheating has only a modest effect and is only mildly related to test ability (r= -.25) then doubling the applicant pool yields you genuine scores of .24 even when 70% of candidates are cheating, and higher scores when the cheaters are fewer in number, such as .31 for 30% cheaters.


Great...but wait. there are two important take-aways relating to fairness. It's true that if we're getting .31 averages instead of .24, our selected candidates should be more job-capable, even some of those who did cheat, and that's a win for whoever's hiring. But in the process we've rejected people who by rights deserved to go through. Essentially, this is a form of test error, and so not a uniquely terrible problem, but it's one we shouldn't become complacent about just because the numbers are in the organisation's favour.

Secondly, and as anyone trained in psychometric use will be aware, increasing selection ratios from top 50% to top 25% is no casual prerequisite. Best practice is that without evidence, such as an inhouse validity study, cut-offs on a single test should be capped at the 40th percentile, meaning you pass 60% of candidates. In particular, raising thresholds can have adverse impact on minority groups, on whom many tests still show differentials (although these are closing over time). As minorities tend to make up a minority of any given applicant pool, such differentials can easily squeeze the diversity out of the process before you even get a chance to sit down with candidates and see what they have to offer in a rounded fashion.

Nevertheless, this paper brings a fresh angle to the issue of test security.


Landers, R., & Sackett, P. (2012). Offsetting Performance Losses Due to Cheating in Unproctored Internet-based Testing by Increasing the Applicant Pool International Journal of Selection and Assessment, 20 (2), 220-228 DOI: 10.1111/j.1468-2389.2012.00594.x

Further reading:

Tippins, N. T. (2009). Internet alternatives to traditional proctored testing: Where are we now? Industrial and Organizational Psychology, 2, 2–10.

Finding the balance between work and home

(We're reporting from this month's Division of Occupational Psychology conference at the Digest. This post is by Dr Jon Sutton, Managing Editor of The Psychologist, and will also feature in that magazine's March issue. @jonmsutton / @psychmag)

Who is responsible for work-life balance? The individual, the organisation, or even the legislative system? That was the question posed at the start of this symposium, and it became clear that ‘one size fits all’ policies and practices don’t exist: we must understand needs and wants in order to tailor solutions.

First up was Dr Ellen Ernst Kossek (Purdue University, US), who identified the importance of feeling in psychological control of boundaries. Based on three validated measures of ‘cross role interruption behaviours’, ‘boundary control’ and ‘work-family identity centralities’, Kossek outlined different profiles. You’re either an ‘integrator’, or a ‘separator’, or you cycle between the two: a ‘volleyer’. Add in consideration of whether your well-being level is high or low and you end up with six styles, including the ‘fusion lovers’ who are happy to integrate work and family life, the ‘job warriors’ who volley away to their heart’s discontent, and ‘captives’ who are the separators with low well-being.

The image of Winston Churchill in his pyjamas, as an early remote worker, cast a large shadow over the talk by Dr Christine Grant and colleagues from Coventry and Warwick Universities. Grant described her work to outline competencies related to the effective e-worker, and to develop an assessment tool. Organisations can provide training for existing and new e-workers, Dr Grant said, before leaving us with the thought that ‘a good manager is always a good manager; a bad manager is worse as an e-worker’.

It’s one thing taking your work home with you when you’re an academic or editor, but another entirely when you’ve just been pulling a family out of some motorway wreckage. Dr Almuth McDowall (University of Surrey) looked at work-life balance self-management strategies in the police force, eliciting 134 behaviours from semi-structured interviews. Some were context-specific, for example in the police it’s actually very important not to take work home with your, as it is confidential and often intrusive material. McDowall highlighted the importance of communication and negotiation over work-life balance, and suggested that there is a separate competence for line managing work-life balance in others.

Finally, Professor Gail Kinman (University of Bedfordshire) tackled a subject close to home for many: work-life conflict in UK academics. She noted that academics vary in the extent to which they wish their roles to be integrated, with many highly absorbed in the job role and most working considerably over the 48 hour working time directive. In Kinman’s survey of 760 academics in at least 99 universities, most academics weren’t getting the separation they wanted. Working at home and ICT use predicted work-life conflict. Kinman called for enhanced sensitivity to variation in boundary management styles and preferences amongst colleagues and supervisors, citing the example of sending e-mails at the weekend as potentially role modelling that behaviour for the recipient.

Another interesting point to emerge from the symposium is that most measures of work-life balance are focused on the impact on families, despite the fact that it’s an issue for the single and childless as well.

Further reading:

Ernst Kossek, E., Lewis, S., & Hammer, L. (2009). Work--life initiatives and organizational change: Overcoming mixed messages to move from the margin to the mainstream Human Relations, 63 (1), 3-19 DOI: 10.1177/0018726709352385

Are organisations led by the limbic system?

(We're reporting from this month's Division of Occupational Psychology conference at the Digest. This post is by Dr Jon Sutton, Managing Editor of The Psychologist, and will also feature in that magazine's March issue. @jonmsutton / @psychmag)

According to keynote speaker Gerard Hodgkinson (Professor of Strategic Management and Behavioural Science at Warwick Business School), ‘Descartes’s error is alive and well in the workplace’. In a bold and wide-ranging address, Hodgkinson made the case for why and how occupational psychology needs to connect with the social neurosciences.


Hodgkinson is bringing psychology into the field of strategic management, trying to help decision makers become more rational. Take how organisations tend to respond to a major threat or opportunity (HMV and Blockbuster come to mind as I write this). Usually there are small, incremental changes, and when it becomes apparent this isn’t sufficient, what does the organisation do? Nothing. There is a period of ‘strategic drift’. Then there is a period of ‘flux’, which on Hodgkinson’s graphic representation looks rather like a tailspin. This is followed by ‘phase 4’, ‘transformational change’ or ‘complete demise’.

But to what extent can psychology shed light on this process? Hodgkinson’s 2002 book ‘The Competent Organization’ argued the case for the centrality of the psychological contribution to organisational learning and strategic adaptation, yet 11 years on, he said, there was still only a passing consideration of affective and non-conscious cognitive processes. Why do we continue to sidestep it?

Using examples from his practice, Hodgkinson demonstrated how strategising is both an inherently cognitive and affective process. Eliciting a cognitive taxonomy from senior figures in a UK grocery firm, he found that although the market conditions had changed dramatically, mental models – individually and collectively – had not. Decision makers were slaves to their basic psychological processes, for example still focusing on the ‘magic number’ of ‘7 plus or minus 2’ competitors.

Hodgkinson showed how he confronts strategic inertia in top management teams, stimulating individual cognitive processes by scenario analysis. Some organisations excel at this: Hodgkinson claims that Shell closed all their facilities within 45 minutes of 9/11. While others were still struggling to comprehend what was happening, their scenario planning had allowed them to take quick and decisive action.

Hodgkinson’s latest research draws on social cognitive neuroscience and neuroeconomics to develop a series of counterintuitive insights. His hope is that these can teach people to be more skilled in their control of their emotional, limbic system. True rationality, he concluded, is the product of the analytical and experiential mind.

Further reading:

Hodgkinson, G., & Healey, M. (2008). Cognition in Organizations Annual Review of Psychology, 59 (1), 387-417 DOI: 10.1146/annurev.psych.59.103006.093612 Pdf freely available here.

One personality to rule them all?

(We're reporting from this month's Division of Occupational Psychology conference at the Digest. This post is from regular editor Alex Fradera, and the report will also feature in March issue of The Psychologist magazine.)

Until recently I was pretty ignorant of the idea of a general factor of personality, a situation which undoubtedly hurt my psychology-nerd cred. I'm back on track now, thanks to an afternoon spent in Rob McIver's symposium on the matter.

The general factor of personality, or GFP, is analoguous to g, the intelligence quotient that predicts to differing degrees the multiple intelligences - verbal, musical, numerical -that sit below it. (The symposium reminded us that whereas Spearman posited g in the 1900s, and Thurstone the differential intelligence model in the twenties, it took until the 1950s for Phillip Vernon to reconcile the models).

While practitioners who use personality emphasise its differential qualities - many facets, no one right profile - the academics who advocate GFP say that on the contrary, there is such a thing as having lots of personality, and that this global factor is meaningful, predicting a range of life outcomes. Critics say this may be down to statistical artefacts, such as an individual's desire for social desirability influencing all their questionnaire responses. So this symposium took us into the science, and particularly what it means for practitioners.

The first session, given by Rainer Kurz of Saville Consulting, was the most technical in focus, introducing a way to get a GFP simply by summing raw scores on each Big Five personality measure. It's an intuitive approach that in his dataset of 308 mixed roles proved as valid in predicting job performance as the standard approach (extracting the 'first unrotated principle component') while avoiding some fiddly statistical issues. However, the GFP was not comprehensive, as after partialling out its variance he found significant influences of personality subcomponents remained, notably assertiveness and achievement. This suggests the add-up method doesn't quite account for their influence. He concluded that this was a promising recipe but the approach will take refining.

His colleague Rob McIver chose to put aside notions of 'the ideal GFP' to explore total personality scores that predict success on a particular capability-set - in most cases, a job. Rather than relying just on factor extraction or the add-it-all-up approach, this starts by developing and shaping tests to pre-fit the criterion you care about.

McIver's data drew on external raters who had judged various facets of workplace effectiveness for the same individuals described by Kurz in his earlier presentation. The individuals had also completed seven different personality tests, and McIver explained how he generated a total personality score for each one using a criterion approach: personality dimensions were mapped onto effectiveness based on logic and previously reported relationships, meaning some dimensions were weighted heavily and others not at all if judged irrelevant to effectiveness. McIver showed how their own questionnaire, developed from the ground up around these effectiveness factors, produced the most powerfully predictive total scores, with an r up to .32.

McIver went further, producing a personality super-score for each participant by totalling all seven tests together. Would it work, given that many of these questionnaires were not developed with this effectiveness framework in mind? It turns out that united they stand, pretty well, with a validity of .27, thanks in some part to the criterion-based pruning and weighing. McIver concluded that this approach may be more profitable than searching for one true GFP.

Between these two talks Rob Bailey of OPP took the floor to question whether, in any case, true GFPs could truly be useful for practitioners. He points out that the literature tends to describe the general factor as reflecting people who are relaxed, sociable, emotionally intelligent, satisfied with life, and altruistic - and that a low score means the opposite of these things. He challenged the symposium to imagine cases where such information could be provided to an individual in any constructive fashion, compared to the conventional profiling approach.

Bailey then went to the data, in this case taken from over 1,200 individuals paid to complete a 16-factor personality questionnaire, the absence of career implications giving them little incentive to 'fake nice' and apply spin to their results. His component analysis suggested the personality data could reduce to two factors, not fewer, and he showed how opting to use the dual factors rather than the 16 original ones weakened the ability to predict variables such as job satisfaction, dropping coverage from 9.3% of the variance to 7.5%.

He concluded that granularity, not fat factors, may be a better bet for predictive power, and also cautioned that the differences he finds (no single factor, more value in the parts than a whole) may result from using a personality measure that isn't built to the specifications of the Big Five, and that in fact dependence on that model may be under-valuing the diversity, and thus relevance, of personality itself.

When the dust settled, the questions remained, but the issue of the GFP will undoubtedly be one we will revisit.

Further reading:

van der Linden, D., te Nijenhuis, J., & Bakker, A. (2010). The General Factor of Personality: A meta-analysis of Big Five intercorrelations and a criterion-related validity study Journal of Research in Personality, 44 (3), 315-327 DOI: 10.1016/j.jrp.2010.03.003

The dark side of behaviour at work

(We're reporting from this month's Division of Occupational Psychology conference at the Digest. This post is by Dr Jon Sutton, Managing Editor of The Psychologist, and will also feature in that magazine's March issue. @jonmsutton / @psychmag)

The face that launched a thousand peer-reviewed journal articles beamed down from the stage as self-confessed ‘well adjusted workaholic’ Professor Adrian Furnham (University College London) began his keynote. Quips were in ready supply, but Furnham is much more than a crowd pleaser: this was a talk steeped in history and theory.

According to Furnham, there are 70,000 books in the British Library with leadership in the title. But most leaders don’t succeed, they fail, with a base rate of bad leadership collated from various studies of 50 per cent. This is due to incompetence (not having enough of something, or being promoted beyond the job they are good at), or derailment (having too much of a characteristic, such as self-confidence, or creative quirkiness). It’s this later problem that Furnham focused on, identifying three root causes: troubled relationships, a defective or unstable sense of self; and ineffective responses to change.

Furnham highlighted three fundamental issues. Firstly, organisations ‘select in’, for the traits they think will help an employee be a success, rather than ‘selecting out’ for what is going to cause problems. Secondly, it’s assumed that competencies are linearly related to success. And thirdly, employers fail to see the dark side of bright side traits and the bright side of dark side traits. For example, what if a self-confident leader pursues a risky course of action built on overly optimistic assumptions?

How do we characterise what makes a leader destructive? Furnham feels that the early ‘trait’ approach to leadership failed because ‘the list of traits grew remorselessly, leading to confusion, dispute and little insight’. Trait theory also ignored the role of both subordinates and situational factors. This oversight was rectified in the work of Tim Judge – who Furnham called ‘the best living occupational psychologist’ (see Digest coverage here)– which showed the ‘toxic triangle’ of destructive leaders, susceptible followers and conducive environments. The influence of the model was clear in Furnham’s own consideration of the ‘Icarus syndrome’. High flyers fall through poor selection, flawed personality, no or poor role models, and because they are rewarded for toxicity in the organisation.

Furnham then cantered through some typical personality disorder problems in plain English: arrogance, melodrama, volatility, eccentricity, perfectionism etc. I was struck by the simple, neo-psychoanalytic conception of Karen Horney from 1950: people move away from others, towards them or against them (something covered recently). Furnham outlined some just published research on the differences between private and public sector dark side traits, with private sector more likely to move against others through manipulation or creating dramas whereas public sector managers were more likely to show moving away traits such as withdrawal, doubt, or cynicism.

A series of his own studies, generally with huge samples, elucidated sex differences in dark side traits and their relationships with career choice and success. From all this, Furnham distilled some key implications for selection and recruitment. Consider using ‘dark side’ measures; beware excessive self-confidence and charm; do a proper bio-data and reference check; and get an expert to ‘select out’ for you. As for management, the message was to beware fast-tracking wunderkinds, and to seek a mentor, coach or at least a very stable deputy to keep these individuals on the rails.

‘Just as a good leader can do wonders for any group, organisation or country,’ Furnham concluded, ‘a bad one can lead to doom and destruction. Understanding and developing great leaders is one of the most important things we can do in any organisation.’

Furnham, A., Hyde, G., & Trickey, G. (2013). Do your Dark Side Traits Fit? Dysfunctional Personalities in Different Work Sectors Applied Psychology DOI: 10.1111/apps.12002


Further reading:
Timothy A. Judge, Ronald F. Piccolo, Tomek Kosalka, The bright and dark sides of leader traits: A review and theoretical extension of the leader trait paradigm, The Leadership Quarterly, Volume 20, Issue 6, December 2009, Pages 855-875, ISSN 1048-9843, 10.1016/j.leaqua.2009.09.004.
Pdf freely available here

DOP Annual Conference Reports: A matter of evidence

I spent last week at the BPS' Division of Occupational Psychology annual conference, from which I'll be reporting for much of the remaining month. A conference highlight was a late night chat with Prof Rob Briner of the University of Bath. Rob is a keen advocate of evidence-based practice, and as the Digest exists to help the world of work operate better through judicious use of evidence, I was keen for his views on what we're doing right and what we can do better.

As well as a bunch of tips and suggestions for me to explore, there were a few wider themes that I think are worth communicating.

Single articles are useful up to a point...
They remind us of the process that sits behind the evidence, the actual running-a-studyness of science: sample sizes, participant demographics, choice of statistics, study limitations. On top of this, they can be encapsulated relatively simply, meaning they get shared or form the beginning of discussions that get people to reflect on the evidence in a particular area.  And they do provide some evidence but only from one study.

....but Reviews are where it's at if you really want to use evidence
Narrative reviews, systematic reviews and meta-analyses all have their own features, but share in common that they look across the research to find patterns and trends. And the aggregate level is the most important level of evidence. In other words, the findings of any single study may be interesting and provide insight but what is much more important is what the whole body of evidence is suggesting.  Reviews can present clear findings about a body of evidence or be complex and contradictory and messy but we need to know what the evidence as a whole is suggesting before we can use it in practice.

Due to this, the Occupational Digest will now cover a review every month, either something just published or a fairly recent review that speaks to an issue of current interest.

In addition, when covering single articles, I'll aim to provide a second reference beyond the article itself, to a review that addresses the topic the article concerns itself with. Hopefully this will provide you, the readership, with signposting to help you find out more.

Further reading:
Briner, R.B. (1998).  What is an evidence based approach to practice and why do we need one in occupational psychology?  Proceedings of the British Psychological Society Occupational Psychology Conference, 39-44. (pdf link)

Psychopathic traits won't give an edge to entrepreneurs

Most people in business would agree that the dog-eat-dog mentality most celebrated in the 1980s has passed its high water mark, giving ground to cultures that value collaboration and mutual benefit. Yet shows such as The Apprentice still depict ruthlessness and uncompromising nature as the hallmark of the business success, and fictional characters from Gordon Gecko to Patrick Bateman further the idea that value-creators are a little psychopathic. Investigation of the 'dark-side' traits of leadership has started to focus on this notion: a recent report describes ongoing research suggesting these traits are present in start-up entrepreneurs, and may even be helpful to their success.  This matters if it were so, and a London-based team of Reece Akhtar, Gorkan Ahmetoglu, and Tomas Chamorro-Premuzic have just published research into this issue, canvassing 435 individuals through an online survey.

The research looked at entrepreneurialism both as a personal trait, using the Measure of Entrepreneurial Tendencies and Abilities (META), and in terms of involvement in entrepreneurial activities. In this study, this was defined widely to include achievements and activities that relate to the researchers core definition of entrepreneurialism: innovation, value creation, and taking opportunities. So success included not only selling ideas and inventing products, but organising events and helping create social institutions.

Participants also responded to Levenson's self-report scale on psychopathic tendencies. The team predicted that different facets of psychopathy might mean different things for entrepreneurialism. Primary psychopathy involves narcissism, manipulation of others and low empathy, and was measured by agreement with items such as one beginning 'success is based on the survival of the fittest'. This was predicted to facilitate entrepreneurialism through competition and exploiting opportunities. Secondary psychopathy, meanwhile, involves lifestyle behaviours such as impulsivity or parasitic dependence, and anti-social behaviour such as criminal activity and recidivism. Given how these traits look likely to create more problems than they solve and alienate others in the process, they were predicted to actually impede entrepreneurial activities.

Akhtar and his colleagues built a statistical model of the data (including demographics and other basic controls) to determine what factors mattered when others were taken into account. META had a strong relationship of .72 with the overall entrepreneurial activity of the individual. Secondary psychopathy turned out to have no significant relationships to entrepreneurial traits (META) or activities. Meanwhile, primary psychopathy - that Darwinian 'me-first' mentality - was moderately linked to META ratings. But once META was taken into account, primary psychopathy had little bearing on whether an individual had entrepreneurial achievements or activities. The only exception that was found was in one sub-domain - building and benefiting society - and this relationship was negative, which is in this case makes intuitive sense.

This study suggests that for a broad definition of entrepreneurialism, the perception that psychopathic traits are needed is a false one. It may be that they can occur alongside other more pertinent traits, but their quintessentially psychopathic elements don't make for a better entrepreneur: they can even undermine their effectiveness. It may yet be true that entrepreneurs value getting ahead over getting along, but far from fetishising these dark-side traits, we should treat them with appropriate caution. Apprentice-makers, take note.

Akhtar, R., Ahmetoglu, G., & Chamorro-Premuzic, T. (2013). Greed is good? Assessing the relationship between entrepreneurship and subclinical psychopathy Personality and Individual Differences, 54 (3), 420-425 DOI: 10.1016/j.paid.2012.10.013