The rise and fall of a killer chart

No single graph has captured the political imagination quite like Leon Feinstein’s killer chart.  In one go, it showed that talent was no substitute for parental advantage even at early ages – and has since been endlessly used to justify early interventions, most recently in Nick Clegg’s Social Mobility Strategy, with Feinstein himself elevated to ‘Chief Policy Analyst at 10 Downing Street’

But this graph has now been challenged by researchers who claim it is simply a statistical artefact. In this post I describe the killer chart, the critique, and the political fallout that results…

The rise…

As befits a chart of such fame, Feinstein’s original 2002 chart can be explained quite simply, and the chart itself is shown below. [I’ve taken this from the Government’s social mobility strategy via the BBC; a free short Feinstein article is available via LSE, and the original article is in Economica in 2003]

What it shows is that the test scores of high-intelligence low-class kids (at two years old) are overtaken by low-intelligence high-class kids by the age of seven. Or in the words of the excellent pop-economist Tim Harford, “The very brightest 22-month-old working-class kids were inexorably overhauled by the very dimmest children of professional or managerial parents– apparently by the age of about seven, and emphatically by the age of 10.” 

He describes this as ‘something to get angry about’, and this is the reason for the influence of the chart – it’s very difficult to see this as anything other than a gross injustice, a damning indictment of our socially immobile society. And it’s been heavily used to win right-wingers over to various educational interventions.

…and fall?

The trouble is that the graph might be wrong, mistaking bad statistics for genuine facts. John Jerrim and Anna Vignoles at the Institute of Education – one of the top institutions in the UK – have recently published a paper that argues that these results are a classic case of ‘regression to the mean’, a noted statistical fallacy that crops up most often when evaluating speed cameras, and (it is claimed) also explains why great books make average movies. (Jerrim & Vignoles’ newspaper explanation of the article is available here).

The problem is this. Feinstein assigns people to the ‘high-ability’ and ‘low-ability’ groups based on a single test. Like all tests, there will be a certain amount of random measurement error involved – in how the kids are feeling that day, in whether they guess right or wrong when they don’t know, in whether the questions ask specific things they get more easily etc. This means the kids that do best on the test are (on average) not just smarter but luckier too. The next time they take the test, they’re likely to be only averagely lucky, and this means that their rank position will go down. So we’d expect the high-ability and low-ability groups to look more similar on the next test.

Jerrim & Vignoles go on to point out that the regression to the mean effect will have different effects for different classes – because we’d expect fewer low-class kids in the high-ability group (due to pre-existing inequalities), random measurement error will be greater for them than for high-class kids. When they apply this to some real-world data (shown below), they find a much less-pronounced narrowing of the gap between the low-ability high-class kids and the high-ability low-class ones.

Political fallout

Much as the critique seems plausible (and regression to the mean is one of the classic statistical fallacies), I don’t want to judge this particular statistical battle – I’m trying to persuade one of the experts in this field to write for the blog in the coming weeks, so I’ll leave the final arbitration to her.

For the time being, what interests me is the story of the rise and fall of a ‘stylised fact’ – the way that Jerrim & Vignoles’ critique has made the papers and BBC Radio 4’s fabulous statistic show ‘More or Less’, and more importantly, has been used within Government by Conservatives opposed to the social mobility strategy (at least according to these news reports).  One of the reports describes the findings as, “In other words, the entire basis for the government’s social mobility strategy is wrong,and this is how they’re being spun.

But such an interpretation seems to me completely ludicrous. The evidence for the influence of class (in a broad sense) on educational outcomes is overwhelming, including the body of excellent research by Feinstein himself, as Jerrim & Vignoles say in their well-judged response to over-excitable coverage:

“far from undermining the coalition’s social mobility strategy, our study actually confirms the previous research evidence on which the strategy is based – which has shown that children from disadvantaged backgrounds have very poor cognitive skills compared to their richer contemporaries, and that this socioeconomic gap emerges early in childhood.”

Despite this, the story that emerges is about the rise and fall of a killer chart – a chart whose rhetorical significance and the crossing of the lines came to symbolise much more than the wider (and accepted) body of research underlying it, and whose (possible) demise may lead the this wider research to be marginalised.

About Ben Baumberg

I am currently a Lecturer in Sociology and Social Policy at the School of Social Policy, Sociology and Social Research (SSPSSR) at the University of Kent. I also helped set up the collaborative research blog Inequalities, where I regularly write articles and short blog posts. I have a wide range of (too many...) research interests, at the moment focusing on disability, the workplace, inequality, deservingness and the future of the benefits system, and the relationship between evidence and policy. You can find out more about me at http://www.benbaumberg.com
This entry was posted in Articles and tagged , , . Bookmark the permalink.

2 Responses to The rise and fall of a killer chart

  1. mel bartley says:

    I have not seen Vignoles and Jerrim’s responses to the media coverage. But lets not be naive here. How much attention would their working paper have received if it had not appeared to be a refutation of Feinstein’s politically significant work? Academics need to be careful about this kind of thing. Same for the critics of Wilkinson and Pickett. Who has hever heard of their more vocal critics (obviously I exclude john Goldthorpe, but Tony Atkinson also had crushing criticism years ago but refrained from raising his profile with it, not that he needed a higher profile which might be significant in itself). We are all under pressure to have ‘impact’ and writing something that could be interpeted as a critique of a politically significant (and highly impact-ful) piece of work is one of the easiest ways to do it. A colleague of mine said the other day “thats the way to make a career, write 100 papers saying that x is related to y and then another 100 saying x is not related to y”. Of course x and y cannot be just any old thing, it would have to be something like “intelligence is the fundamental cause of health inequality”.

    • Ben Baumberg says:

      Thanks Mel – yes, it does seem to be that the best way of getting noticed is to say something controversial that turns out to be wrong, but everyone has to cite you when explaining why it’s wrong…

      The trouble is, that lots of the facts that emerge into political/public debate are not fully accepted in their field – the Wilkinson & Pickett hypothesis is a good example of this. Feinstein’s basic conclusions are accepted, but the crossover in the graph may well be wrong. I can’t help thinking it would be better to present these results ORIGINALLY as uncertain, so that the (inevitable) later caveats are seen as a continuation of the line of research, rather than a refutation.

      This leads into the entire debate about uncertainty in social science however. I think that the public and politicians are perfectly able to cope with uncertainty if presented clearly (and not as a get-out clause for academics). But others disagree.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s