No single graph has captured the political imagination quite like Leon Feinstein’s killer chart. In one go, it showed that talent was no substitute for parental advantage even at early ages – and has since been endlessly used to justify early interventions, most recently in Nick Clegg’s Social Mobility Strategy, with Feinstein himself elevated to ‘Chief Policy Analyst at 10 Downing Street’
But this graph has now been challenged by researchers who claim it is simply a statistical artefact. In this post I describe the killer chart, the critique, and the political fallout that results…
As befits a chart of such fame, Feinstein’s original 2002 chart can be explained quite simply, and the chart itself is shown below. [I’ve taken this from the Government’s social mobility strategy via the BBC; a free short Feinstein article is available via LSE, and the original article is in Economica in 2003]
What it shows is that the test scores of high-intelligence low-class kids (at two years old) are overtaken by low-intelligence high-class kids by the age of seven. Or in the words of the excellent pop-economist Tim Harford, “The very brightest 22-month-old working-class kids were inexorably overhauled by the very dimmest children of professional or managerial parents– apparently by the age of about seven, and emphatically by the age of 10.”
He describes this as ‘something to get angry about’, and this is the reason for the influence of the chart – it’s very difficult to see this as anything other than a gross injustice, a damning indictment of our socially immobile society. And it’s been heavily used to win right-wingers over to various educational interventions.
The trouble is that the graph might be wrong, mistaking bad statistics for genuine facts. John Jerrim and Anna Vignoles at the Institute of Education – one of the top institutions in the UK – have recently published a paper that argues that these results are a classic case of ‘regression to the mean’, a noted statistical fallacy that crops up most often when evaluating speed cameras, and (it is claimed) also explains why great books make average movies. (Jerrim & Vignoles’ newspaper explanation of the article is available here).
The problem is this. Feinstein assigns people to the ‘high-ability’ and ‘low-ability’ groups based on a single test. Like all tests, there will be a certain amount of random measurement error involved – in how the kids are feeling that day, in whether they guess right or wrong when they don’t know, in whether the questions ask specific things they get more easily etc. This means the kids that do best on the test are (on average) not just smarter but luckier too. The next time they take the test, they’re likely to be only averagely lucky, and this means that their rank position will go down. So we’d expect the high-ability and low-ability groups to look more similar on the next test.
Jerrim & Vignoles go on to point out that the regression to the mean effect will have different effects for different classes – because we’d expect fewer low-class kids in the high-ability group (due to pre-existing inequalities), random measurement error will be greater for them than for high-class kids. When they apply this to some real-world data (shown below), they find a much less-pronounced narrowing of the gap between the low-ability high-class kids and the high-ability low-class ones.
Much as the critique seems plausible (and regression to the mean is one of the classic statistical fallacies), I don’t want to judge this particular statistical battle – I’m trying to persuade one of the experts in this field to write for the blog in the coming weeks, so I’ll leave the final arbitration to her.
For the time being, what interests me is the story of the rise and fall of a ‘stylised fact’ – the way that Jerrim & Vignoles’ critique has made the papers and BBC Radio 4’s fabulous statistic show ‘More or Less’, and more importantly, has been used within Government by Conservatives opposed to the social mobility strategy (at least according to these news reports). One of the reports describes the findings as, “In other words, the entire basis for the government’s social mobility strategy is wrong,” and this is how they’re being spun.
But such an interpretation seems to me completely ludicrous. The evidence for the influence of class (in a broad sense) on educational outcomes is overwhelming, including the body of excellent research by Feinstein himself, as Jerrim & Vignoles say in their well-judged response to over-excitable coverage:
“far from undermining the coalition’s social mobility strategy, our study actually confirms the previous research evidence on which the strategy is based – which has shown that children from disadvantaged backgrounds have very poor cognitive skills compared to their richer contemporaries, and that this socioeconomic gap emerges early in childhood.”
Despite this, the story that emerges is about the rise and fall of a killer chart – a chart whose rhetorical significance and the crossing of the lines came to symbolise much more than the wider (and accepted) body of research underlying it, and whose (possible) demise may lead the this wider research to be marginalised.