Right agenda, flawed programme: lessons to be learned from Troubled Families
Lessons to be learned from the Troubles Families programme
In two articles for LGC and Public Finance, Localis chief executive Liam Booth-Smith writes on the lessons of the Troubled Families programme. A recent National Institute of Economic and Social Research report suggests that the programme has had no measurable effect on school attendance, employment or behaviour.
For LGC, he writes on why, for all the programme’s criticisms, the troubled families agenda is still worth fighting for – you can read the full piece here or below. For Public Finance he considers whether we have had a ‘Moneyball moment’ with Payment by Results- you can read the full piece here or below.
The troubled families agenda might still prove a success
“Style is not neutral,” wrote author Martin Amis, “it gives moral directions.”
How we choose to frame things matters. References, idiom, syntax, all betray intentional and unintentional meaning. For politicians and the government, a choice of words is especially important. You’re judged not just on performance but the measure of meaning you attach to words. We probe them for hidden messages, coded language, for direction. Often times such coda isn’t there; banal cliché being a stout defence against double-speak. The goal is to say exactly what you mean, or say nothing at all.
Reflecting on the recent Troubled Families debate I’ve been most struck by government’s failure to understand this. Take the elastic snap of the ‘troubled family’ definition. Stretched to great length it covers all manner of ills, only to whip back to its original size, trapping a legion of problems under the banner ‘troubled’. It neither says what it means, or nothing at all. Louise Casey, the civil servant who set up the programme, conceded even the phrase ‘turned around’ shouldn’t be used in the future to describe the impact it has on a family because it gave an inaccurate impression.
I share this because I’ve always been careful to draw a distinction between the Troubled Families programme and agenda in my own writing. The semantics mattered a great deal to me when I realised that many places had begun to use the impetus of the programme to start working differently with other government agencies locally. Much of it had nothing to do with the Troubled Families programme, but the latter was undeniably the progenitor. I made the same point last year to Durham University’s Stephen Crossley, a vocal critic of the programme, who shared his excellent work with me on both the failures in data reporting and definition of the families. Since our exchange I’ve become more sceptical of the programme, but still maintained my belief the agenda is worth arguing for. Recent news hasn’t dissuaded me of this position.
It may be a minority opinion, but I still believe the troubled families’ agenda got its central analysis correct. Certain families and communities have been locked into intergenerational and cultural deprivation for decades; traditional services just haven’t reached them. We needed to try something different which cut across silos and respond to the actual barriers they were facing to create a material difference in their lives. Even the Ecorys independent evaluation acknowledges the problems these families faced were far more ‘entrenched’ than initially thought.
How does one measure success when dealing with chronic psychological issues or when, as Jenni Russell’s article in last week’s Times noted, someone doesn’t have doors in their home or a working bathroom? When you’re dealing, en masse, with the most complex social cases, what control group can accurately offer a comparison? As the philosopher Bryan Magee wrote, “out of the crooked timber of humanity was nothing straight ever made”. We’ve learnt that data and reporting needs to improve, we’ve also learnt (as if we ever needed to) that payment by results creates perverse incentives, but what we haven’t learnt definitively is that the programme was a failure.
Despite his own consistent criticism, even Jonathan Portes agrees that many social policy experts thought the “basic underlying principles of the programme made a lot of sense”. These underlying principles – inter-agency cooperation, a focus on wider social determinants, a family focused approach – are based on the experience of people working on the front lines in social services. They know too many kids come into care because of the behaviour of Mum and Dad, or that there’s no way some will be able to excel at school until the domestic violence at home stops.
The troubled families programme wasn’t really a programme. It was an experiment; a messy, complex, but worthwhile experiment. Similarly the families it helped weren’t turned around, they just took their first steps in a new direction. But when you haven’t been able to move someone for decades, can a few steps fairly be described as having ‘no impact’?
Our choice of words matters. Those criticising the programme have grounds to do so, but they’re also bound by the same rules of semantics. The evaluation – the prime source material for those now on the attack – stated that it’s possibly “too soon to draw a conclusion”. I’m willing to accept that the troubled families agenda might be a failure. Time will tell, but are critics willing to accept that it still might be a success? I guess this depends on your definition of success. Style is not neutral.
Troubled Families: the Moneyball moment for payment-by-results?
The 2003 book Moneyball is the real life story of how the Oakland Athletics baseball team general manager, Billy Beane, and his assistant Paul DePodesta, turned the sport on its head by using a new analytical, data driven approach to building their team. I recently watched the 2011 film, starring Brad Pitt and Jonah Hill, based on the book and there is one line that’s stuck with me. Hill’s Peter Brand (a composite character of Beane’s assistants) is explaining why “baseball is in the dark ages” and how teams’ don’t understand what they’re actually buying when they sign a player. “When they look at a player they see athleticism and a great swing, when I look at him I see a set of statistics. The purpose isn’t to buy players, it’s to buy wins.”
The central message is figure out what you really want to achieve. Last week’s Troubled Families row is a case in point and in particular in relation to ‘payment-by-results’ contracts. Payment-by-results (PbR) in principle makes sense, government wants an outcome not a process so will notionally pay on receipt of the outcome. But relying on the provider to collect the data has repeatedly proven problematic. The incentive will always be to make the result fit the contract. At its worst PbR pays almost no attention to results, and becomes by ‘payment-by-activity’. The head of the National Audit Office, Amyas Morse, has previously pointed out the dangers for commissioners in “using this mechanism in circumstances to which it is ill-suited, to the detriment of value for money.” This is certainly the line of attack being used against the Troubled Families programme.
How you judge what constitutes an effective outcome when dealing with complex social problems, let alone contract for it, is one of the most important legacy questions created by this programme. At the Public Accounts Select Committee hearing last week Department for Communities & Local Government permanent secretary Melanie Dawes acknowledged that PbR put “too much focus on the short term”. Her colleague and programme lead Joe Turke similarly suggested that it may not be the “best way” to incentivise what the department wanted. But with the Treasury demanding value for money, and rightly so, it’s difficult to see what alternative approach could have been deployed at the pace required by our political masters.
What’s clear is that PbR is a damaging route to failure if you’re not completely clear on what win you’re trying to buy. Its emphasis on empirical evidence exposes weakness in data. The central criticism of the Troubled Families programme has been the evidence doesn’t show a “significant impact”, either because no such impact was created or we haven’t found a way of collecting and measuring it correctly. One person I spoke to responsible for running a Troubled Families programme in their local authority area said getting access to other agencies data was “painfully slow”.
The issues being dealt with were also far more entrenched than initially thought. If I had a serious criticism of DCLG and the Troubled Families programme it would be that they weren’t specific enough about what they were trying to achieve. In deploying a contractual mechanism that is sensitive to provider context, but blind to nuance in outcomes, for a national programme where the criteria for success was so broad, Troubled Families was always going to be more experiment than programme. In this regard we’ve learnt some valuable lessons about using PbR at scale.
To use PbR effectively commissioners need to be sure they’ve had a Moneyball moment. They need to get past the initial activity and focus on the precise outcome they’re trying to buy, the win. With something as complex and broad as the Troubled Families programme, however, the difficulty is there’s no single definition of a win.
Photo credit: Thomas Leth-Olsen Family walk via photopin (license)