Is Failure Necessary to Breed Success?

Being concerned that the failure rates of an organisation are too low is rare. But is it madness or genius?

“I thought that was brilliantly insightful,” says Tim Harford of the time his then boss at the World Bank made such a complaint. According to its internal evaluation unit, World Bank projects had something like an 85 per cent success rate. Harford’s boss said: “That’s far too high. Either the figures are being rigged or we’re being far too conservative. We do business in very difficult environments, successful ideas have a huge payoff. We need to get the failure rates of this organisation up.”

Harford’s latest book, Adapt: why success always starts with failure, is based on the idea that because the world is so complicated it’s not possible to get things right the first time. Whether the aim is to reduce carbon emissions, defeat the insurgency in Iraq or run a hospital, things will constantly turn out in an unexpected way.

“The question then is all about error collection,” says Harford. “How do you generate new, potentially useful ideas, how do you spot what’s not working and how do you close it down before it does any damage.”

But surely that can’t apply to the NHS? Lives are at stake. Public money could be wasted.

It all depends on the context, says Harford. In Silicon Valley, constant experimentation is desirable because there’s a potentially huge upside and if an idea goes pear-shaped no one is going to die. Experimenting at a nuclear power station – such as switching off several safety systems to see if the reactor can cope – has few upsides (at best it will continue to perform as expected) and the downside is huge. A similar dynamic exists in finance, where the potential for things to go catastrophically wrong is large.

“I would put healthcare delivery somewhere in the middle of that,” says Harford. “You don’t want constant, undisciplined, random experimentation. But you do want people to be trying new things out with appropriate regard to safety.”

The greatest system for experimentation that has ever been designed is all about evaluating the effectiveness of health treatments – the randomised, double-blind controlled trial. “That is trial and error and that’s experimenting with people’s lives,” says Harford. “We’ve got over that [because] we’ve realised that we don’t really have an alternative.”

This way of experimenting is considered acceptable because it has to be done in a systematic way and with proper oversight. It shows that experimentation is possible in healthcare. But politics make it awkward. How many managers relish questions about why public money was wasted on a new computer system that was defunct within a year or on redesigning emergency services with no impact on waiting times? Being accountable leads to extreme risk aversion.

But Harford says voters are also at fault: “We’re the ones who complain when things go wrong.”

Take the scrapping and then reinstating of the radio station 6 Music as an example. Just after it was announced that it would be axed, BBC director general Mark Thompson was asked by John Humphrys on the Today programme if he was going to apologise for wasting money on the station. 6 Music was later reprieved, but the point is that because it was being stopped it must have been a waste, not that it was worth a try. Harford says: “People are facing this idea that the success rate has to be 100 per cent or you’re not doing your job right.”

Higher tolerance

What level of failure is acceptable in order to get those great results eventually? There are two different questions there, says Harford. What level of failure should be acceptable, and what level of failure is acceptable. “I think we give people very strong incentives to be opaque, not to be transparent about their results, and not to evaluate anything rigorously. Just try some stuff out and cover it up if it doesn’t go well.”

That mitigates against sharing best practice, sharing lessons learned, and properly evaluating because proper evaluations tend to become public. “I think that we should have a much higher tolerance of failure if it’s the right kind of failure,” he says.

Again, it depends on the context. A bit more experimentation and therefore risk of failure is something that voters, politicians, business leaders, organisational leaders and individuals should encourage and allow. A failure of ethics, safety procedures or basic competence is unacceptable. But it is perfectly acceptable to try out something uncertain which has a reasonable chance of success and in the end doesn’t succeed. “That’s called experimentation and it should be okay,” says Harford.

In the health service a new project might be deemed worth piloting because if it doesn’t work there will be small financial cost and some minor disruption. Whereas if it does work the organisation will gain a lot and other departments and hospitals can copy it.

Harford gives a wonderful example in Adapt of a bureaucrat in the British Air Ministry. In the 1930s Air Commodore Henry Cave-Browne-Cave approved a prototype fighter that was a long shot. The prevailing belief at the time was that fighters were redundant and only bombers were necessary. It was also thought that a fighter should have two seats, not one. “This proposed aeroplane was the wrong design to do a job that couldn’t be done anyway,” says Harford. “He funded it and it turned into the Spitfire, the plane that saved the free world.”

The prototype cost the government just £10,000. That’s about £500,000 in today’s money or, as Harford puts it, roughly the price of a nice house in London.

Ray of hope

It remains to be seen what role trial and error will play in the new NHS. Will the restructuring – which was conceived as a grand vision from the top – help or hinder experimentation? “It’s a sort of highly centralised, highly ambitious, attempt to decentralise,” says Harford. “So there’s a tension here.”

Decentralisation can lead to different places doing different things to varying standards – a phenomenon that many people regard with suspicion, referring to a postcode lottery. It doesn’t worry Harford. “I think that diversity has its cost but it has massive benefits as well if through that decentralisation you get new ideas emerging and those new ideas spread,” he says. So that’s the glimmer of hope.

But he is generally pessimistic about the reforms. First, there does not seem to be any evidence base for the changes and it appears to be an act of faith that it’s going to work. Second, everything looks to be changing at once. “Step by step is normally a good idea because the world is so complicated,” says Harford. “Things are going to go wrong, right? Which is why you do things gradually.”

Third, it’s unclear how best practice will be spread. He says: “When this decentralisation happens and the diversity of experiments emerges and a thousand flowers bloom – let’s be optimistic – what then is the mechanism by which people copy successful experiments? It’s not clear.”

And what are the incentives to copy successful ideas and shut down unsuccessful ones? Any system – be it market based or regulated – needs both.

Another unanswered question is whether organisations in the reformed NHS will be reluctant to share good ideas because they will be competing for business. In a market what usually happens is the company that develops the successful thing either grows quickly or starts buying other companies. Or sometimes managers leave that company, set up their own companies, and use the knowledge they’ve learned.

“In a cluster of organisations like those that make up the NHS it’s not clear that that dynamic is really at work,” says Harford. A hospital that comes up with a novel and effective way of dealing with patient appointments is unlikely to start buying other hospitals and installing that system. Similarly, staff at a successful hospital are unlikely to leave to set up their own new hospital. Harford says: “It’s possible to imagine a healthcare delivery system in which it did work but it would be so different from anything we’ve had in this country for 60 years I just don’t think that’s realistic.”

Harford is often asked which are more experimental, small or big organisations? It’s pertinent to the reformed NHS – will small commissioning groups owned by employees be better placed to experiment successfully? “There’s a straightforward trade-off,” says Harford. “To successfully experiment you need a diversity of different approaches and you need to have a very clear idea of what’s working and what’s not.”

Small organisations suffer on diversity because they can’t make lots of different bets, they have to put all their eggs in one basket. On the plus side, they tend to be much more sensitive if something’s not going well.

On the other hand, really big organisations such as Tesco or Google can experiment a massive amount. The challenge there is knowing what’s going on. What often happens is staff create their own fiefdoms, are not answerable to anybody, and the top of the organisation doesn’t know what’s going on at the bottom. Large organisations with good information flows for what’s going on tend to be very good at experimenting.

With small organisations the challenge is to allow experiments without killing off the organisation. Or to create a situation where it doesn’t matter if the organisation gets killed off. “If we had a situation where commissioning groups could disappear and be replaced by new commissioning groups – they could just go bankrupt because they got it wrong – that would be fine as far as the innovative dynamic was concerned,” says Harford. But he admits it’s unlikely that anyone is planning to see that happen.

Harford writes about the importance of keeping communication open between the front line and bosses at large organisations. It’s because details matter, and the top dog often does not see where things are going wrong or going spectacularly right.

In Adapt he describes how General David Petraeus – who recovered the US military’s situation in Iraq – rewrote the US army doctrine on counterinsurgency by canvassing ideas from all quarters, including dissenters. “That was partly about getting the very best ideas and also partly, very cleverly, about building consensus because everybody who mattered got to have their say,” says Harford.

When Harford spoke to Petraeus, he said leaders have to communicate the vision and provide a clear sense of direction, then leave people on the front line to figure out how to deliver it because ultimately they have to get the results on the ground. After all, you can’t micromanage an army.

Trust your people

“I think the same has got to be true of an NHS trust,” says Harford. “You can give strategic direction but in the end you’ve got to be trusting people further down to deliver in the best way they know how and holding them accountable for the results.”

It takes an enormous amount of courage to be a leader who encourages trial and error and accepts that it can lead to success or failure. Each chapter of Adapt has a hero who put his or her neck on the line and dealt with a tremendous amount of pressure before finally producing the innovations that mattered. It’s not how Harford conceived the book, but that’s how it emerged. “I’d like to say there is a really painless way to do it without taking any risks and everyone will applaud you but that just doesn’t seem to be the way it’s worked out,” he says.

His advice to NHS managers in a complex world of fast changing reforms and financial challenges is to “make lots of small bets which may well fail but have a potential big upside”.

“Make sure that you are disciplined about measuring which of those bets is succeeding and which is failing,” he adds. “And don’t be afraid to very quickly get rid of the ones that are failing and put your weight behind the ones that are succeeding. Keep repeating until retirement.”