The hidden costs of cost-benefit analysis

The hidden costs of cost-benefit analysis 13th January, 2022

I began last week’s column by declaring that life is full of difficult decisions. Many of us, faced with such dilemmas, resort to a list of pros and cons. Perhaps the most famous such list is that composed by Charles Darwin in July 1838, as he contemplated proposing to his cousin Emma Wedgwood.

Under “Marry” he included: “Children — (if it Please God)— Constant companion, (& friend in old age) who will feel interested in one . . . better than a dog anyhow . . . Charms of music & female chit-chat.— These things good for one’s health.— but terrible loss of time.”

“Not Marry” included: “Freedom to go where one liked . . . Conversation of clever men at clubs — Not forced to visit relatives, & to bend in every trifle.— to have the expense & anxiety of children — perhaps quarrelling — Loss of time.”

In the end he decided that even after double-counting “loss of time”, the charms of “a nice soft wife on a sofa” were undeniable. Charles proposed. Emma accepted. I am not aware if she made her own list before doing so. The couple had 10 children.

Economists have their own version of the list of pros and cons, called cost-benefit analysis. The principle is very similar, although the hope is to be a little more systematic. Darwin, for example, did not quantify the relative merits of the conversation of clever men at clubs and the charms of music & female chit-chat. For a proper cost-benefit analysis that will not do; clever conversation may be worth twice as much as female chit-chat, or half as much, but the relative value must be quantified. (Dollars are a convenient unit of measurement, but what matters is not that everything is given a dollar value, but that everything is compared on the same scale.)

On the surface, cost-benefit analysis is all very rational, and admirably well-suited to decide whether to build a bridge or abolish a regulation. However a new article in the Journal of Benefit-Cost Analysis by Bent Flyvbjerg and Dirk Bester argues that cost-benefit analysis “is broken”. Flyvbjerg and Bester assemble and analyse a data set of more than 2,000 public projects — bridges, buildings, bus transit systems, dams, power plants, railways, roads and tunnels. They conclude that costs have been systematically underestimated. This is broadly true regardless of when or where the project was commissioned, or what type of project it is.

Even allowing for this tendency for projects to run over budget, there are more catastrophic cost overruns than one might expect. There is less data on benefits, but there is some evidence of benefits being overestimated for public transit projects, and no sign that the large cost overruns are in any way compensated for by surprisingly large benefits.

These findings are sobering, if not entirely surprising. As Flyvbjerg and Bester are at pains to point out, “cost overrun” isn’t really the right description of what keeps happening. “Cost underestimate” is better. The problem is not that every project engineer in the world is incapable of delivering to a reasonable budget; it is that the budgets are never reasonable. The infrastructure we see around us is born of rose-tinted spreadsheets and hype.

So what is the point of cost-benefit analysis if all the costs and benefits being analysed are flights of fantasy? The only answer I can think of, with apologies to Winston Churchill, is that cost-benefit analysis is the worst form of evaluating decisions, except for all those other forms which have been tried from time to time.

Alternatives include decision-making by HIPPO (highest-paid person’s opinion), decision-making by sound bite or decision-making by whatever polls well with people who may not have spent a moment’s thought on the matter. These are not good ways to think through complex, long-term infrastructure projects or wide-reaching regulations.

In his book The Cost-Benefit Revolution, Cass Sunstein (an academic and Obama administration official) argued that cost-benefit analysis was a useful way to structure decision-making, forcing us to think systemically, articulate our assumptions, ask good questions and notice where we lack good answers. This might be true; then again, it might not.

An alternative view is that cost-benefit analysis is a black-box process that hides inconvenient assumptions and shuts down debate.

I find myself reminded that cost-benefit analysis is itself a kind of algorithm. Last week I argued that algorithms serve us best when they are properly scrutinised and tested. That is certainly true for cost-benefit analysis; it should help to structure decision-making and inform public debate, rather than to bypass both.

Having argued that cost-benefit analysis is “broken”, Flyvbjerg and Bester propose fixing rather than replacing it — for example by improving the accuracy of cost estimates through better data, independent audits and performance incentives. I agree. The method is open to misuse but is too valuable to abandon.

Darwin, in the end, seems to have made his decision in an outburst of emotion: “My God, it is intolerable to think of spending ones whole life, like a neuter bee, working, working, & nothing after all.” That is probably as good a way as any to decide on marriage. But if the question is whether to build a high-speed railway line, to deny and then issue emergency visas to truck drivers, or to leave a large trading bloc, decisions made in an outburst of emotion may be decisions we will find ourselves regretting.

Written for and first published in the Financial Times on 8 October 2021.

The paperback of “How To Make The World Add Up” is now out. US title: “The Data Detective”.

“One of the most wonderful collections of stories that I have read in a long time… fascinating.”- Steve Levitt (Freakonomics)

“If you aren’t in love with stats before reading this book, you will be by the time you’re done.”- Caroline Criado Perez (Invisible Women)

← Assorted Links