Monday, November 2, 2015

Cost-Benefit Analyze This!, Part (a)

This here blog post is hereby designated as Part 2(a) of the ongoing 7-ish part series "The Emperor Has No Clothes: How One Little Document Can Illustrate All That's Wrong With the Way We've Been Doing Things Since the Middle of the Last Century...And Other Stuff".

This blog entry is the one where I show that someone at AHTD appears to have manually changed a bunch of big negative numbers into a bunch of big positive numbers to beef up the Benefits side of their Cost-Benefit Analysis for 30 Crossing.* (See Part 2(a)(i) below).

This blog entry is also the one where I show that AHTD's Cost-Benefity Analysis is so fundamentally flawed that the State would see a bigger 'benefit' if if BOUGHT me a roundtrip plane ticket to Baltimore instead of letting me drive on my own dime. (See Part 2(a)(ii) farther below).

Part 0 of this series gave a little background about the Arkansas Highway and Transportation Department's 30 Crossing project and the U.S. DOT's denial of a request for $200 million in grant funding.
Part 1 delved into one table of data in the grant application to point out a few inconsistencies (as an aside to add to that post- an Acxiom employee pointed out that the employment number listed for them is the company's WORLDWIDE employment, not employment in Central Arkansas, and definitely not employment within 3 miles of the project corridor).
Parts 2(b) - 7 will keep digging.

Now let's look at cost-benefit analyses, shall we? Trust me, this is going to be fun in that shaking-your-head-angrily-while-throwing-your-blog-viewing-device-across-the-room-and-accidentally-hitting-your-pet-chupacabra kind of way!

The Setup 

When the people in our government are thinking about spending a lot our money on something, it makes sense for them to compare the costs and the benefits when evaluating the worth of said expenditure. So, I don't think anyone will be surprised to learn that just such an analysis was performed for 30 Crossing. There are many, many critical things to be said about cost-benefit analyses in general and the false assumptions, inaccurate inputs, and bad math that make up AHTD's version in particular. Given time constraints before Tuesday night's big dog-and-pony show though, I'm going to drill down into just two of them in this post and hope to circle back around to some of the others in Part 2(b) later this week or next.

Here's the basic structure of AHTD's cost-benefit analysis framework:
Image Credit: Manual Division of MoveArkansas's Graphics Department

The cost side is easy. How much will it cost to build and how much will it cost to maintain? Now, any sentient being should be able to come up with a whole host of other costs associated with massive urban highway expansion, but I'm going to live in AHTD's arbitrarily constrained world for the purposes of this post.

Benefits are similarly easy. The whole point of highway expansion in the name of congestion 'relief' is to save people's precious time so we can play more on Facebook or blog more about transportation policy when we get home. So, we're going to account for 'travel time benefit.' Likewise, if you can cut down the physical on-the-ground length of someone's trip, then that means they drive fewer miles and save money on things like fuel and car maintenance. Finally, there's a certain number of wrecks that happen per any number of miles driven. As with car savings, if we can cut down the distance people drive then the raw number of wrecks will also go down, meaning the number of serious injuries and fatalities should go down. Suppose there are on average 10 serious wrecks for every 100,000 miles driven. If we can cut down the number of miles driven to 90,000, then we should expect to see only 9-ish serious wrecks, all else being equal. That's a benefit, so count it.

Then, once you've totaled up the costs for a project and the benefits (both calculations using AHTD's purposefully narrow definitions), you can divide the second number by the first and get the all powerful Benefit-Cost Ratio! As long as that number's greater than 1, then the benefits outweigh the costs. The more bigger the ratio is than 1, the more benefit-ier the project.

All it requires is adding and dividing really big numbers, which is what these days, 3rd grade math? 4th grade?

Anyway, here's what AHTD came up with to put in its TIGER grant application to the feds:


So, 30 Crossing has a Benefit-Cost Ratio of 7.37! In other words, for every dollar of upfront cost in building the darn thing AHTD says there will be $7.37 in benefits. Oooooh, nice! We can't afford NOT to do this project, amiright?! (We're going to ignore that discounting stuff on the right. It'd just be an unnecessary distraction for now. Note also that this is not a break-even analysis... another topic for another day).

Well... back to that grade school math...

Blog Post Part 2(a)(i): AHTD added when they should have subtracted


Ryan Gosling says wait for it...
Seriously. Here's the spreadsheet used to generate those numbers above. Two of their columns in the benefit side should be negatives instead of positives*. I know looking at spreadsheets can be as dry as last year's chupacabra scat still sitting in the Mojave Desert, but looky here...
Image credit: Digital Division of MoveArkansas's Graphics Department
That's Page 1 of the spreadsheet. We're just focused on the columns in black font to avoid a messy discussion about the time value of money. The far left column shows the construction costs. (note that this doesn't include any maintenance costs- an oversight to be covered in a later post). The next column  shows the Travel Time Benefit in each of the years between now and 2041 with a total at the bottom. Then there's the Vehicle Operation Cost Benefit and finally the Safety Benefit. The time column is a result of a change in the number of Vehicle Hours Traveled. Vehicle Operation and Safety benefits are functions of the change in Vehicle Miles Traveled.

Still with me? Excellent. Fortunately, we're going to avoid the big ol' elephant in the room that is the time savings benefit and just focus on part of the vehicle and safety benefits. So, we're only concerned with the change in Vehicle Miles Traveled (VMT) that 30 Crossing would induce. Here's that spreadsheet:
Image credit: Digital Division of MoveArkansas's Graphics Department
Column 1 above shows the anticipated Daily VMT after the project is built; Column 2 shows anticipated Daily VMT if the project were not built; and Column 3 shows the reduction in VMT when comparing building the darn thing with not building it. Logically, turning a 6-lane highway into a 10-lane highway will allow more people to drive, leading to more Vehicle Miles Traveled. And sure enough, AHTD's model bears this out. See how those numbers in the first column are bigger than the numbers in the second column? That means the 'reduction' in VMT is a negative number, as shown in the third column. In other words, there's no reduction. The 30 Crossing Project will cause an increase in VMT. There's absolutely nothing earth shattering about that. It's a highway expansion that doesn't create any new short cuts. I just tried to make the expectation of increase abundantly clear enough to set up the travesty that is the next slide:
Credit again to MoveArkansas's awesome graphics team
Ok, Column 1 here comes from Column 3 in the previous sheet: the reduction in automobile VMT (they apparently just gave up when it comes to trucks, so we will too). AHTD calculates the per mile cost of car operation to be $0.398 (also a topic of a future post). Remember, the numbers in the Reduction of VMT column are negative numbers, because we're talking about a huge highway expansion that will increase VMT. Multiply the 'reduction' by the cost per mile and get the answer in the Daily Benefit (2014) column. Take that daily number, multiply it by 365, and voila- there's the final column of annual benefit resulting from reductions in the cost of car operation.

"But, but, but, Mr. McKuin, I thought you taught us in 9th grade maths class that a negative number multiplied by a positive number results in a negative number?!" And you would be correct, grasshopper. AHTD magically turned the extra costs associated with an increase in VMT into a benefit for the drivers. Those are the numbers that showed up in that first sheet way up top showing all the cost and benefit numbers.

They made the same 'mistake' with the Safety Benefit.
Credit: Our amazing graphics department, again. Love those guys!
VMT goes up with the highway expansion. Logically, more people driving more miles results in more wrecks, all else being equal. Death and maiming is a negative benefit to society. AHTD calculates the cost of such death and maiming to average out to about $0.379 per mile driven. So, more miles driven should result in negative benefit. But, AHTD magically dropped that negative sign again.

Still don't believe me? Well, here's the CBA from the TIGER grant application we sent to the feds to help pay for an interchange on 67/167 in Cabot. Note the much more nuanced respect for positive and negative numbers in it.

Let's fix 30 Crossing's Cost-Benefit Analysis:
Credit: co-effort between the Graphics Department and our in-house team of abacus experts.

The benefits, as defined by AHTD, actually total up to $3,802,241,633 between now and 2041, give or take a dollar or two. So, the Benefit-Cost ratio is only 5.85 instead of 7.37.

---------------------------

This might seem like a basic, honest math mistake. What's the difference between 5.85 and 7.87, really? We have to remember though- these analyses are part of what the largest department in Arkansas State government generated and submitted in an application for $200 million to help offset some of the costs for what will be the most expensive single contract in Department history. Was this just an honest math mistake or a deliberate attempt on someone's part to make the project look better on paper? Another way of looking at this: if you peruse other CBA's on AHTD's website, you quickly see that they tend to use the same spreadsheet format for all of them. Hopefully that template has the correct formulae baked in from the get-go so nobody's having to reprogram it every time. If that's the case then someone had to manually change the numbers in the CBA for 30 Crossing and put the result in our TIGER grant application.

---------------------------

As hinted at periodically above, there are other massive problems with how the C-B analysis works, and I hope to get to some more of those in Part 2(b) later this week.

-----------------------------

Part 2(a)(ii): Using their own methodology and numbers, it makes more sense for AHTD to buy all of us round-trip plane tickets to Baltimore than to proceed with 30 Crossing!


This was actually the impetus for this whole post, but in working out the numbers I ran into the problem that dominated the discussion above. So, I'll make this fast...

AHTD's cost-benefit analysis computes the cost of a public expenditure on one side and then adds up three benefits on the other: time saved, vehicle operation costs saved, and safety savings.

They say my free time is worth $15.05 per hour, it costs $0.398 per mile to operate my car, and the costs of death and serious injury from traffic crashes works out to be about $0.379 per mile.

So..., let's take a trip! Hmm... I hear Baltimore is great in January. I've got some friends and family up there I haven't seen in a long time, and I could bop on down to D.C. and visit the Department of Transportation to learn more about TIGER grants. Flying always seems so excessive though, so I'll probably drive. Shortest route, of course:
Google Maps

The Great Google says it's 1,051 miles one way, so 2,102 round trip. Also, 15 hours and 42 minutes each way, or 31:24 round trip, or 31.4 hours.

Oh, what the heck. Let's look at flying just to see what we're missing. Wow, cool. Southwest has a sale right now for January between LIT and BWI for just $215.96  and it'll only take 10.67 hours round-trip:
Southwest Airlines

Using AHTD's numbers for not driving, my operating savings would be 2,102 x $0.398 = $836.60, and society's safety savings will be 2,102 x $0.379 = $796.66. The time savings would be worth (31.4 - 10.67) x $15.05 = $311.99.

Add 'em up: $836.60 + $796.66 + $311.99 = $1,945.25 in Benefits and only $215.96 in Costs. That means the Benefit-Cost Ratio is 9.01!! My lands, that's bigger than the doctored numbers used for 30 Crossing! You can't afford not to buy me a plane ticket to Baltimore, people.

Ok, AHTD, let me know when you're ready to book the ticket and I'll send over my details. Thanks.

------------------------

*Of course, as is my tendency, it is so hard to wrap my head around the possibility that this egregious of a mistake could have been made in the first place and then not been caught by anyone in the review/proofreading chain before sending it to Washington in a request for $200 million, my strongest reaction is to think I must be wrong. But, just like noting that 6 and 10-lane highways are not the same thing as 4-lane highways, I can't figure out exactly how I'm wrong here. Please feel free to point out my error if you can find one! I think we'd all sleep better at night knowing some hack two-bit blogger messed up this very basic math instead of a skyscraper full of engineers.

3 comments:

  1. I'm kinda reading this with my jaw on the ground. That's pure criminal accounting!

    ReplyDelete
  2. To a hammer everything is a nail....

    ReplyDelete
  3. You are correct.

    First of all, this needs to be reported to the city council.

    Second, this needs to be submitted as a formal public comment in the Environmental Impact Statement process.

    Because this can be used to *invalidate the entire process* and stop the project. Take the information to court and you can throw out the sutdy and make them start over.

    ReplyDelete