American Dream

2005.08.25
FORECASTING
S&P continues critique of forecasts - update finds further faults

First few paragraphs:

Bond rater Standard & Poors (S&P) continue to find serious fault in the forecasts for new pikes. A report on an extension of their database of forecasts shows no sign of improvement and casts doubt on the notion that forecasts improve significantly a few years out from startup.

"Optimism bias - overforecasting asset use - and error remain prevalent," they conclude.

The average outcome for first year traffic in S&P's sample of 104 projects is 0.77 of forecast. That is: a prudent reader of forecasts will on average get a better view of prospective traffic by immediately multiplying any forecaster's figures by 0.77 - discounting 23% of the traffic number.

This is the basis for S&P's harsh conclusion that forecasting presently has a "systemic tendency toward optimism bias."

And the standard deviation is quite large at 0.26 - the same as a year ago - so great uncertainty will remain about any particular forecast after correcting for systemic bias.

Not just ramp-up problem

The past few years have seen forecasters saying they think the problem has been misestimation of 'ramp up'. The implication has been that although the first year traffic forecast is uncertain the forecasts improve for subsequent years - once the facility has matured and had time to develop a clientele the forecasts look better. (We've subscribed to that view.)

It was true, near here, of the Dulles Greenway Dulles-Leesburg VA. It was woefully off forecast in its first 2 to 3 years but came
good and now 10 years later is very much on the track of the original Vollmer forecasts. That forecast was characterized as a 'ramp-up' estimation problem.

However if ramp-up is the most troublesome aspect of forecasting, then forecasts should on average look better as the years pass.

There's a tiny hint of this in the S&P data. The average traffic/forecast ratio rises just a little for each year: Yr1 0.77, Yr2 0.78, Yr3 0.79, Yr 4 0.80, Yr5 0.79. But the improvement is small and S&P analysts dismiss it as statistically insignificant.

Looking at the graphs of traffic/forecast for a bunch of projects there does seem to be a generally rising trend for all but a handful. Those handful drop precipitately after exceeding year 1 forecast and are enough to virtually wipe out the gains of the larger number of forecasts showing an improving performance over 5 years.

S&P report however: "If actual traffic performance had systematically improved over time (in comparison with their respective forecasts) a general upward trend in the ratio of actual to forecast traffic to more than 1.0 would be observed over time. It is not. Instead, a mixed picture emerges, with a number of case studies failing to match their forecasts by Year 5 or, in some cases, beyond. Clearly some caution is required at this stage, because our sample size prohibits the drawing of definitive conclusions. This preliminary analysis, however, suggests that there is no automatic improvement in traffic forecasting accuracy after Year 1. The extent of optimism bias and error in the case study traffic forecasts from Years 2 to 5 is similar to that observed for Year 1 data... (N)either the mean of the distribution nor its standard deviation alter significantly during the first five years of operations."

Their data for the out years are pretty limited. Of the 104 case studies only 41 go back 5 years or more, and 71 back 4 years or more. So in our view they have not entirely demolished the notion that a good bit of the forecasting error is in the ramp-up period.

[see URL above for more]