Cost Benefit Newsletter No. 185, ´ROI That Never Arrives: The Devil is in the Assumptions´
ROI estimates in business fail primarily because managers give too much attention to the "pay out" odds, and too little attention to measuring and managing "probability" odds. A good risk and sensitivity analysis of the assumptions behind the predictions allows you to do both.
Somewhere between 70 and 80% of all major IT project proposals in the industrialized world now require a return on investment (ROI) analysis before funding. The figure varies, depending on which analyst you read, but there's no question that the "ROI Required" rate is high and going higher
The same analysts then turn right around and tell us that more than half of such projects never deliver the expected ROI. Several years ago, for example, Nucleus Research published a now-famous study after interviewing SAP reference customers. Nucleus found that 57% of them did not receive a positive ROI. Yet SAP had for years made return on investment (or, "return on information") a centerpiece of its marketing and sales strategy.
"Why then," you might ask, "do decision makers continue to focus on ROI when so many ROI figures don't predict what actually happens? If ROI can't do any better than that, doesn't ROI cause more bad decisions than it prevents?
Two Kinds of Odds
The term "odds" means two different things to gamblers. Serious gamblers know that one kind of odds has to be weighed against the other kind, in order to maximize the chances of coming out ahead.
At the racetrack, "odds" can mean (1) the ratio of the winning pay out to the cost of the bet (just like simple ROI !). A winning $2 bet on a 20-to-1 long shot puts $40 in your pocket. The skill in betting lies in knowing how the pay out odds compare to the other kind of odds, (2) the actual probability of winning.
One glance at the tote board shows you the current pay out odds. (One glance at the bottom line of your ROI spreadsheet shows the predicted returns on your business investment, if everything goes as planned).
However, if you do go to the races, just watch the serious gamblers working, furiously, down to the last minute before the betting windows close, trying to estimate the probability odds. They visit the paddock area to have a look at the horse, they pour over racing forms and tip sheets, and they work and re-work their racing calculators. Otherwise, they will tell you, it would all be "luck" and nothing more.
If the gambler estimates that a horse has a one-in-five chance of winning (20% probability of winning), but the posted pay out odds are 20-to-1 for a win, that is a "good bet." (in statistical terms). If only business people went through the same thought process.
I believe that ROI estimates in business are often unlucky because managers give too much attention to the pay out odds and too little attention to the probability odds.
Very few know how to how to maximize ROI by measuring and managing uncertainty.
Assumptions, Assumptions, Assumptions
Every forward-looking ROI estimate and business case stands on a foundation of assumptions. It can be no other way, after all, because the case predicts the future. Everything about the future is an assumption. A detailed case for a major business investment may require dozens or hundreds of assumptions about such things as future:
- Business volume
- Competitor's actions
- Market share
- Fuel prices
- Government regulation
- Software development time
- Salary increases
- Productivity improvements
- Implementation costs
- "Ramp up" time
- Labor requirements
- Raw materials prices
Some assumptions come with high certainty, others come with high uncertainty. But every assumption adds some uncertainty to the bottom line cash flow or ROI estimate. It's tempting to lump all these factors together into a single risk figure.
The key to managing your investment so as to minimize risk and maximize returns, is not to treat risk as a single factor, but instead carefully weigh individual risk factors to determine:
- Which assumptions carry the most weight in driving results?
What happens to results if we don't manage critical success factors to target levels?
Sensitivity analysis addresses questions like these. - What is the likelihood of other results besides the most likely predicted outcome?
Which risk factors have to be watched carefully, as indicators that predictions need revising?
Risk analysis answers questions like these.
One of our clients, for instance, had a proposal to begin a major ERP system implementation. This bottom line prediction was an incremental cash flow gain of $19.5 million over five years. That was a good looking "R" on an "I" of about $3 million.
Here, however, is part of what turned up in a sensitivity analysis of individual assumptions:
The table shows the correlation between each assumption and the overall projected results, based on Monte-Carlo simulation with the same financial model that predicted a $19.5 million gain.
There were some messages here that made management sit up and take notice.
- The ROI model assumed the ERP system would help lower new product development time by 30%. The high correlation between this assumption and financial results shows how the attractive ROI depends on reaching that target. In this case, a reduction of only 15% in development time (instead of 30%) brought the predicted gains down to $5 million.
- The ROI model also assumed the ERP system would be fully implemented in 2 years (50% implementation per year). Clearly, missing the 2 year target would adversely impact the overall return on this investment. Stretching the implementation out to 4 years would raise costs and lower benefits to bring a zero net gain.
It's one thing to know generally that factors like implementation time and reduced development time are important. It's another thing entirely to bring home the importance of those factors in concrete terms.
If you know ahead of time where the risks are, you can manage them (or at least watch them) and avoid unpleasant surprises down the road.
Take action! Learn more about business case building from the Business Case Guide. Learn and practice proven methods for building your cases at a "Building the Business Case" Seminar.
Marty Schmidt
4 May 2009
mschmidt@solutionmatrix.com
www.solutionmatrix.com
The figure above is one of several Monte Carlo output screens from Crystal Ball, a risk, sensitivity and forecasting tool from Oracle . Crystal Ball ® is a registered trademark of Oracle.
© Copyright Solution Matrix Ltd. 2004 - 2010
