Monday, June 8, 2015

Measuring Success

A team works for months on a software project which is nearing completion. The PM is pushing toward the goal of getting the beast into the wild. Finally, the day has's up and an all out celebration ensues. The PM declares the project a success and everyone does shots and pats each other on the back.

Fast-forward 2 years. The product needs some changes. Several Bugs need patching, some features need modification, and some new functionality is desperately needed in order to make the product live up to its promise.

The core team is gone. The PM has moved on with the momentum of a successful project propelling him/her upward at another company. No one asked the questions about the long outlook and now its here staring the current team starkly in the face.

From an ROI perspective, this product is a complete failure since it was projected to return costs in 18 months and instead has costed more in terms of construction AND ongoing costs. Plus there were indirect costs that not a single person measured.

This fictional scenario may or may not sound familiar to you or someone you know. If so, try the following...take deep breaths and count to 10 slowly. Feel better? If not recurse, else read on.

Lets consider a few parameters of a project that is rationally based on cost savings (as opposed to increasing income). What parameters might one expect? Lets put those into some sort of cost-benefit model:

costs: Building the Beast, Maintaining the Beast, Running the Beast, User Impact, Morale Impact

benefits: Lower Insurance Premiums, Lower Risks, Increased Productivity, Staff Reduction

Some of these properties are one-time such as build costs; others are recurring like licensing, user impact, and lower premiums; some are readily calculated, others not so much (looking at you morale). For the not so quantifiable properties, lets consider a fuzz factor that modifies a cost value. Costs are negative numbers, benefits are positive numbers. Fuzz is +/-. For estimation make the numbers relative.

Building app : -40, Fuzz 10. Range -30 to -50.
Increased Productivity: 3/month. 2. 1 to 5.

I won't bore the reader with details here, rather leave those up to the reader as an exercise. In the end, compute the median, best and worst case scenarios once fuzz factors are applied to all parameters of one-time and ongoing costs and benefits. Take the high range of costs and low range of benefits for worst case and vice-versa for best case.

Once all the ranges are established we can see that an ROI in the worst case can never be achieved due to negative ongoing impacts. The main impact drivers of long-term costs are quality related. The important qualities that affect long-term costs are user experience, technical design, and technical debt. Some costs are hard to measure but easy to explain.

Suppose users of a system have had regular interactions with other people, but as a course of automation those interactions have been reduced significantly. There is certainly a benefit to having face-to-face interactions with certain people. Take the automated phone tree as a prime example.

While I'm certain automation has reduced direct costs of having staff to answer phone calls, some of the best companies have turned back towards having real people on the other side in order to elevate brand experience. Why? It increases brand value...sure there is a cost, but there is also a cost of automation - frustrated customers who feel no loyalty. We've all been there.

How much loss has there been as a result of automated phone trees? Fuzzy right? Lets say worst case you lose 10/yr at a cost of 300 for implementation. Your ROI is never achieved and in fact returns are more negative over time as a result of loss which is more costly ground to recover than customer retention. Short-term gain, long-term loss. Fuzz factor - 5/yr. Range 5 to 15. What if loss is 15 due to implementing automation? That a rapid fail!

Now lets say 2 employees leave as a direct result of product support frustrations. Now you have loss of productivity and cost of replacement which is considerable collateral damage. Not only that, but lets say other projects are directly impacted as a result of resource drain to support the fruits of the project. Would you still deem this project a success after it has been truly tested in the wild? Now the shots become draws to sleep at night.

Lets reconsider the project parameters...

Long-term success should be defined up front and measured throughout the life of the product. The recurring parameters are the most important factors throughout the life of the product. Consider the recent air-bag recalls - how much has that cost in the long-term? Would a higher initial cost have prevented the damage? Apply that mindset to your projects.

Spending more up front can reduce long-term costs, so long as the spending focuses on the qualities that will drive those costs. Plan defensively!

No comments:

Post a Comment