I'm not going to pretend I have the answer to that. If I did I'd be writing a book instead of a blog and all my code would be written for fun and curiosity because I wouldn't need to write code for a living anymore. However, I have a suggestion for where the problem lies, and that is in estimation.
There are lots of ways to be wrong during the estimation process... it's a lot easier to be wrong than right. I think those reasons have been covered ad nauseum. You know the least about the project in the beginning, you don't know if the client has described their needs correctly, you aren't sure the client even knows their needs or has the ability to articulate them and, above all, you don't know what you don't know. For the most part, that stuff is given and there's very little you can do about it.
Many software companies try to use past performance on similar features to indicate future results. However, in general past performance on a feature has very little to do with future results unless the problems are identical. Most people are familiar with the concept that sometimes it's the last 5% of the work that takes 90% of the time. So we know and accept that small differences in functionality can make huge differences in development time.
There are several reasons agile projects succeed where waterfall projects fail, first is adaptability. It's easy to succeed when you simply modify the definition of success to fit into what you can accomplish given other constraints. However, one could also say the definition of success is usually wrong from the beginning so hopefully it moves correctly to a more realistic definition.
Second is brutal honesty as early as possible... and this is probably where most agile projects that fail fall apart. Being brutally honest with clients is not very easy and no one's idea of a good time.... let alone doing it before anyone else perceives it as a problem. The logic enters in that "maybe I can fix this before anyone notices it was a problem". 63 hours later in a task that was supposed to last 8, you are forced to admit that your attempts have failed. The difference is that now you have reached the same result and there are 10-12 other tasks that have suffered because of it. So you try to work extra to catch up causing more problems in your ability to perform professionally as well as in your personal responsibilities to your family and to yourself.
How does that have anything to do with story points as an estimation tool? The answer is, honesty matters. Not that developers are being intentionally deceptive, but we have a tendency to estimate the best case or what we think is the worst case. No matter how honest a developer tries to be in giving an hours estimate, there is always a tendency for the thought to lodge in the back of his mind that they have to give a number of hours that is acceptable rather than the number of hours he actually thinks the work is going to take. In addition, especially in the beginning of the project or in the estimation process, the tasks are so large that most just throw a rock in the general direction and call it good because there's no way they can get enough detail to respond with an estimate in time to be considered unless they do.
Estimating in points is an estimation of complexity... something that a developer probably has a better concept of than how many hours it will take to do something she has never done before. If you think about the estimate in hours, the normal conclusion is, "If you've done it before, this time it will take less time to finish". However, complexity does not change based on the number of times you have done something. In fact, an estimate of complexity becomes more accurate with repetition. Past experience may have an impact on velocity or it may not, it all depends on the impact of the slight changes, but those should be factored in during the estimation of complexity.
The best part about estimating in points is, historical performance CAN be a valid input. We can say that, in general, our teams finish x points per week(velocity). This allows us to develop a timeline based on average velocity that is more accurate than using past performance by feature. How can I make that assertion? Because quite simply, even similar functionality is still different. So creating a one page checkout for an e-commerce site is not necessarily going to take even a similar amount of time across different projects. Since points apply to overall complexity, not to individual features, the sample set is larger so you have more data to draw on and you can develop a better average.
The idea is to get an average baseline that is more accurate so you deliver on time and within budget or recognize that you are off your timeline earlier, allowing you to set customer expectations properly. The more you can expand your sample set, the greater your chances of achieving an accurate average. By using a point system, you have a unit of measure that stretches across many different projects with different functionality. The result is a wider cross section and more data from which to base your timeline estimates. While I do not propose that this as the magic answer to the problem of inaccurate estimates, I do believe it is a way to be more accurate. When paired with other agile principles such as brutal honesty and adaptability, you are in a better position to succeed in your project as a whole while maintaining happier customers and a happy, well balanced staff.
