The top five reasons software projects fail (and how Extreme Programming mitigates them), Solutions!, Online Exclusives, August 2004

Most large-scale software projects fail. Failures range from the worst-case scenario of projects that are not completed, to projects that are over budget or that fall short of expected goals and objectives. The larger the project, the more likely it is to go belly up, and this often leads to a costly washout for an organization. However, new software development methodologies have been created that address this global problem.

In this article, we’ll discuss how Extreme Programming (XP) can now be used to effectively solve many of the major issues that lead to project complications. XP is a software development methodology that evolved from processes created by developers of the object-oriented programming language Smalltalk in the mid-1980s. With the publication of Extreme Programming Explained, written by Kent Beck in 1999, this development methodology gained attention in the software development world, although it is still not widely used within the conservative corporate IT community.

Even though each less-than-successful project has its own unique causes, we’ve pinpointed five of the primary factors that underlie most problems, and how XP helps address each of these issues:

1. Poor communication
2. Incomplete requirements and specifications
3. Scope issues
4. Inadequate testing
5. Integration

Problem One: Poor communication
A main reason projects fail is unclear, inadequate or inconsistent communication between stakeholders. Standard development methods do not include methodologies that support the ongoing and meaningful sharing of information between all project participants. Business users, technical management, outsourced services providers, and individual programmers are often isolated from each other.

The XP solution: In traditional development projects, the customer’s only close involvement in the project is during the requirements and deliverables phases. Such "waterfall" projects begin with the creation of a requirements document that serves as a perpetual and mostly static roadmap. The project moves forward in sequential waves, from design through coding and testing. These waves are often the work of different groups of developers. Within waves, most work is done by individuals working in relative isolation; they post their work to a project manager who then handles the exchange of information between project members.

XP views communication and collaboration as essential keys to the software development process. The customer is included on the project team and is an important part of the ongoing development cycle. By creating "user stories" that describe specific features the system should support, and by evaluating and approving each component as it is developed, the customer becomes an integral and driving force through the entire life of the project. Programmers benefit from direct access to the customer, and are able to create better systems because they understand why the features are important and exactly how they should be implemented. On the development side, programmers work in pairs as a close-knit team during the programming process. Through brief and relevant daily and weekly meetings, a constant flow of communication is ensured.

Problem Two: Incomplete requirements and specifications
Traditional waterfall development begins with the creation of a comprehensive requirements document. This often massive guideline is designed to provide the blueprint for the entire project. The problem is that, in every project that lasts more than a couple of weeks, specifications will unavoidably change over time. Business requirements change. New aspects of the project come to light. What seemed workable on paper proves untenable in implementation. Despite the almost universal need for changes in requirements and specifications over the course of a project, traditional methodologies do a very poor job in systematically integrating such changes.

The XP solution: XP solves this problem by taking an iterative approach to development. Instead of sequential development, XP works through a project by implementing the most important features first, and by breaking the features down into useful functions that can be created within a week or two. Each component is designed to meet a specific individual requirement, based on user stories created by the customer immediately before the development iteration begins. If requirements change down the road, it is a simple manner to change whichever individual components are impacted, provided that the practices of continuous re-factoring and elimination of code duplication are followed. Therefore, the development of each feature is as close as possible to the creation of the requirements of that feature by the customer, as opposed to traditional development, where work on specific functionality may not commence for many months after the requirements document has been created.

Problem Three: Scope issues
One of the most common areas of contention in software projects is "scope creep," which causes discrepancies in both cost and delivery time. This is not surprising—you would need a crystal ball to anticipate the changes that may be needed in the life of a large-scale software development project whose entire scope can range in months or years. It has been stated that when estimating large projects, you can either be lucky or wrong. Unfortunately, most estimates aren’t very lucky.

The XP solution: Rather than solve the "problem" of constantly changing scope, XP embraces it. XP was designed to be used in an environment where requirements are likely to change. By taking advantage of a customer-centric iterative model, XP keeps commitments focused on short-term deliverables that can be accomplished on time and on budget in most cases. Instead of a long marathon, development is divided into a series of shorter sprints, or iterations. Each iteration is focused on a mutually agreed-upon number of user stories, which are created, tested, and approved by the customer before moving on to the next. This system makes it virtually impossible for projects to go very far off-track, since the customer and development team continuously evaluate the work and make sure that it is consistent with the needs of the customer as they evolve over time. Scope is strictly frozen, but only for very short periods of time. Customers are welcome to ask for a feature in one iteration, and then completely change their minds in the next; as long as the story for the current iteration remains the same. This is generally acceptable to even the most demanding customers, since they know that they have complete control over what gets developed when.

Problem Four: Inadequate testing
Traditional development methods save testing to the final phase of a project. This is very problematic because issues with core architecture may not be exposed until a project is almost completed. At that time, correcting such fundamental problems can be very costly or may even require an almost total revamping of the code base. Another factor that adds to testing issues in traditional development is that testing is often under funded and constrained by time. By the end of the project there is usually tremendous time pressure to deliver the completed software, so the first roll-out of the new software is often an unofficial beta test.

The XP solution: XP provides an elegant solution by turning the tables and creating a test for each component as the first step of each programming task. Creating the test before undertaking the coding insures that the developer understands exactly what each component should accomplish and provides a verifiable benchmark to insure that they meet the customers need. Problems are found quickly and can be remedied at relatively low cost.

Problem Five: Integration
Large projects using traditional development methodologies often fall apart during the integration process. It is often the case that different teams of programmers, who may be geographically dispersed, work in relative isolation on individual parts of the project for long stretches of time. As they finish their work, they typically turn over the code to the project manager who then assigns it to a new team that attempts to integrate the disparate components and make them all work together. Since there had been no consistent communication between the teams during the development process, there are often significant problems at this stage as different interpretations of the requirements or their implementation becomes apparent.

The XP Solution: XP mitigates this issue in a number of ways. First, ongoing meetings and a collaborative ethic insure that the entire team has an understanding of the entire project. Instead of working on their own task in isolation, each participant is aware of and involved in the discussions about the entire project. This makes it much less likely that bad assumptions about other project components will result in compatibility issues. XP also utilizes a "round robin" approach to integration. Every project team integrates its own code into the project. As teams check the code in and out, there is a continuous process of integration and ongoing problem solving. Further, since iterations are only a week or two in length, integration happens by necessity at the end of each iteration, which greatly decreases the complexity and risk of the process. This system not only lessens the likelihood of integration problems, but also makes solving any problems that arise less problematic.

Conclusion
We’ve seen first-hand how traditional development methods can create severe problems that can ripple through an organization. One of our first engagements was bailing out a very large-scaled project that a “big five” services company couldn’t deliver to a leading financial services client. The intention was to recreate the client’s legacy systems so that they could extend to partners and customers online.

When the client finally called us, not only was the project millions of dollars over budget and many months behind schedule, but the provider just couldn’t get it to work after billing more than $40 million. We ended up scrapping their entire code base and providing a solution through integrating their existing systems, rather than trying to recreate the wheel. This provided a solution in less than six months at a cost of under half a million dollars.

We have found the XP methodology relatively easy to implement internally. Although clients must be educated initially, we’ve found that it greatly increases customer satisfaction and provides them with software that closely meets their needs, even if those needs change during the course of a project.

ABOUT THE AUTHORS:
Nate McKie is the CTO of Asynchrony Solutions, a leading IT services firm based in St. Louis, Missouri. He has more than 15 years of industry experience in corporate development and project management. He has worked on significant global integration projects and has experience in a broad range of financial services companies, including finance companies, insurance, mortgage banking, and more. David Elfanbaum is Vice President of Asynchrony Solutions. Contact Asynchrony Solutions by phone at 314.678.2200, or at their website: info@asolutions.com.