Can you really plan for the long-term future?

I attended a "planning conference" last night as part of my involvement in Cub Scouts with my son. The purpose of this meeting according to the leader was to "knock out the schedule" through summer of 2008 and we had the next 4 hours to do so. Initially, as we focused on the next few months things went pretty well. However, things begin to break down quickly the further out we went. One of the primary reasons is that the next scouting year would begin in the Fall of 2007 and that will cause some changes with kids, parents and leaders in that process. Therefore, it was difficult to firm up dates because the people involved by that time will be different. Yet, we were told by the leader to just give it our best shot and go on. The other reason that became frustrating is that the leader wanted particular dates and times that we would have different events. It was easy for the group to determine particular months, but much harder to nail down days and almost impossible to determine times. Why? Because who knows their own schedules a year from now (much less the next month). In the end, we created our "project plan" of activities but it left everybody a little frustrated, very tired, and most everybody skeptical that the plan would reflect reality a year from now.

I don't place blame on the leader of this because she was doing what had been done before. It was my first time, and knowing the problems I have had in the past with long term software project planning I was perhaps a little more sensitive. But, I had to ask myself, was it really worth spending hours on something that was probably only good for a few months of accuracy? Sure, there are probably some events we need to reserve well in advance because others are asking for it. However, we really didn't know if we would do these events until we get closer. There were just too many unknowns in the equation and things that could happen between now and summer of 2008. Despite our best planning, we can't really predict what will happen to the day and time. Plus, we already know that the people will change and they may want to do some things differently later on.

That's the beauty with Agile. It puts a shorter "time box" on everything. Let's focus on the next few months with a greater precision in the short term, and intentionally leave the future as fuzzy as possible. When we absolutely have to make decisions, let's make them at the last responsible moment. All the time changing the plan to better reflect the current state of things. Then, when people change (or change their minds), you can change the plan with it. I don't know what I will be doing at 10 am on June 1st of 2008 but I can give you a better idea of what I will be doing in the next six weeks.

The Predictability Paradox

We had just several months earlier started our implementation of Scrum to all of our project teams. As we were learning what to do, more importantly we were learning things not to do. One of the things I found challenging in our adoption of Agile was that many felt that the traditional (Waterfall) way we were delivering software before was more predictable and that Scrum would be totally unpredictable given its emphasis on focusing one iteration to the next instead of heavy upfront requirements, design and planning. I knew in my heart that Waterfall really couldn't predict any better but really couldn't put my finger on why. It was then that I read this paper called "Lean Development and the Predictability Paradox" that I got my answer. It was this paper that changed my thinking and introduced the Lean concepts to apply to what I had already learned with Scrum. Here is the introduction (read more including more details on each aspect of Lean by downloading the paper here):

Wall Street has little sympathy for companies that can’t meet their forecasts every quarter. In turn, senior management expects department managers to make and meet forecasts. By the time these expectations arrive at an IT department, meeting forecasts often becomes a significant challenge. Unfortunately, it seems that a large fraction of software projects fail to deliver on their promises for one reason or another.

Why is it that software development projects seems to have so much difficulty delivering predictable outcomes? It seems that all too often projects are late or over budget or they deliver the wrong system or all of the above. How can the predictability of software development outcomes be increased? This executive report discusses a dilemma: in our zeal to improve the reliability of software development, we have institutionalized practices that decrease, rather than increase, the predictability of outcomes.

The best way to achieve predictable software development outcomes is to start early, learn constantly, commit late, and deliver fast. This may seem to cut against the grain of conventional project management practice, which is supposed to give more managed, predictable results. But predictability is a funny thing; you cannot build with confidence on a shifting foundation. The problem with conventional approaches is that they assume the foundation is firm; they have little tolerance for change.

The paradox is that trying too hard to create predictability creates opposite effect. Conventional practices are fragile in the face of change, and even in the face of learning. And yet, the more complex the system, the more necessary learning becomes. What is needed is an approach that encourages learning, and does not commit until learning is complete.

It should be obvious that decreasing the amount of speculation involved in making a decision increases predictability of the outcome. If you can make decisions based on facts rather than forecasts, you get results that are more predictable. Lean development is the art and discipline of basing commitments on facts rather than forecasts. It starts earlier, encourages change, freezes decisions later, and delivers faster than traditional practices, but nevertheless lean development produces outcomes that are more predictable.

The paradox of lean development is that you have give up some of the trappings of predictability in order to get true predictability. You have to abandon some conventional wisdom to gain the benefits of making decisions with more certainty. Fundamentally, you have to develop the capability to respond to events as they unfold, rather than hold dear the capability to orchestrate events in advance.

Are you ready for Agile?

Paul Klipp, a guest author at Agile Advice (one of my favorite blogs on Agile), poses a question to those thinking about Agile with his post entitled, "To be or not be Agile". Here's his introduction:

At it's core, an agile process is designed to address a long-standing problem with traditional development methods: scope-creep. Most traditional processes begin with a thorough description of the desired product and then code until it's done. The weakness of these approaches is that in the event of a change in the business need or a reevaluation of the plan, much work can be lost and deadlines can easily slip out of control, and costs with them. The traditional way to address this problem is with change documents. A change document basically is a way of telling the customer what it will cost to make a change to the plan after development is underway. Agile processes are designed to do away with the cost of change so that the client is free to evolve the system under construction toward the ideal end goal, even if it is a moving target.

He further gets down the heart and soul of agile with this statement:

The beauty of iterative development combined with continuous integration is that if we approach a piece of software with the intention of working until all features are done (traditional approach), then at no point in the project is there usable code except at the end. Whereas with iterative development, the client can pull the plug at any time and have a working product, even if not fully-featured. For example, halfway through a large, waterfall or RAD (Rapid Application Development) style project we might have had a working administrative interface but not working user interface, or no user authentication system; in other words, a fundamentally flawed and unusable product. Halfway through an agile Project (at the end of any iteration) we have a product which, though lacking some intended features, actually had all the essential components of a software tool finished, tested, and ready to deploy if desired. The customer is not married to the project and the developers can't hold the project hostage until the bitter end; If it concludes early, the investment is not a sunk cost.

So, are you ready to be agile or not? Read more of his post to decide.

When Agile got its legs

If you work within the software industry, you are probably familiar with Agile and many of you may be developing this way. If you aren't, you may see this as a lean approach that could be translated into other industries. Before I get into what this type of software development looks like, let's discuss what has been traditionally been the way to develop software - Waterfall.

Waterfall software development was named as such because work is performed in phases, with each phase needing to be completed before the next phase started. People will call these phases different things, but essentially they are the following:

Research
------> Plan
----------> Design
----------------> Implement
-------------------------> Test
------------------------------> Release
-------------------------------------> Review

Each of these phases would concentrate on the entire scope of the project, and people could not move to the next phase until the previous phase was completed. Because of this, an emphasis on formal documentation was needed due to the amount of scope going through each phase and the fact that team members may not have participated in the previous phase and need information to start their part.

End users and other primary stakeholder are usually involved in the first couple of phases, then would not see the end result until it was ready for release. The review is usually some kind of project post-mortem with lessons learned. Depending on the overall timeframe of the project, people that participate in this review may have been on the project months ago, and many people cannot remember what the start of the project was like.

Several years ago, people started to get together and determined that there must be a better way. They also started their own "lightweight" or "agile" process that suited their environment. One of the better known methodologies out there is XP or Extreme Programming.

When I started seriously thinking about a different way for my organization to develop, my research quickly sent me to the Agile Alliance. What is the Agile Alliance? Here is what the site says:

In the late 1990's several methodologies began to get increasing public attention. Each had a different combination of old ideas, new ideas, and transmuted old ideas. But they all emphasized close collaboration between the programmer team and business experts; face-to-face communication (as more efficient than written documentation); frequent delivery of new deployable business value; tight, self-organizing teams; and ways to craft the code and the team such that the inevitable requirements churn was not a crisis.

Early 2001 saw a workshop in Snowbird, Utah, USA, where various originators and practitioners of these methodologies met to figure out just what it was they had in common. They picked the word "agile" for an umbrella term and crafted the Manifesto for Agile Software Development, whose most important part was a statement of shared development values.

The Manifesto struck a chord, and it led to many new Agile projects being started. As with any human endeavor, some succeeded and some failed. But what was striking about the successes was how much both the business people and the technical people loved their project. This was the way they wanted software development done. Successful projects spawned enthusiasts.

The Agile Alliance exists to help more Agile projects succeed and to help the enthusiasts start more Agile projects. This particular page is to help people learn more about Agile software development. In keeping with the Agile emphasis on face-to-face communication, we urge you to visit a users group and talk to your peers about their experience.

With this in hand, I was ready to learn more about the details of what it meant to be Agile...

Early observations around projects

As my previous post may have indicated, we became focused on heavy processes around software development and project management. We tried to manage these processes around tools. We did have moderate success around projects. However, we also had some fantastic failures. Here were some early observations that I had with those projects. At the time, I didn't know how to resolve them but I was able to find some interesting patterns. Here is the list of my top 10:

1) Traditional Waterfall assumes that you can figure out as much as possible early in the process. What's ironic is that we also understood the Cone of Uncertainty, which states that the reliability of estimates will become greater as you move through the process. We learned that we couldn't plan or predict the unexpected. We could prepare somewhat through risk management but not necessarily prevent things from happening.

2) Shorter range projects (less than 6 months) were more successful in meeting deadlines than larger range projects. It seemed the larger the project the harder it was to stay on track.

3) Projects that involved internal resources outside of the Development group were more successful than projects that didn't. These resources included Sales, Marketing, Technical Support, Customer Training, etc.

4) Projects that involved some interaction with outside customers were more successful than solely internal representatives.

5) Changes were harder and more resistant to make later in the process because it meant reinvesting a lot of time going upstream in the process. The process assumed that you wouldn't have to do a lot of that. Because of this, we would deliver functionality that the customer really wanted something different with the promise that we would look at it in the future (but rarely did).

6) Because of that, customers and/or customer representatives would ask for anything they could think of whether or not they really needed it because there was no other opportunity to do so.

7) Most of the projects involved managers who felt they had to have control over the project - control over what each resources worked on, control over what the customers could have, control over what stakeholders should know. However, on those rare projects where the manager was more transparent, more accommodating, and empowered the team to determine how they do the work and what work they should do; we found that everybody was happier, more motivated and produced better results.

8) Tools to manage projects became harder to use because it assumed that the work breakdowns, dependencies, resources allocated would hardly change and that the only thing that would change is time. That was rarely the case, and project managers found themselves managing the tool instead of the project itself.

9) It was assumed that estimates were reliable from those that provided it. Therefore, all milestones were expected to be hit by the stakeholders even though how we had figured out the milestones was very uncertain. Projects that were similar to those in the past fared better than those that were different. Most projects fell into the latter category and involved either domain or technical experience that was new to at least one member of the team at any time.

10) Though we had documentation for things such as requirement specification, technical design, etc, we found that as the process moved along those didn't reflect reality. It became too time consuming for people to go back to the documents. Therefore, the working solution and code behind it were the only thing that people could count on. We had thought that documentation were be used as a reference for future projects, but instead weren't used. Also, the documentation wasn't always developed by the team but many times passed around. Therefore, there were many disconnects because of misinterpretation of the documentation. These disconnects caused many of the bugs and incorrect solutions that caused much fix and test cycles as well as go back and fix the incorrect solutions sometime after the solution has been provided to the customer through a production release.

Even though we saw these patterns, and knew that something was wrong with our processes and approach to both software development and project management, we assumed that we were using the processes or tools wrong. It took us some time to finally figure out that perhaps the processes and tools were the culprit.

We started to look around to see if others were experiencing the same thing. And, that's how we discovered Agile...