Wednesday, February 28, 2007

Innovation and Corporate Culture - Part 2

In Part 1, I promised I would get back to you on the results of a meeting I attended with the Oregon Innovator's Forum. As mentioned, the topic was "Innovation, Corporate Culture, & Getting Out of Our Own Way". Here is just some of the discussion that took place (thanks to Jon Marshall for providing the notes):

"Silos are a huge factor in this conflict of culture vs. innovation. I’m sure that silos must serve some function or they would have collapsed a long time ago. Is change happening in the appropriate place in the organization, and if not, how do we enlighten the organization or help the organization understand the need for the change to happen across the organizations?"

"Our company went from a startup to having silos. My perspective is that at some point in time, you can’t play that role of doing everything – there’s just too much going on. There are functions in the business and silos serve to organize and encapsulate these functions."

"I think there is specialization that you need to have. But what gets lost is that people lose the perspective that there is a cross functional aspect that you can’t have silos for . So we have to have a function that makes sure the right people continue to get together so that we can make the right decisions. I don’t know how you do it in a larger organization: we’re struggling with 100 employees. We have to remember to go back to not just saying IT will handle it because they’re the technology part, but that all of the employees can be a part of INNOVATING THE COMPANY."

"Have you had experience with that cross-functional piece?"

"When I came on board, the software projects were my department – we were the product development team. Yet we needed marketing involvement, we needed sales involvement, we needed technical support, we really needed every other area of the company to be a part of the product development process. What we found is maybe we did a really good job of releasing our product, but then the other pieces weren’t ready, we had some disconnects and things really weren’t working well. What we decided to do was we transformed our projects to work across the organization. That works for us because we have a cross functional team that’s not an outrageous size. This really worked because it broke down a lot of the barriers."

"So is there some generalization from that experience that you would offer to us?"

"I think innovation has to ultimately become an organizational cultural attribute. It’s not just IT or marketing, but the whole company has to be involved in creating innovation. The companies that have discovered that are successful at doing it."

"I’m interested in physics and in cosmology, the theory is the universe is expanding increasingly till it starts to rip into parts.The big rip. I think this is a great metaphor for what happens in companies as they grow. When you’re small, the average distance between any two employees is relatively small. As the company grows, if no one is paying attention, eventually you will form silos as essentially different pieces rip off of the whole. Like the model of the BIG RIP of the universe where communication ceases between the parts, the silos rip out of the whole and effective communication stops in many key areas. No one has to do anything to cause this, it is just a manifestation of growth where it becomes too much trouble to try to communicate across an increasing distance. When communication stops, the other parties essentially cease to exist for any given silo. One of the things I’ve been involved with is to repair these rips in the fabric of companies. Eg, the quality assurance department is not talking to the development department any more. They only see each other at the weekly status meeting. You always hear excuses on why that is. Too hard, too many meetings, too much work, too few people on our team.
But in reality, the communication in lining up the departments is almost THE most important thing. It doesn’t matter if your team comes in early or late if no one else knows what’s going on. Everyone else needs to be flexible to your improvements or your failures. Without that your going to fail as a team because someone in the fabric is going to fail. But if your organization remains in good communication and when one area starts to fail another group can pick up the slack, then you can survive the inevitable local failures."

"About the project team BEING the cross functional facilitating entity. Apparently companies are using project teams to be the thing that carries information across company silos and serves as an integrating force. But as you both were talking, it was occurring to me to suspect that we don’t have the right teams in a lot of cases. In other words, we’ve taken the minimum number of people we think needs to be on that team to get the project out the door, not realizing that secondary function of going across the silos; it may be that we need to significantly broaden out the teams.
One of the things we’re struggling with now is sales says “we’d like to help, but I need to get back to my job”. What we’re finding is that maybe we have to redefine what the sales position is."

These were some of the things said in the meeting around the challenges of innovating in the corporate culture.

By the end of the session, we realized that you cannot force culture to change. However, that doesn't mean that cultures can NEVER change. It just takes time. Where do you start? Changing the language. Telling stories. Getting people on board through small wins. It may take some time but before you know it the culture will change and people may not even realize it in the process. Get people to talk about innovation. Educate people on it.

By the way, if you are in the Portland area and are interested in attending the next session, please send me an email. The next session is scheduled for April 18th. I will be the moderator for this session (lucky me!). The topic will be on "Tools and Techniques that Drive Innovation". For more information, go here.

Friday, February 23, 2007

Going to a Lean Development Course - Update!

Next week, I will be attending a 2 day course in Seattle on Lean Software Development. The course is being conducted by Net Objectives and led by Alan Shalloway. I have enjoyed reading Alan's contribution to the Lean Development Yahoo Group and can't wait to learn more. Here is a synopsis of the class taken from their website (you can learn more by going here):

The software industry is looking for ways create quality code in an effective, efficient manner that is rapid, repeatable and reliable. Attempts such as CMMI and Agile methods have found some success but also seem to have some inherent drawbacks.

Of the many methods that have arisen to improve software development, Lean Software Development is emerging as one that is grounded in decades of work understanding how to make processes better. Lean thinking focuses on giving customers what they want, when and where they want it. It provides a way to maximize value while minimizing waste.

The varied background and history of Lean makes it very valuable to practitioners. However, the varied perspectives available from which to look at Lean sometimes makes it a challenge to learn how to apply it. This course takes four prominent perspectives of Lean and integrates them, creating a consistent, comprehensive view of how Lean can be applied to software development. These perspectives are:

Lean Manufacturing from Toyota (from Taiichi Ohno of Toyota)
Lean Thinking (from Womack and Jones’ work)
Lean Software Development (from Mary and Tom Poppendieck)
Lean Product Development System (from Toyota, as described by Michael Kennedy)

The course centers on how to create a fast, flexible flow of customer value-add. Principles of Lean that facilitate this are:

Eliminate Waste
Create knowledge
Respect people
Build quality in
Defer commitment
Deliver fast
Optimize the whole

The course teaches how to manifest these principles within the context of software development as product development. This is one of the distinctions of this course over mere agile training. Agile project management focuses on managing a single project and perhaps coordinating several projects together. However, true software development should be focused on the products these projects relate to. Selecting products, scheduling projects for the products, balancing product loads, are business perspectives that Lean addresses while agile methods do not. The focus of this course is on the mindset of Lean and the Lean-Agile Software Development process. The issues of architecture and how to evolve designs is only dealt with at a superficial level.

P.S. I found out about Net Objectives through subscribing to their Lean-Agile Straight Talk podcast. If you want to learn more about how Lean and Agile work together, visit their site. But just promise to come back here to learn more!

Update: If you are trying to implement Agile or Lean in your organization, you NEED this course! Though there are plenty of books on the topics, Alan did a great job of blending various experts and ideas out there and be able to understand the big picture. This isn't your typical training course however. We were able to have a lot of discussion among the group about our own experiences so far. Alan, being an Agile/Lean coach as well as trainer, provided a lot of good ideas for us to take back. It was like getting free consulting/coaching along with the cost of the course (which, by the way, at $1000 is a fantastic bargain). Check their web site for more details on the next courses as well as other courses in the area of Agile.

Wednesday, February 21, 2007

Values, Principles and Practices

When I was creating this blog, I wanted to make sure I had a tag line that accurately reflected what the focus and goals of this blog were to be. Here's what I came up with:

A manager's quest to improve his organization through Lean and Agile development values, principles and practices.

Technoetic had a post awhile back that I found to be a great representation of the overall goals - values, principles and practices. While some use these terms somewhat interchangably, I truly believed that each of them represented a particular piece of the whole. The author of this post apparently agreed with me and came up with perfect definitions of each of the parts. Here they are:

Values - A description of preference between alternatives. Often the alternatives represent candidate courses of action and the value guides the selection among the alternatives. Sometimes values are basically axioms. They can be irrational in the sense that we might not be able to explain them intellectually or they might be primarily based on emotions.

Principles - Statements describing a model. A scientific principle is a statement describing an aspect of physical reality. Principles might also describe a model of social activity, for example. Principles can be fundamental or logically derived from fundamental principles. It’s commonly believed that principles are derived from values, but sometimes people have certain values because of specific principles. For example, the “value” of communication can be derived from the principle that communication improves cooperation and coordination effectiveness. This circularity between values and principles can be confusing at times.

Practices - A pattern of activity that can be repeated to achieve goals within a context. A person defining a practice is applying principles within a framework of their values (preferences). A related collection of practices might be called a process or methodology.

To make it even simpler, I could summarize the difference between these as:

Values are the feelings of what the right things are for us to do.

Principles are the guidelines of how we are to demonstrate those values.

Practices are the actions of what we do to support those principles.

As I write future posts, I will refer to these terms frequently to describe the parts of both Lean and Agile. Before I went there, I thought it would be good for us to have the same context going forward.

Tuesday, February 20, 2007

Innovation and Corporate Culture - Part 1

This afternoon, I have been asked to join a group of 15 other innovators in Oregon to meet as part of a monthly Oregon Innovator's Forum. While I am honored to play a part, I am the first to admit that I'm not one of the top innovators in Oregon. That doesn't mean I don't have a lot to talk about in the area of innovation and my experiences around it.

I call this Part 1 because I hope to gain a lot of knowledge from the meeting to share back to this group (which of course will be Part 2). I believe that innovation can play a significant part in making Agile and especially Lean successful.

The topic for this session is called "Innovation, Corporate Culture, & Getting Out of Our Own Way" Here is a description of what will be talked about:

Have you noticed roadblocks to innovation that defy remediation?

Are you working around counterproductive business practices or systems?

Do you find that you can’t even talk about these roadblocks and awkward practices because the topics are taboo?

Then you’ve probably stumbled upon some immutable corporate cultural icons!

Please join other Oregon innovators for an informal discussion on how we can evolve the seemingly inflexible aspects of managing businesses so they better support the actual results the business hopes to achieve. The process of defining these stubbornly unyielding business practices can be difficult because your culture, like the water around a fish, is invisible. As participants in corporate culture, we do something a certain way because we think it is the only way, the right way, or at least the familiar way. These practices and systems then resist change when the business climate demands innovation.

Cultural inertia resists the changes that innovators attempt to invoke.

It's true that cultural inertia can provide certain advantages: No one wants to try to survive in a culture that is constantly changing—it would be too stressful and confusing. But here's the problem: when the “pain” associated with the current status quo becomes great enough, cultures look to their innovators for ideas on how to improve the situation. However, this hope for positive change often comes with an unconscious expectation that the “improvements” will only require minimum alterations to what we value in the cultural status quo: A “you can’t change that!” mentality.

Corporate culture thrives on the myth that issues which affect the entire company can be “thrown over a wall” to be fielded by people who will somehow handle the details by “fixing” things only on their side of the wall, with small impact anywhere else in the company. Unfortunately, that rarely works. One example where it doesn’t work is in the area of incentives for project teams. There is a growing body of evidence which indicates that providing incentives to teams produces more manageable and successful projects than providing incentives either to individuals or managers. But is your corporate culture is ready to hear that??

How do we turn this cultural Titanic around?

What are some corporate cultural icons you’ve bumped up against in the course of your travels and how did they need to change? How have you dealt with them, or how do you wish you could have dealt with them? How do we get out of our own way?

Bring your victories and war stories and let’s see if together we can shine a light on some of those change-resistant areas of corporate culture which might be blocking our paths to innovation.

I would like to hear from my readers on this topic. Please share via comments either your victories or war stories and I'll compare it to our discussion later today.

Monday, February 19, 2007

An Agile Approach to adopting Agile

Several years ago, we started our quest towards a better way to develop software. In the process I discovered Extreme Programming or better known as XP. Though I liked many of the concepts that I was learning about, I definitely had my reservations on parts of it. However, the largest reservation I had was that it seemed that the founders and proponents of XP were requiring that this methodology was an "all or nothing" type of approach. If you didn't incorporate all of XP, you really would not benefit from the results that others have had with XP. In fact, some would say that you might actually experience worse results than Waterfall with a partial implementation.

This worried me greatly because I knew that past attempts of trying to implement an entire methodology had failed. Why? Too much to take on too fast. What I found interesting is at least for XP they were taking a Waterfall approach to adopting Agile. Instead, why not take an incremental and evolutionary approach to the adoption? Start with the highest priority items, incorporate them, get feedback on their adoption, and made improvements as needed. Continue this process at a pace that your organization can handle. Ultimately deciding how much agile is enough for your organization at any point of time.

At the same time this was going on, I had an interesting discussion with a colleague. He had told me he hated the idea of methodologies. Why? Because it makes the assumption that you can go buy a tool belt full of tools and assumes that you will not only use all of the tools but will know which tool should be used for which purpose. His approach? Find the tools you need that resolves the issues you need when you need them. To better find the tools you do need to spend the time understanding what tools are available and how to use them. However, it is important to note that knowing and using are two different things. Just carry the minimal tools that you need to get the job done. His approach was much more Agile in my opinion and one based on adopting a set of Best Practices than adopting entire Methodologies.

I eventually found the answer to our problems. One of the "founders" of Agile, Alistair Cockburn, had a "methodology" called Crystal Clear. Unlike others, it really wasn't a methodology but a discussion on what practices are available from XP, Scrum, FDD, and others. He truly advocated the "cut-and-paste" approach, find something that is close to what you need and put it into your own processes. Then, make it your own! He also advocated the incremental approach, at least start with SOMETHING than it will take a life of its own. He focused more on educating the reasons behind each of the practices, and being able to sell it to others through your own adoption.

This approach allowed us to explore all other methodologies as Best Practices to find the right set of tools for our tool belt at a pace where it can be learned and truly understood. If you are struggling with Agile, chances are you are going too fast and taking on too much. Slow it down and take the Agile approach. Do only the minimum of what it takes to address the highest priority issues you are dealing with. Then tackle the next one, and the next one, and so forth.

What's funny is that the founders of each of these methodologies took the same approach. They started out with something, then realized that something else was needed until they came out with a set of "somethings" that worked for their particular situation. They found a pattern as they continue to implement in different companies, and thus they developed their particular methodology. They took the incremental and evolutionary approach. They customized it through constant feedback and changes to make it work for them - for their cultures, their people, and the technologies and products they were developing. You have to do the same thing in your journey towards adoption!

Though this approach, we have grown to appreciate what each of the methodologies bring to the table. However, we had to learn what we were missing to truly appreciate what we could get with each of the Best Practices. In other words, we had to realize that we needed a particular tool to accomplish the job much better than the tools that we currently had in our possession. Bottom line, most implementation of processes fail because of lack of understanding. Either understanding of why they are valuable or understanding of how they can be used for your particular needs. If you don't know how to use a hammer, and you don't know why you would use a hammer, you probably aren't going to find the hammer that useful. Think about that for a moment!

Friday, February 16, 2007

The Anatomy of a Manifesto

Though different forms of Agile processes have been around for many years, they lacked a common voice. There was even a debate on what to call these various methodologies - iterative? lightweight? evolutionary? agile? Many of the founders of these methodologies realized this several years ago and set out to settle differences and go forward in the same direction. Thus, the Agile Manifesto was born. According to the website, here is some of the history of that event:

On February 11-13, 2001, at The Lodge at Snowbird ski resort in the Wasatch mountains of Utah, seventeen people met to talk, ski, relax, and try to find common ground and of course, to eat. What emerged was the Agile Manifesto. Representatives from Extreme Programming, SCRUM, DSDM, Adaptive Software Development, Crystal, Feature-Driven Development, Pragmatic Programming, and others sympathetic to the need for an alternative to documentation driven, heavyweight software development processes convened.

While the Manifesto provides some specific ideas, there is a deeper theme that drives many, but not all, to be sure, members of the alliance. At the close of the two-day meeting, Bob Martin joked that he was about to make a "mushy" statement. But while tinged with humor, few disagreed with Bob's sentiments that we all felt privileged to work with a group of people who held a set of compatible values, a set of values based on trust and respect for each other and promoting organizational models based on people, collaboration, and building the types of organizational communities in which we would want to work. At the core, I believe Agile Methodologists are really about "mushy" stuff; about delivering good products to customers by operating in an environment that does more than talk about "people as our most important asset" but actually "acts" as if people were the most important, and lose the word "asset". So in the final analysis, the meteoric rise of interest in and sometimes tremendous criticism of Agile Methodologies is about the mushy stuff of values and culture.

So, let's look at each part of the Manifesto in more detail. (You can read the entire Agile Manifesto and other related information at

We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:

Just like the act of creating software is a discovery process, so is figuring out how best to develop that software in the first place. Agile recognizes that there is a constant need for improvement - not just in the software itself, but especially in how you get there. Through constant feedback from the team, and acting on that feedback, the team can own the process and make it better over time. This really goes against conventional thinking that management should control those processes to ensure that others follow them. The team owns how they do the work, the customer owns what functionality they want produced, and management owns how are we going to get there.

Individuals and interactions over processes and tools

There was a discovery that processes were becoming too heavy and that people were hiding behind those processes and the tools to manage those processes. Contrary to what some have thought about Agile, processes and tools in some form are used by all Agile teams. It's important to note that those should be in place only if they help the team produce better software more efficiently. In other words, they help you do your job better and with higher quality than without it. However, NO tool or processes should keep people from talking and working together. Project managers can't hide behind Gantt charts, they need to work with the team and talk about the progress of the project. Developers can't hide behind their tools as well, such as email, when their other team members are within distance to talk about things. The fastest and most effective form of communication is face-to-face, yet people tend to forget about it and use less effective forms.

Working software over comprehensive documentation

The goal for any software development should be to get the product out to customers as quickly as possible assuming that the level of quality meets standards. Just like tools and processes, if you are spending too much time on documentation when you could have the end product "speak for itself" than you are not meeting that goal. The code should be the documentation as much as possible. Conventional wisdom would say that customers should review and sign off of designs before product is developed, so that time is not wasted on being product the customer does not want. However, customers don't always know what they want until they see it. Paper cannot take the place of something more tangible. Once they see it, customers want to play with it, mold it and make it their own. Therefore, release often and early and provide the ability to respond to the customer using the working software product as the tool. We aren't talking throwaway prototypes here or "smoke and mirrors", we are talking software that works for the functionality that has been provided at the time.

Customer collaboration over contract negotiation

Think about past projects that you may have been involved in. How was the relationship with the customer? How much were they really involved throughout the project? Too much? Too little? Wish they were there? Wish they weren't? Most people would say that the relationships have usually been strained at best. Traditionally, most customers are involved primarily at the start and end of the project. At the start, they are asked to provide specs whether or not they understand software and how something should or could work. They are asked to provide EVERYTHING they can think of because the scope will be frozen for the remainder of the project. They are asked to SIGN IN BLOOD before the team will start development of the product. They are asked to refrain from any changes because they will cost the team especially late in the project. However, that is when most customers finally get a chance to see the real results of the project. Because many of these projects are already late, the customer "lives" with the end result most of the time (even though they know it isn't meeting their needs). Ultimately, either functionality is never or seldom used or there are future projects to redo a major portion of the developed product. And the vicious cycle continues...

Would it be better if the customer felt part of the team instead of on the other side of the table? That they felt they were being heard and were allowed to react to the product while it is being developed? Wouldn't it be the best interest of the customer to know what is going on since they are investing a lot of money into the effort? Wouldn't the team want to make sure that they are producing product that the customer will REALLY use and meets their needs? Of course!

Responding to change over following a plan

If we are to let the customer "mold" the product, we need to allow changes naturally to happen. If the customer wants something, why shouldn't we provide it to them? After all, they are buying our product! If we aren't providing something of value to them, why would we even work on those things in the first place. Traditional thinking says it is important to stay on time and within budget. Agile thinking manages the scope of the project and takes time and budget out the equation. If you are building the highest priority things for the customer, let them decide when the product is ready to be used. Why should we decide it for them?

That is, while there is value in the items on
the right, we value the items on the left more.

Critics seem to miss this last statement, and it's a VERY important one! Agile isn't saying that documentation, processes, tools, and planning is bad. If fact, you will find a little of all of these things on a typical Agile project team. However, it is recommending that a little goes a long way and not to lose site of the things that are most important to be successful (which are the items on the left). Remember to do the things on the left first, then make sure the things on the right are necessary to further support the left.

Thursday, February 15, 2007

Video introduction to Lean

The following video was presented by Mary Poppendieck on December 2006 at Google. It is called "Competing on the Basis of Speed". She presents many of the Lean concepts as it applies to software. I will dive into how I got into Lean thinking later, but thought this would be a good starting place for those unfamiliar with the concepts.

Wednesday, February 14, 2007

Are you ready for Agile?

Paul Klipp, a guest author at Agile Advice (one of my favorite blogs on Agile), poses a question to those thinking about Agile with his post entitled, "To be or not be Agile". Here's his introduction:

At it's core, an agile process is designed to address a long-standing problem with traditional development methods: scope-creep. Most traditional processes begin with a thorough description of the desired product and then code until it's done. The weakness of these approaches is that in the event of a change in the business need or a reevaluation of the plan, much work can be lost and deadlines can easily slip out of control, and costs with them. The traditional way to address this problem is with change documents. A change document basically is a way of telling the customer what it will cost to make a change to the plan after development is underway. Agile processes are designed to do away with the cost of change so that the client is free to evolve the system under construction toward the ideal end goal, even if it is a moving target.

He further gets down the heart and soul of agile with this statement:

The beauty of iterative development combined with continuous integration is that if we approach a piece of software with the intention of working until all features are done (traditional approach), then at no point in the project is there usable code except at the end. Whereas with iterative development, the client can pull the plug at any time and have a working product, even if not fully-featured. For example, halfway through a large, waterfall or RAD (Rapid Application Development) style project we might have had a working administrative interface but not working user interface, or no user authentication system; in other words, a fundamentally flawed and unusable product. Halfway through an agile Project (at the end of any iteration) we have a product which, though lacking some intended features, actually had all the essential components of a software tool finished, tested, and ready to deploy if desired. The customer is not married to the project and the developers can't hold the project hostage until the bitter end; If it concludes early, the investment is not a sunk cost.

So, are you ready to be agile or not? Read more of his post to decide.

Tuesday, February 13, 2007

When Agile got its legs

If you work within the software industry, you are probably familiar with Agile and many of you may be developing this way. If you aren't, you may see this as a lean approach that could be translated into other industries. Before I get into what this type of software development looks like, let's discuss what has been traditionally been the way to develop software - Waterfall.

Waterfall software development was named as such because work is performed in phases, with each phase needing to be completed before the next phase started. People will call these phases different things, but essentially they are the following:

------> Plan
----------> Design
----------------> Implement
-------------------------> Test
------------------------------> Release
-------------------------------------> Review

Each of these phases would concentrate on the entire scope of the project, and people could not move to the next phase until the previous phase was completed. Because of this, an emphasis on formal documentation was needed due to the amount of scope going through each phase and the fact that team members may not have participated in the previous phase and need information to start their part.

End users and other primary stakeholder are usually involved in the first couple of phases, then would not see the end result until it was ready for release. The review is usually some kind of project post-mortem with lessons learned. Depending on the overall timeframe of the project, people that participate in this review may have been on the project months ago, and many people cannot remember what the start of the project was like.

Several years ago, people started to get together and determined that there must be a better way. They also started their own "lightweight" or "agile" process that suited their environment. One of the better known methodologies out there is XP or Extreme Programming.

When I started seriously thinking about a different way for my organization to develop, my research quickly sent me to the Agile Alliance. What is the Agile Alliance? Here is what the site says:

In the late 1990's several methodologies began to get increasing public attention. Each had a different combination of old ideas, new ideas, and transmuted old ideas. But they all emphasized close collaboration between the programmer team and business experts; face-to-face communication (as more efficient than written documentation); frequent delivery of new deployable business value; tight, self-organizing teams; and ways to craft the code and the team such that the inevitable requirements churn was not a crisis.

Early 2001 saw a workshop in Snowbird, Utah, USA, where various originators and practitioners of these methodologies met to figure out just what it was they had in common. They picked the word "agile" for an umbrella term and crafted the Manifesto for Agile Software Development, whose most important part was a statement of shared development values.

The Manifesto struck a chord, and it led to many new Agile projects being started. As with any human endeavor, some succeeded and some failed. But what was striking about the successes was how much both the business people and the technical people loved their project. This was the way they wanted software development done. Successful projects spawned enthusiasts.

The Agile Alliance exists to help more Agile projects succeed and to help the enthusiasts start more Agile projects. This particular page is to help people learn more about Agile software development. In keeping with the Agile emphasis on face-to-face communication, we urge you to visit a users group and talk to your peers about their experience.

With this in hand, I was ready to learn more about the details of what it meant to be Agile...

Monday, February 12, 2007

Re-thinking what makes a project successful

In 1994, The Standish Group published a study called the CHAOS Report. This study was conducted to focus on what causes projects to not be "successful". Success being defined as on-time and on-budget. I have seen this report used by many Agile practitioners as a basis on why traditional waterfall develop projects do not work. Now that I have been using Agile practices for some time, I thought I would go back and review the report first-hand to get some insights.

I found the Introduction interesting, here is what it said:

In 1986, Alfred Spector, president of Transarc Corporation, co-authored a paper comparing bridge building to software development. The premise: Bridges are normally built on-time, on-budget, and do not fall down. On the other hand, software never comes in on-time or on-budget. In addition, it always breaks down. (Nevertheless, bridge building did not always have such a stellar record. Many bridge building projects overshot their estimates, time frames, and some even fell down.)

One of the biggest reasons bridges come in on-time, on-budget and do not fall down is because of the extreme detail of design. The design is frozen and the contractor has little flexibility in changing the specifications. However, in today's fast moving business environment, a frozen design does not accommodate changes in the business practices. Therefore a more flexible model must be used. This could be and has been used as a rationale for development failure.

But there is another difference between software failures and bridge failures, beside 3,000 years of experience. When a bridge falls down, it is investigated and a report is written on the cause of the failure. This is not so in the computer industry where failures are covered up, ignored, and/or rationalized. As a result, we keep making the same mistakes over and over again.

I find it interesting that in my many years of doing software projects, that there are always project stakeholders who view projects this way and compare these kinds of projects to either building bridges or houses. However, for the reasons mentioned above, this assumes that once completed the software will last forever (or at least a very long time). This kind of thinking doesn't handle change well. It assumes that you have a control over the future and can predict change. It assumes by restricting change you will have success. So, was that true? Here's what they found:

In the United States, we spend more than $250 billion each year on IT application development of approximately 175,000 projects. The average cost of a development project for a large company is $2,322,000; for a medium company, it is $1,331,000; and for a small company, it is $434,000. A great many of these projects will fail. Software development projects are in chaos, and we can no longer imitate the three monkeys -- hear no failures, see no failures, speak no failures.

The Standish Group research shows a staggering 31.1% of projects will be canceled before they ever get completed. Further results indicate 52.7% of projects will cost 189% of their original estimates. The cost of these failures and overruns are just the tip of the proverbial iceberg. The lost opportunity costs are not measurable, but could easily be in the trillions of dollars. One just has to look to the City of Denver to realize the extent of this problem. The failure to produce reliable software to handle baggage at the new Denver airport is costing the city $1.1 million per day.

Based on this research, The Standish Group estimates that in 1995 American companies and government agencies will spend $81 billion for canceled software projects. These same organizations will pay an additional $59 billion for software projects that will be completed, but will exceed their original time estimates. Risk is always a factor when pushing the technology envelope, but many of these projects were as mundane as a drivers license database, a new accounting package, or an order entry system.

On the success side, the average is only 16.2% for software projects that are completed on-time and on-budget. In the larger companies, the news is even worse: only 9% of their projects come in on-time and on-budget. And, even when these projects are completed, many are no more than a mere shadow of their original specification requirements. Projects completed by the largest American companies have only approximately 42% of the originally-proposed features and functions. Smaller companies do much better. A total of 78.4% of their software projects will get deployed with at least 74.2% of their original features and functions.

This data may seem disheartening, and in fact, 48% of the IT executives in our research sample feel that there are more failures currently than just five years ago. The good news is that over 50% feel there are fewer or the same number of failures today than there were five and ten years ago.

Would you call that success? Here we are 12 years later, yet we hear or have experienced projects that took longer, cost more, and didn't meet customers' needs. Those doing Agile have come up with a different definition of project success - "Continually producing increments of working software with the customer that is prioritized based on highest value for the cost (ROI) all while improving the software and how you produce it over time through constant customer and team feedback." Success if defined by meeting or exceeding customer expectations early and often, and not necessarily on the costs it will take to get there.

Read more of the report for yourself here.

Friday, February 9, 2007

Early observations around projects

As my previous post may have indicated, we became focused on heavy processes around software development and project management. We tried to manage these processes around tools. We did have moderate success around projects. However, we also had some fantastic failures. Here were some early observations that I had with those projects. At the time, I didn't know how to resolve them but I was able to find some interesting patterns. Here is the list of my top 10:

1) Traditional Waterfall assumes that you can figure out as much as possible early in the process. What's ironic is that we also understood the Cone of Uncertainty, which states that the reliability of estimates will become greater as you move through the process. We learned that we couldn't plan or predict the unexpected. We could prepare somewhat through risk management but not necessarily prevent things from happening.

2) Shorter range projects (less than 6 months) were more successful in meeting deadlines than larger range projects. It seemed the larger the project the harder it was to stay on track.

3) Projects that involved internal resources outside of the Development group were more successful than projects that didn't. These resources included Sales, Marketing, Technical Support, Customer Training, etc.

4) Projects that involved some interaction with outside customers were more successful than solely internal representatives.

5) Changes were harder and more resistant to make later in the process because it meant reinvesting a lot of time going upstream in the process. The process assumed that you wouldn't have to do a lot of that. Because of this, we would deliver functionality that the customer really wanted something different with the promise that we would look at it in the future (but rarely did).

6) Because of that, customers and/or customer representatives would ask for anything they could think of whether or not they really needed it because there was no other opportunity to do so.

7) Most of the projects involved managers who felt they had to have control over the project - control over what each resources worked on, control over what the customers could have, control over what stakeholders should know. However, on those rare projects where the manager was more transparent, more accommodating, and empowered the team to determine how they do the work and what work they should do; we found that everybody was happier, more motivated and produced better results.

8) Tools to manage projects became harder to use because it assumed that the work breakdowns, dependencies, resources allocated would hardly change and that the only thing that would change is time. That was rarely the case, and project managers found themselves managing the tool instead of the project itself.

9) It was assumed that estimates were reliable from those that provided it. Therefore, all milestones were expected to be hit by the stakeholders even though how we had figured out the milestones was very uncertain. Projects that were similar to those in the past fared better than those that were different. Most projects fell into the latter category and involved either domain or technical experience that was new to at least one member of the team at any time.

10) Though we had documentation for things such as requirement specification, technical design, etc, we found that as the process moved along those didn't reflect reality. It became too time consuming for people to go back to the documents. Therefore, the working solution and code behind it were the only thing that people could count on. We had thought that documentation were be used as a reference for future projects, but instead weren't used. Also, the documentation wasn't always developed by the team but many times passed around. Therefore, there were many disconnects because of misinterpretation of the documentation. These disconnects caused many of the bugs and incorrect solutions that caused much fix and test cycles as well as go back and fix the incorrect solutions sometime after the solution has been provided to the customer through a production release.

Even though we saw these patterns, and knew that something was wrong with our processes and approach to both software development and project management, we assumed that we were using the processes or tools wrong. It took us some time to finally figure out that perhaps the processes and tools were the culprit.

We started to look around to see if others were experiencing the same thing. And, that's how we discovered Agile...

Thursday, February 8, 2007

When processes were my passion

It's interesting what others will say when you start something new or describe who you are and what you do. I have really appreciated the support so far and thank those that have mentioned this blog. One of them (who shall remain nameless) referred to this blog being more about processes than people. Actually, as you will discover as we go along, while processes are talked about the emphasis will be more about what is around those processes. In other words, processes will not be the core of the discussion.

However, there was a time when I was a process junkie. As I have found with many managers in my industry, I came up the ranks. I started as a programmer, and my manager saw some leadership qualities in me and was looking for a manager to handle daily operations while he spent time focusing on future initiatives. One thing led to another, and I began my journey from the programming world into management.

I didn't know how to be a manager at first, so like I did with programming I looked to others for advice. I didn't go to school and learn about management. So, I looked at what others were using as "tools" in the industry. As I had done with programming, I had assumed that if I just find a framework it will get me much faster to becoming a good manager. So, I went looking for a framework.

After some searching, I found some frameworks out there that seemed to be prominent at the time (this was in the late 90s). In particular, I found SEI's Capability Maturity Model (now CMMI) for software development and PMI's Project Management Book of Knowledge (PMBOK). These were THICK books full of processes. I had thought I had died and went to heaven! I couldn't wait to take this framework and apply it right away. And I did just that.

In later posts, I will get into more details of how I think about these frameworks now. For now, let's just say it didn't go well. I quickly learned that you can't just push process on people, especially if you don't really understand the concepts, values, and principles behind those processes. I was looking for process as a short-cut as well as a way to have better control of things as a young manager establishing myself. Instead, I came off as a young, cocky manager who had no idea of what he was doing and in the process making life miserable for those around him. People who used to like me as a programmer thought that the manager job had gone to my head. And you know what? I started believing it myself.

I know there are other managers who believe that if you put heavy formal processes, procedures, policies, etc. that you will be considered a great manager and that things are so structured and disciplined that anybody can do that job. These are the managers who talk in terms of subordinates. I quickly learned that I didn't want to become that kind of manager. It really wasn't my style, and I needed to find "tools" that supported what my strengths were as a manager. While at the same time, allowing those that I manage to play to their particular strengths. Also realizing that whatever was to be in place for processes needed some flexibility for the unpredictability that the future brings and would provide an environment that would not stifle creativity and innovation.

So, am I the "process guy"? Not anymore, I'm proud to say. If you could put any label on me, I guess you could call me the "enabler guy". I figure out how teams and individuals can become better. If they are better, my job is easier. If they are better, the company is more successful. Now why wouldn't I want those things as a manager?

Wednesday, February 7, 2007

A better way for conferences

Ever heard of Open Space? If you haven't heard about it and experienced it, you will never think of a conference the same way again. I had the pleasure to attend a 2-day conference here in Portland for Agile Open Northwest.

Here's how it works, according to Wikipedia:

In Open Space, a facilitator explains the process and then participants are invited to co-create the agenda and host their own discussion groups. Discussions are held in designated areas or separate rooms known as 'breakout spaces' and participants are free to move amongst the discussion groups. Each group records the conversations in a form which can be used to distribute or broadcast the proceedings of the meeting (in hard copy, blog, podcast, video, etc). Online networking can occur both before and following the actual face-to-face meetings so discussions can continue seamlessly. In a multi-day Open Space, participants have the opportunity to announce new discussion topics / late-breaking sessions each new morning. At the end of the day (or 2 days or 2.5 days) the full group reconvenes for comments and reflection. This helps participants to re-engage in the full group over the duration of the meeting.

While the mechanics of Open Space provide a simple means to self-organize, it is the underlying principles that make it effective both for meetings and as a guidepost for individual and collective effectiveness. The Law of Two Feet -- a foot of passion and a foot of responsibility -- expresses the core idea of taking responsibility for what you love. In practical terms, the law says that if you're neither contributing nor getting value where you are, use your two feet (or available form of mobility) and go somewhere where you can. It is also a reminder to stand up for your passion. From the law, flow four principles:
* Whoever comes is the right people
* Whatever happens is the only thing that could have
* Whenever it starts is the right time
* When it's over, it's over

The organizing theme of an Open Space meeting is that people who care about the subject will come together. The initial meeting notice takes the form of an invitation, thus the people who have attended have chosen to be there and are willing to contribute. The objectives for the meeting and the time available affect design decisions such as whether action planning is included in the Open Space or not.

What did I like about it?

1) The content was tailored to the audience. Nobody selling their services or products that people may not care about. Topics were relevant to the desires of the people attending. Otherwise, nobody attended!

2) Many different perspectives were represented. I had expected that it may be more developer-centric, but there were several sessions for customers, product owners, testers and other roles. People attending also came from all of those perspectives.

3) Great respect for everybody in each session. I truly expected hidden agenda, egos, holy wars, and other things to dominant discussions. Never happened! Differences of opinions were just that - differences! No question was stupid, no perspective was invalid.

4) You could decide how much you wanted to participate. There were many that didn't say a word (or not much). Others felt very comfortable from the beginning. Some got more comfortable (like me) after some time had passed. Sometimes the experts talked. Sometimes they listened. It was very comfortable.

5) Tremendous sense of energy and community. I really can't describe it, but I never felt there was a slow time. Lots of things happening. You could feel the energy. I felt at the end a sort of sadness that the two days were ending. In most other conferences, I couldn't wait for it to end!

What could have been improved?

1) Some of the meeting rooms had more that one session in it and one of the them was also the gathering place for people between sessions. This made it very difficult to focus at times. Felt like we were on a party line competing with each other to talk. It would have been better to have one session per room and the general communal place to be located away from sessions.

2) Some of the sessions I felt that we talked problems but little solutions. Perhaps it was to be that way so that we could all go back to the real world and come up with solutions. Not sure I expected all of the problems solved but was a little frustrated because of that.

3) I didn't take any notes because all notes were to be gathered by each of the session "hosts" and placed on a common wiki for everybody to access. Unfortunately, some of the sessions never made it to the wiki or didn't have complete information. I would have filled in the blanks, but didn't take notes to focus on the conversation. This might have been just this conference as I have seen notes from other ones that seemed more complete.

If you have a chance to go to a conference or seminar that is set up this way, I highly recommend it. Don't come with too many expectations. Be open to anything happening. And hopefully my experience will encourage you to go. You will find those conferences to be the best you have attended. You will also wonder why you ever did conferences a different way. Lastly, you will probably never go to a typical conference again!

Tuesday, February 6, 2007

Before there were projects

Though project management has been around for longer than I have been born, in the mid 80's I wasn't really exposed to the idea of software projects managed by project managers. As a programmer working in both an IT shop as well as consulting, life was much different.

I was assigned tasks by a manager (functional or account managers). Sometimes these tasks were very specific, such as "add this information to an existing report". Other times these tasks were more generic, such as "customer would like a new report that shows this kind of information". The requirements were always clear, but there was always flexibilty in the design. This applied to everything from traditional maintenance tasks (bug fixing, smaller enhancements) to major new product or system development from scratch. As a junior developer, I appreciated the extra support I had as I was learning the tools and systems necessary to excel at my job. Once I got there, I appreciated the flexibility that my manager gave me to figure out the solution with the customer more on my own. In other words, the tasks moved from specific to generic as my experience increased. It became my job to get to the details.

The manager was the initial "go-to" person when there were any questions. He or she needed to have a certain level of knowledge about what the customer wanted. However, they weren't the customer so their responsibility was to get clarity from the customer when needed. Sometimes, that could be done "off-line" so that I could focus on other things. Other times, it was determined that I needed to be there if the discussion became more technical in nature.

Another key difference is that there was little or no specifications. Essentially, most requests if written were a simple one page form providing some high-level information enough to get the conversation going. Other requests were verbal in nature. There were no project plans to review, drafting and review of a large functional specifications document, etc. So where did you capture this information, you ask?

Most of the conversations involved at that point were verbal, with the occassion taking of notes to go back to my desk. However, and here is the important part, the code WAS the specification. It represented the requirements and design. It was expected that you would go through several cycles of the changes. These weren't prototypes, but early looks at the finished product. At first, some things weren't finished or even developed yet. However, the point was to get feedback not just from the manager but many times from the end customer. "Are we on the right track?" "Is this what you were expecting?" In the process, we learned technical limitations such as "we had to redesign the report because the new data would not fit" and couldn't have expected some of those things until we got into the code.

As I progressed in my career, I look back to these early years and wonder if things are better now. Sure, technologies have changed but so have other things that perhaps have gotten in the way. All I know is that things seemed much faster in those days even though the technology was slower. But I have to believe that with improvements of technologies that things should be screaming now. However, they aren't. But that's a discussion for another day...

Monday, February 5, 2007

An Introduction

Before we start this journey together, I feel an introduction is in order. Let me tell you a little about myself.

I have been in the IT or software industry for over 20 years. I started in programming and eventually moved into management, having been a CTO for the last six years in my current position. I have had experience with various technologies over the years - mainframe, client/server, networking, hardware, wireless, web, open source. I also have worked in several different capacities - in-house IT, in-house development, staff augmentation and outsourcing. I have worked in many different industries - retail, health care, financial, publishing, and automotive aftermarket. That's my credentials in a nutshell.

I am also not new to blogging. I started another blog, Random Thoughts from a CTO, in February 2005. That blog focused mostly on management and leadership topics and was moderately successful. Last October, I decided that I had said what I wanted on those topics and took a break from blogging. I wasn't sure that I was going to get back to it. But here I am!

This blog will focus more on the work itself than how to manage the work and people. Over the last few years, I have gained a lot of knowledge with Agile and experimented/implemented it in my current organization. In the last year, I have also gained exposure to Lean Product Development starting with the Toyota Production System trying to determine how the principles and practices could be applied to software development. I have learned a lot. I want to share my journey with you to this point. I want to share my quest towards an agile and lean organization. I believe that I will expand my knowledge through this blog and your feedback and support. I will not call myself an expert, but will share what I know with those not familiar with either Agile or Lean. I am not here to sell products or services, but to talk about the realities of life and to work in it. I am not here to talk theory, but will try to provide application to those theories.

Though the focus will be on how to apply to software development or IT, since that is my experience, I believe that much of what is discussed can be applied to other industries. I also believe that there is application to your personal life in addition to your professional life. "How you are" in addition to "what you do".

I look forward to the experience with each of you!