Leaning Towards Agility

Moving On

2007-07-05T10:24:00.000-07:00

To my readers,

When starting this blog back in February, I had no idea where life would take me. In late February, I attended a 2-day class on Lean Software Development. While I had read the Poppendieck books and had made the connection between Lean and Agile, I was eager to get more perspective.

I attended the course at Net Objectives headquarters in Bellevue, Washington (a short 3 hour drive away). Alan Shalloway, the CEO, conducted the course. I had found out about Net Objectives through their podcast called Lean Agile Straight Talk. I already knew the podcasts were very good, but the class exceeded all expectations. Not only was Alan very inspiring and knowledgeable on the topic, but I felt right at home with the material and soaking in every bit of knowledge from Alan. While there, Alan and the rest of the staff that I got to meet left a big impression with me.

Several weeks later, I got an email from Net Objectives wanting to do a blogging project with me. They had been impressed with Leaning towards Agility and wanted to collaborate together on something. It was at that time that I expressed my interest in seeing if there was a longer term engagement available. While there wasn't anything available at the beginning, we kept the conversation going while building a relationship.

I am happy to announce that I will be joining Net Objectives in mid August, as one of their Senior Consultants. Initially, I will be an Agile Mentor/Coach for their clients and then hope to expand into other areas of the organization. As part of that, I am retiring this blog and you will see me soon on the Net Objectives Thoughts blog.

I know it was a short time, but I appreciate the support I received by you. I am not going away by any means and you can expect me to be even a greater force in the Lean and Agile community in the months to come. Thanks!

List of Lean or Agile Community Groups

2007-06-25T14:28:00.000-07:00

Here is a list of both Google and Yahoo groups that have some focus on Lean and/or Agile Development. You will need to sign up with Google and Yahoo to join these groups. Most of these groups are very active and you will find it's a good resource for both newbies as well as seasoned agilists. Enjoy!

Google Groups:
Agile Tangents (http://groups.google.com/group/agile-tangents) - Good overall discussions around all things Agile

Yahoo Groups:
Agile Project Management (http://finance.groups.yahoo.com/group/agileprojectmanagement/) - Focus on the management side of Agile projects

Test Driven Development (http://tech.groups.yahoo.com/group/testdrivendevelopment/) - Everything you want to understand about this practice also known as TDD

Scrum Development (http://groups.yahoo.com/group/scrumdevelopment/) - Emphasis on just Scrum brought to you by the Scrum Alliance

Lean Development (http://tech.groups.yahoo.com/group/leandevelopment/) - Focus on the content of the Lean Development books by the Poppendieck's

LeanAgileScrum (http://tech.groups.yahoo.com/group/leanagilescrum/) - A newer group started by Alan Shalloway from Net Objectives. Brings together Lean, Agile and Scrum discussions in one place and talks about how they work together.

If there are other community groups, please let me know by responding via comments. They can be other Google or Yahoo groups or other types of communities that focus on Agile or Lean.

Good document on Scrum

2007-06-14T09:44:00.000-07:00

Graeme Matthew has provided an in-depth look at Scrum. You can get the document here.

Overcoming Fears with Collaboration

2007-06-13T08:24:00.000-07:00

Since Agile is so team-centric, collaboration between team members and customers is essential for success. However, for some people collaboration isn't an easy thing. There may be some fears that need to be overcome to fully see the benefits of collaboration.

Fears fall into these categories:

1) External Fears - The organization may have fears of being too transparent to their customers, competitors or other outside interests. Sharing too much information could be considered giving away company secrets to your competitor. Having your customers talk with other may uncover issues they have with your organization. Your customer may have information you might not want to hear (but should).

2) Organizational Fears - Executive management is supposed to drive the long term vision and strategy of the company. Operational teams are supposed to implement this strategy. But what if the two groups should clash? What if the operational teams come up with ideas that will cause a need to change the direction of the company? What if strategy is on a "need to know" basis and executive management doesn't give enough information to operational teams? What if management doesn't trust operational teams? What if operational teams don't trust management?

3) Individual Fears - Perhaps the individual hasn't had the education and skills developed that encourage collaboration. Most people grew up going to school where the teacher talks and everybody listens. The teacher gives you the homework and you follow the instruction. No coloring outside the lines. No challenging ideas. Speak when spoken to. Then, that person starts their career where the manager talks and you listen. The manager gives you tasks and you complete them. Same structure, but highly different than a collaborative environment.

So, how do you combat these fears?

1) External Fears - Managers worry about transparency because they think that outsiders don't know what is going on in the company, but in this Information Age the opposite is true. Customers will talk with other customers. Competitors will find out what you are doing. The competitive advantage is no longer how you hold on to information, but how quickly and better you can implement those ideas. Once you understand this, you will be more open to knowing what is out there. The more transparent the organization can be in an collaborative environment, the more accurate you will meet the needs of those "outsiders". The competitor who is most open and both listens and responds to what is being said, the better they will be.

2) Organizational Fears - Collaborative environments have one thing in common - an environment of high trust and respect for each individual. Strategies need to be shared. Everybody may have knowledge on any topic so every topic should be shared. People need to understand the vision. Most of all, everybody in the organization should help influence where the company needs to go. It's not just Executive management's role, but everybody who has a stake in the company.

3) Individual Fears - New people have the most opportunity for collaboration because they come with a fresh and naive perspective. No questions are "dumb" ones. Organizations need to encourage new employees to contribute in any way possible as soon as possible. There needs to be retraining and rethinking about how individuals should contribute. Some people are more comfortable in a verbal discussions. Others need time to digest information then respond. Yet others feel more comfortable in "tweaking" existing ideas instead of coming up with new ideas. Others like to respond in writing. You need to provide an environment that plays to each person's strengths and allows them to participate.

Pipelining is not Lean

2007-06-12T08:31:00.000-07:00

LeadingAnswers has a thought-provoking post today called "The Pipelining Anti-Pattern". Here's how Mike describes it:

If you have analysts working ahead of development, or have testers working significantly behind development, then you may have “Pipelining” problems. Pipelining is the term used to describe the situation when business analysts are working ahead on the requirements for a future iteration; the development team is working on the current iteration, and the test team is engaged on a previous iteration. In some circumstances analysis may be several iterations ahead and testing several iterations behind. To some people this may seem an efficient use of resources with each group running at their optimal speed, unfettered by the co-ordination constraints of different groups. However from an agile and lean perspective this is problem, a bad-smell that needs fixing.

Here are the problems with pipelining:

Three teams not one – in a project where pipelining is occurring we do not have one cohesive team we have three teams (or more). It is hard enough co-ordinating the members of one team towards a common goal aligned to business benefits. When there are three teams it is just too easy for people to claim that they did their bit and problems lie with other groups. Yet, the fact remains that if the software does not meet business satisfaction then it is everyone’s problem.

Increased Work In Progress (WIP) – Requirements whether they are in the form of user stories, use cases, or formal specifications all represent work invested that has not delivered value to the business. The same goes for code, until this functionality has been tested to the satisfaction of the business it is not valuable. As the time increases between capturing the requirement and finishing the last test two problems occur. The first is classic accounting, money have been invested for no return yet and there is a risk associated with future returns. The second is that requirements decay; the longer we keep requirements around for, the higher the likelihood that they will no longer be required or will have to change.

Increased time from defect injection to defect remediation – the cost of change increases the longer a defect goes undetected. In a pipelining project, defects introduced by faulty analysis could take months to be detected in testing or user review. Fixing the problem after this period of time will entail refactoring far more code (for the work happening in the interim) than if it was detected earlier, and will increase technical debt...

Time fragmentation costs when groups collaborate – on pipelining projects the test group could be working on iteration 2, the developers on iteration 4, and the analysts on iteration 5 or 6. If a developer has a question for an analyst or a tester for a developer then task switching costs will be incurred. Task switching is the time required to respond to an off-topic interruption. We must mentally “park” our current work, try to think back to whatever it is this person who has interrupted us is asking about, recall the details, answer their questions and then resume where we left off. The problem with this is that studies have shown people, while great at many things, are generally very poor at task switching. Our recall of past details is poor, defense mechanisms in our brain designed to dull unpleasant experiences erode memories of nasty problems, and we are terrible at remembering where we parked our current thoughts. The net result is that just a few requests to recall past events really mess up our performance and yield low quality answers to the enquirer.

Throughput optimization – three or more teams running through work at their own speeds with fewer dependencies on others may seem like it is working, but from a systems perspective it is a problem. Processes running at different speeds require work buffers to avoid running out of work and they build piles of unfinished work. This inventory is waste in the system and leads to scrap and more rework as processes change and partially completed work needs to be updated.

I really think Mike is on to something. I have seen this struggle in our own product teams. Read more of the post to understand this problem as well as see how Mike addresses it.

A Simple Look of Lean and Agile

2007-06-11T12:53:00.000-07:00

Somebody asked me if I were to talk about the relationship between Lean, Scrum and XP what would that be in the "simplest way possible" (At our organization, we are implementing all three). Here's how I answered:

Lean introduces the principles, or the "why's" of producing software. Focus is on maximizing customer value while minimizing waste to provide the most optimal ROI.

Scrum takes those principles and introduces the management practices, or the "when's" of producing software. Focus is on incremental delivery of working solutions that improve over time.

XP takes Lean principles and Scrum practices and introduces the technical skills, or the "how's" of producing software. Focus is on constant improvement of skills (and the underlying infrastructure) through analysis, design, and implementation of producing software.

It is my strong feeling that you really need all three to maximize your effectiveness in the development of software. Individually, each part can be implemented on it's own but you will discover something missing. Together, these really cover all aspects of developing a successful software solution.

The Exposure of Lean and Agile

2007-06-11T09:32:00.000-07:00

A few years ago, I attended a local conference on Software Quality. We were just starting to roll out our own Agile implementation, so I was eager to hear some discussion from others at the conference. In the keynote, there was an introduction to Agile. The speaker polled the audience to see what they knew about it. I would guess that probably 75% hadn't even heard of Agle, and out of the other 25% only a few were actively using Agile practices. There was absolutely no discussion around Lean development as well. I was sorely disappointed that there wasn't more of a buzz, as I could see great potential in adopting Agile. This was 2004.

Last week, I attended a one-day seminar on Collaboration. This was by no means a technical conference, in fact there were many people who weren't even in the software industry. For those that did develop software, they were more of the project manager or program manager types. I surely didn't expect much to be talked about Lean or Agile. I was VERY surprised. It seemed that most people were actively using some Agile practices, and those that were not were eagerly listening to others. It seemed most talked had something around Agile or Lean, or the discussions following the talks would discuss either one. There were a few who had already made the connection between Lean and Agile. Most surprising of all, there was much discussion with the people outside of the software industry about how Lean and Agile could be adapted to work outside of software development and be used for other types of projects! Now, in 2007, people are seeing that potential for their own.

Talk about a transformation in such little time! It's an exciting time to be involved with software development.

Is something missing in the Agile Manifesto?

2007-06-04T09:53:00.000-07:00

Brian Marick believes so. His post, "Six years later: What the Agile Manifesto left out" explores some missing values that aren't readily apparent by reading the Manifesto but are essential to have to be successful in Agile. Here's a summary of it (I've made bold the values that he believes is missing):

Now Agile is more respectable, a safer choice. The challenge isn’t so much getting a chance at Agile as it is executing once you’ve gotten the chance. There, the values of the Manifesto are not so helpful. Except for “individuals and interactions”, they face outward from the team. They don’t strongly apply to the relationships between team members, between the teams and their environments, and between the teams and their code. That’s a problem, because it seems to me that many new Agile teams are not executing. They’re floundering.

What should teams do with the time they’re not spending going too fast? They should invest in one of the four values I want to talk about: skill. Two skills that apply to pretty much any software project are refactoring and programmer testing (test-driven design). Those are skills that require a great deal of discipline. It’s awfully tempting not to write a test when it’s harder than writing the code, or to refactor some icky code you stumble on when that would mean blowing your estimate. Some of XP’s practices help with discipline. Pair programming turns what could be a solitary vice into a social act: you and your pair have to look at each other and acknowledge that you’re about to cheat. Peer pressure comes into play, as it does because of collective code ownership. Someone will notice the missing tests someday, and they might know it was your fault. Maybe the key driver of discipline is the practice of creating working software—running, tested features—at frequent intervals. If you’re doing that, you just don’t have time for sloppiness. You have to execute well.

I’ve observed that a characteristic of a good code base is that it’s habitable, that changes are comfortable, that the process of programming is one of ease. Agile teams suffer because they don’t think that wanting to make their lives easy is a relevant value, but it should be. There’s absolutely nothing wrong with finishing up a story by spending a half hour or so tinkering with code you’ve touched, just improving it—doing the software equivalent of hanging power within easy reach because the power strip on the desk is always annoying you.

I think Agile is suffering today because these fundamental values didn’t get written down and are too easily forgotten. As Agile moves into bigger companies and into less adventurous ones, those undocumented values are getting watered down. If that continues, I’m afraid Agile will be this decade’s fad that changes nothing. That would be sad.

Can CMMI and Agile play well together?

2007-06-01T15:56:00.000-07:00

Brad Appleton on his ACME Blog has a very extensive list of links that say "YES". As a follower of CMMI in the past, this will be interesting reading to me to see what others are thinking.

Metrics in the Lean and Agile World

2007-05-31T10:10:00.000-07:00

I ran across an interesting list of metrics from this site. The author of the post had gathered these in a brainstorming sessions with his organization on defining potential metrics on how they were doing with Agile. The author also poses a question to his readers on what metrics could be used by the team to possible deceive management. It's an interesting read. Here's the list of metrics that they came up with:

Short Iterations - Planning Game, Variable Scope:
* Relative velocity (measures accuracy of estimation and amount of developer time spent on other activities)
* Number of stories planned for the delivery vs number delivered
* Overflow of stories (measures accuracy of estimation)
* Degree of story change (measures activity in the business area and stability of business process)
* Number of problems fixed from previous iteration vs number of new stories implemented

Continous Integration:
* Time to build
* Number of successful/failed builds

Available Customer:
* Hours spent with developer/customer per week
* Speed of feedback vs resolution
* Number of raised features vs number of implemented features

Refactoring:
* Lines refactored
* Time spent refactoring vs new code
* Most frequently changed files
* Number of files changed per feature

PM Tools:
* Effort/time spent per feature
* Actual effort vs expected effort

Frequent delivery into Production:
* Amount of defects not caught by test/review

Regular demos for feedback:
* Degree of incorporation of feedback into iterations (measures accuracy of story gathering and accuracy of implementation)

Lessons learned for each iteration:
* Traceability of lesson as applied to future iterations/practices

Test-Driven Development:
* Degree of test case coverage
* Test case failure rates

Many managers would look at this list and think it's pretty good. In fact, they probably would think of some other measures to add to the list. However, what is the return for that investment (ROI)? Some of these metrics might be easy to gather, others could require more tools or manual effort to get. Even if you do get the information, how are you going to use the information? If you don't know how to interpret the data, you may find yourself managing the wrong things.

For these reasons, I have always struggled with finding useful metrics that are easy to gather and are promoting positive behavior. Metrics seem to be used more to punish bad behavior than to encourage better behavior. Lean software development advocates are pushing to measure up. What does that mean? Metrics are typically managed at too low a level in the organization. Therefore, companies come up with dozens of metrics (similar to a list like above) and ultimately get consumed with too many things to keep track of. Therefore, why not find metrics that can give you just enough information that measures things in terms of customers and deliveries? What if these metrics would encourage position behavior while helping you know where to look in more detail for inefficiences?

Following are the measures that I am considering in our adoption of Lean and Agile. Note, I haven't tried these out yet as the focus right now has been more on getting the practices and principles down before looking at efficiencies. However, based on my reading and understanding of both Agile and Lean here is my list:

Story Cycle Time (Measuring One Piece Flow): Average time from Customer Request to Customer Implementation for each Story. Encourages smaller pieces (Stories) that bring value. Encourages finding ways throughout organization to deliver those pieces faster.

Iteration Efficiency (Measuring Value vs. Waste): Percentage of Time spend on Value (Customer Features) vs. Waste (Defects, Refactoring, anything else the customer doesn't believe provide value). Encourages those activities that the customer sees value. Discourages those activities that the customer sees as waste.

Iteration Return on Investment (ROI) (Measuring Value vs. Cost): Ratio of Total Value (Customer and Business Value) against what it took to implement stories. Encourages the prioritization of things based on not just value but against the cost of implementation. Confirms to the team that they are working on the most important things.

Story Acceptance Measure (Measuring end user acceptance): Average acceptance (user rating) of each Story after it has been implemented. Encourages making sure the customer needs were really met. Encourages quality of solution from a customer standpoint. This may be the most difficult to gather because it requires polling at least a good sample of customers.

What do you think? Did I miss any measures at the organizational level?

Team Power

2007-05-30T13:47:00.000-07:00

In both Agile and Lean circles, there is constant emphasis on the team rather than individual efforts in resolving issues, making decisions and suggestions for improvement. "Two heads are better than one" rings true! Over at LeadingAnswers blog, the latest post called "Team Solving: The Under Utilized Power" demonstrates this fact. Here is a synopsis of the findings:

Power #1: By asking the team for a solution we inherit consensus for the proposal. The fact that the solution came from the team and not me, meant I did not have to sell it, it already had support. It is easier to steward a sub-optimal solution with good support to a successful outcome than build support for an optimal solution and ensure its successful execution. If challenges arise and people subconsciously ask “who’s bright idea was this?” the answer comes “Oh yeah, it was ours” and they are more likely to continue.

Power #2: Accessing a broader knowledge of the facts. Team members are closer to the details and bring additional insights other than your own. I did not know that the business folks bowled, this was collective knowledge (Chris knew three users bowled and Julia knew two others did also) together we found a new piece of useful information. Asking the group taps the collective knowledge of the team.

Power #3: Solutions are practical. Anyone who has worked hard to craft a solution only to be told “That will not work here because...” will know how frustrating and disheartening these words are. Team sourced solutions have been vetted for practicality and, because created internally, solutions found for implementation issues.

Power #4: When consulted people work hard to generate good ideas. The simple act of asking for suggestions engages team members beyond the role of “Coder” or “Tester”. People appreciate having their inputs valued and generally work hard to create innovative and effective solutions. Treating workers as interchangeable resources is a poor model inherited from the command-and-control methods of the industrial revolution. Leading companies such as Toyota and 3M recognize that their best ideas come from inside their companies and we need to make use of this intellect. It is partly due to these methods that these companies innovate better, have higher quality products, and better labor relations.

Power #5: Asking for help shows confidence not weakness. Asking for ideas and solutions to problems is not a sign of incompetence or an inability to manage. Just because we ask for input it does not follow we are dumb. Instead it demonstrates valuing the opinion of others, and being thoughtful. In essence it demonstrates how all problems should be tackled, which is the next power.

Power #6: Seeking ideas models desired behavior. Project managers have a leadership role of “modeling the desired behavior” i.e. behave as we wish others to behave. If we stay silent, make decisions with incomplete awareness of the facts, and do not ask for help when we need it, what message is this sending to the team? Well, it is an obvious message that we expect team members to behave the same way and work in isolation. Lots of time and money is wasted on team building activities that are then eroded my management-in-a-vacuum.

Here are also some cautions where the team can actually lose power:

Caution #1: Real problems - We should use this on real problems only, not on which brand of printer toner to buy. Remember we are modeling desired behavior and if people take it too far and consult their team mates when deciding what color dialog box to create; no work will get done. It is a tool for when you are stuck and the problem is important.

Caution #2: Poor team cohesion – If the team is fragmented and has opposing groups then resentment for “fixing their problems” will undermine the process. We need to get the team aligned for team solving to be most effective.

Caution #3: Team and Project Changes – Over a long period if a significant portion of the team changes, we need to re-canvas the team to ensure they are still on-board with the approach. Exercising the bright ideas of others is nearly as bad as not being consulted; we need to ensure people still agree this is a good policy. Likewise if the project changes significantly, we need to checkpoint in light of these new facts and get the team to review the approach.

Caution #4: Follow-through – once you ask for solutions make sure you follow through on them with execution. It is pretty demoralizing to be asked to work on a solution and then see it wither. It is fine to go back with implementation problems that need to be solved, but don’t waste peoples time by asking for input and then ignoring it.

Exploring what makes Scrum successful

2007-05-25T10:11:00.000-07:00

Alan Shalloway from the Net Objectives Thoughts blog is digging a little deeper into the "soul" of Scrum to truly understand why Scrum works in his post "Challenging why (not if) Scrum works". Please read his post in more detail, but here is some reasons he came up with:

I have repeatedly heard that “Scrum succeeds largely because the people doing the work define how to do the work” (see From The Top by Ken Schwaber for his full text). However, I do not think that is even a third of why Scrum works - and be clear, I am certainly not questioning that Scrum works - I just want to get at the why it works.

OK, so why does Scrum work? Remember, I am not challenging that it does, just asking why. I would say Scrum works for the following reasons:

* the iterative nature of Scrum development (including the re-assessment of priorities between iterations)
* the workcell nature of the Scrum team (everyone who needs to work together is together)
* the fact that many Scrum teams are given the chance to co-locate
* that most Scrum teams work on one project at a time
* that many old, cumbersome checkpoints/forms/status reports that used to be required are abandoned because the team is doing something new
* a focus on defining acceptance tests concurrent with requirements
* the availability of a product champion (product owner to Scrum enthusiasts)

I think this is a really good list, I would also add a couple of items to it:

* frequent communication of status and removing of impediments through the Daily Standups
* continue emphasis on feedback and improvement through sprint reviews
* the team nature of the Scrum team (everybody on the team should understand and commit to the sprint through sprint planning)

For those that are using Scrum, you are well familiar with these benefits. For those that have not moved to Agile or they are having problems with their practices, consider looking at Scrum to gain these benefits. Scrum by no means is a "silver bullet", but it has already proven to be very successful where it has been implemented.

Balancing Perfection against Innovation

2007-05-24T10:15:00.000-07:00

Jeffrey Phillips from Thinking Faster is struggling with finding the balance between the desire to have perfection with products against the desire (sometimes by the same people) to provide innovative products. His post, "Trading off perfection and innovation" shows us a comparison of the two. Here's just part of his thinking on this (read more of his post to get everything):

One one hand, we have the constant demand for perfection. Six Sigma, Lean, eliminating mistakes and errors to create an incredible, consistent, useful product or service. On the other hand, the need to innovate, to create something new and take new risks to dramatically change a product or market. Both approaches are correct, and relatively mutually exclusive. What does a manager do?

When choosing perfection, we make a series of assumptions about the product or service, its competitors and our ability to keep up with the market. Once a product or service is perfected - it it every really is - even a small change to the process or product can cause major disruptions for a customer, so we prolong the life of any product or process and minimize changes to achieve perfection. If you can dominate your market, or if perfection is important to your customer, then this is a great strategy. However, it assumes that there will be little or no disruption in the market, and that the customer demands perfection over innovation and new capabilities.

When you choose innovation, on the other hand, you will not be able to provide a perfect product or service. If your innovations are truly new, you can't determine in advance exactly what "perfection" is - you can only try to provide what you think customers will value and iterate from that point once you discover and uncover real wants and needs. This approach assumes that customers don't demand perfection, but do demand interesting, innovative products and services that solve unspoken or unmet needs. After all, if you can solve a problem that I have even fairly well that has never been solved before, I won't ask for perfection yet.

The real crux of the matter is that if your position is perfection, then you can't afford to make mistakes, and if you do, you need to quickly and completely correct them. On the other hand (aren't you glad we only have two hands) if your position is innovative, you can't afford not to make mistakes, and you want your customers to participate in and help you learn from your mistakes - you can't learn if you don't make mistakes.

In the end, you have to give up some level of perfection if you want something innovative. Because the very definition of innovative is doing something that hasn't been done before (or at least not that often). You also have to be able to take on risks and make mistakes as part of the experimentation that comes with innovation. I have to agree with him. Too many times, I have seen people take the "tried and true" approach to a solution just to ensure they don't cause inefficiencies in hitting dates. Even though they would agree there is a possible better solution, they aren't willing to pay the price to get there. I like Jeffrey's thinking here, your customers need to understand the tradeoffs. The only way for them to do that is to be more part of the process. For companies that can figure out how to do this, those will be the ones who provide innovative products that their customers believe is more than "good enough".

Commitment, Delivery and Customer Expectations

2007-05-23T08:29:00.001-07:00

Since we have been on the road of adopting Agile and Lean in our organization, I have had increasing pressure from stakeholders to give them an idea of when we will deliver functionality beyond the current release. Our releases are currently around 4-6 months at this point, so they are asking for things that will be delivered towards the end of the year and into next year. One stakeholder wanted to go out even further than that (up to 24 months in advance)!

At first, I thought that they just wanted to get an idea internally by developing a high-level product road-map to help us with our priorities. However, my thoughts were that this road-map wasn't going to focus on dates. Why? Because that information would not have any sense of reality. This information is at best high-level and with that not enough knowledge and information to make a better prediction of what could be delivered. That better prediction won't happen until we do release planning for the release that we are considering the future functionality. Even with that said, we really won't be able to commit to dates until we get really into the details of the work during iteration planning.

So, there are really three levels of precision: "fine grain" in the current iteration (90% or greater chance of stories in the iteration getting done), "coarse grain" in the current release (60% or greater chance of stories in the release getting done - this number could go down if our releases were even shorter), then "swags" in future releases (40% or less of a chance of stories getting done in the expected time period - this number going down the further out you go). These aren't exact numbers but you get the idea.

What really worried me is that they wanted these long-term dates so they could make commitments to key customers on when future functionality would be delivered. When I discussed my concerns with the stakeholders and talked about the various levels of precision, they believed we could actually get better at our precision if we just planned a little more. However, that would become waste in Lean terms because we would have a lot of work (inventory) "in-progress", plus what we know now about something that will be started six or more months in the future WILL change once we get there. Even with all of that said, my greater concern was making commitments to customers before we can make commitments with the development teams that they can deliver. It should be the other way around in my opinion. Let's figure out what we can commit to and deliver, then start setting expectations to customers.

One of our vendors lives by this philosophy. If we ask for an enhancement, if it isn't something they are committing to for the next release, they will give the answer "we'll put this in our backlog of priorities and consider this for a future release". Then, they will focus my attention of things of value that are coming in the next release and get my feedback on them. Even if my particular request isn't there, I find great value in what is about to be delivered. Sure, there may be some features I don't plan on using or care about but there are usually features that I care very much about. As a customer, this vendor is setting my expectation based on what they know they can deliver, then based on what they hope they can deliver. Hopes don't cut it! If they had promised me a future date, and then miss that date I would have been much more frustrated as a customer. I would rather not know until they were really confident they could hit that date. I appreciate that this vendor has delivered what they promised and has built a reputation with me that they are making constant improvements to their product. Even if those improvements aren't everything that I have asked for!

This is just one of the bad habits we need to break by moving to Traditional development over to Agile. It wasn't that we were better at providing estimates and hitting dates using Traditional development, because we weren't (especially for features out several releases). Sure, we could try to hit those arbitrary dates by cutting scope, freezing scope changes, adding people, working overtime, cutting corners on quality, and other "tricks of the trade". However, would we really provide what the customer really wants? Isn't the goal of Agile to provide what the customer finds value, even if the cost of that value is sometimes difficult to determine? If the quality isn't there, they don't want it early. If the functionality doesn't really do what they expected, they don't want it early. Why this importance of hitting dates because of commitment to customers, when in the end the customers will be disappointed because what they got in return missed their expectations? Let's commit to dates first as a development team and "internal" customers, then set expectations to our "external" customers as we deliver what is important to them.

I believe we got into this dilemma because we developed a reputation of over-promising and under-delivering (because of the problems with Traditional development). Plus, what we would deliver (usually 8-12 months) didn't really cut it with the customer and left them disappointed and frustrated with how long things take. Therefore, their lack of trust in hitting dates causes them to even push harder for us to commit to completing functionality that is long overdue. We need to build trust again with our customers by delivering high-value and high-quality features more frequently in order to gain back a reputation of under-promising and over-delivering. This will take time, but in the interim we need to change the way we have set future expectations with our customers. This may cause some short-term frustration, but will quickly change as customers will change their focus as well on what is being delivered then what is promised to be delivered. After all, isn't that what really counts?

The Case for Story Point Estimates

2007-05-21T10:30:00.000-07:00

I have been a fan of story points ever since I attended a seminar several years ago where Mike Cohn presented the concept. I never really trusted other estimating practices such as function points and time-based estimates. Why? Software development projects are rarely similar from project to project, yet these practices focused entirely on past experience. Therefore, to get a "reliable" estimate of time for every new project you needed to gain a lot of experience. In other words, you have to figure out up-front how you will do the work. Not only does this take a lot of investment up front, it also does not account that the work you do later could change based on the work you do now. The estimates assume that nothing will change in the effort of doing the work, which is definitely not true in the Agile world. What I like about Story points is the focus is on the relative size of "things", then how they will be accomplished. As Mike would say, "Estimate size now, derive duration later".

Even though I am sold on story points, I have had much difficulty selling the concept and value to others. I'm having an easier time with the development team, but really struggling with stakeholders/product owners. These groups always want to know when things will get done as soon as possible. Story points in itself don't give you the answer, however they help determine velocity of the team which eventually will help you better predict what could be part of a release. For various reasons, it is too much of a paradigm shift for some to accept an estimate that isn't time based, yet they have much comfort of the team throws out some arbitrary number early on a release without much knowledge of how events are going to unfold.

It looks like I'm not the only one dealing with this struggle. Tobias Meyer, on his blog Agile Thoughts, addresses some of the arguments others have had about using Story Points as estimates. Following are the arguments, read more on his post "Estimation: Time or Size" on how he address each arguments. Good stuff!

Argument 1: Estimating in hours allows a developer to measure his estimate against his actual, and use that data to improve his estimates in future.

Argument 2: Our customers/product owners don’t understand story points; they need to know how many hours developers are working so they know how much the work will cost.

Argument 3: Story Points will map to time anyway, very soon we’ll see that (e.g.) one story point is worth 2.5 hours, so it is better to skip the intermediate step and just measure in hours.

Argument 4: Story Points don’t allow you to improve your estimation techniques.

Agile or agile, Lean or lean?

2007-05-17T08:29:00.000-07:00

Recently, I have seen a trend where other bloggers and authors are having difficulty or are trying to define the difference in using Agile (capital-A) vs. agile and Lean vs. lean. Some would say that by putting a capital in front of it that it means you are focusing on a particular methodology or defined Agile process. So if you are referring to something like XP or Scrum, you should use Agile. Otherwise, if you are talking about principles and particular practices, you should use agile. Others don't want to put a label on "Lean" or "Agile" because they are afraid I guess of watering down or maybe commercializing those terms? Other people would believe that you should use capitals in some situations and not in others.

Frankly, I just don't get it. All this brings is more confusion to the community. To me, putting a capital on Lean and Agile allows my readers to know that I am talking about the values, principles, practices, tools, measures, etc of a way to develop software differently. By not putting the capitals, I don't want to confuse people with the generic terms of lean and agile which are:

lean

1. To hang outwards.
2. To press against.
3. (of a person) slim not fleshy or having little fat
4. (of meat) having little fat
4. Having little extra or little to spare; as a lean budget
5. (of a fuel-air mixture) having more air than is necessary to burn all of the fuel; more air- or oxygen- rich than necessary for a stoichiometric reaction

agile

1. Having the faculty of quick motion in the limbs; apt or ready to move; nimble; active; as, an agile boy; an agile tongue.

Agile and Lean are states of mind, are a way to develop software, have common values and principles but many practices and tools to implement. agile and lean? They don't tell me much. I'm going to use capitals!

Addressing Technical Debt

2007-05-15T13:17:00.000-07:00

What is Technical Debt and how to you manage it? These were two questions that is addressed over at the Lean Software Engineering blog.

In the first post, called "7 sources of technical debt" Bernie creates his list of where technical debt comes from:

1. Undiscovered bugs, failing tests, and open bugs in your database
2. Missing test automation (unit, feature, scenario, or system)
3. Missing build and deployment automation
4. Scenarios with incomplete user experiences
5. Code that’s too difficult to understand
6. Code that’s too difficult to extend
7. Code that’s isolated in branches

In the follow up post, called "7 strategies for measuring and tackling technical debt" Bernie provides his list of potential remedies to reduce or eliminate this debt:

1. Bugs. Measure and shrink your cycle time from when code is first written until it is tested (TDD is perfect for this, of course). Treat open bugs as work in progress (WIP) — and too much work in progress is evil. Constantly measure and manage WIP down with strategies like bug caps (no new features until existing bugs get below a defined bar).

2. Missing tests. Measure code coverage. Have a clear definition of “done” at each scale of your product: unit, feature, scenario, system. Treat units of code as incomplete until code, tests, and documentation are delivered as a set.

3. Missing automation. Make build and deployment first-class features of your project. Deliver them early, and assume (like other features) that they will constantly evolve with sub-features over the course of the project. Measure time-to-build, time-to-deploy, and time-to-test — including human time expended each cycle — and then drive those numbers down by adding automation work to your backlog.

4. Incomplete scenarios. Assign scenario ownership roles among the team. Have a clear definition of “done” for each scenario. Pre-release working scenarios to select customers as soon as possible, making special note of these bugs. And of course, minimize the number of scenarios each team focuses on at once (one at a time, if possible).

5. Not understandable. Like most open source projects, generate docs directly from the code (literate programming), including why the code does what it does. Systematically conduct code reviews. If the team can’t afford to review every line, always review a sample set of functions from each developer and component. Ask reviewers to rate code for understandability, track this data in code comments or a spreadsheet, and add workitems to the backlog to clean up those most needing improvement.

6. Not extensible. Encourage design pattern thinking. Buy your team a library of design patterns books and posters to help form that common vocabulary. Look at how open source projects tend to build big things out of many small, independent pieces — treat every component as a library with its own API and a clear (and minimized) set of dependencies.

7. Not integrated. Minimize the number of branched codelines, whether they’re formally in source code control, done on the side for customers, or sitting on developer’s disk drives. Unintegrated code is debt — it has work and surprises waiting to happen. Strive for continuous integration. This isn’t to say don’t use branching — even the smallest project should think about their mainline model — but always keep everything in source control, then track and minimize the number of active branches.

This is by no means an extensive list as I'm sure there are other contributors of debt and possible solutions. However, I agree with Bernie that these are the most common things to be on the lookout for. Just think how better the quality of software, and the ability of a team to deliver new features can be maximized if teams would see things in terms of debt!

Read more in both of these posts.

Participate in the making of a new book on Agile

2007-05-14T08:45:00.000-07:00

James Shore, a local consultant here in Portland along with Shane Warden, are writing a book called The Art of Agile Development to be published later this year. If you want to get a glimpse of what the book will be about, they have a working draft on Jim's website.

Here's a portion of the preface to give you an idea of what the book is about:

We want to help you master the art of agile development.

Agile development, like any approach to team-based software development, is a fundamentally human art, one subject to the vagaries of individuals and their interactions. To master agile development, you must learn to evaluate myriad possibilities, moment to moment, and intuitively pick the best of course of action.

How can you possibly learn such a difficult skill? Practice!

Most of this book is an étude. An étude is a piece of music that's also a teaching tool. Études help the artist learn difficult technical skills. The best études are also musically beautiful.

Our agile étude serves two purposes. First and foremost, it's a detailed description of one way to practice agile development. It's a practical guide that, if followed mindfully, will allow you to successful bring agile development to your team—or help you decide that it isn't a good choice in your situation.

Our second purpose is to help you master the art of agile development. Team software development is too nuanced for any book to tell you how to master it. Mastery comes from within: from experience and an intuitive understanding of ripples caused by the pebble of a choice.

We can't teach you how your choices will ripple throughout your organization. We don't try. You must provide the nuance and understanding. This is the only way to master the art. Follow the practices. Watch what happens. Think about why they worked... or didn't work. Then do them again. Watch what happens. What was the same? What was different? Why? Then do it again. And again. Watch what happens each time.

At first, you may struggle to understand how to do each practice. They will look easy on paper, but putting each practice into action will sometimes be difficult. Keep practicing until it's easy.

As it gets easier, you will discover that some of our rules don't work for you. In the beginning, you won't be able to tell if the problem is in our rules or in the way you're following them. Keep practicing until you're certain. Then break the rules.

Parts I and II of this book contain our agile étude. Part I helps you get started; Part II provides detailed guidance for each practice. Parts I and II should keep you occupied for many months.

When you're ready to break the rules, turn to Part III. A word of warning: there is nothing in Part III that will help you practice agile development. Instead, it's full of ideas that will help you break rules.

One day you'll discover that rules no longer hold any interest for you. After all, agile development isn't about following rules. "It's about simplicity, and feedback... communication and trust," you'll think. "It's about delivering value—and having the courage to do the right thing at the right time." You'll intuitively evaluate myriad possibilities, moment to moment, and pick the best course of action.

When you do, pass this book on to someone else, dogeared and ragged though it may be, so that they too may master the art of agile development.

But just don't take a look at this draft of the book, help the authors out! You can participate in the review process by joining the art-of-agile mailing list! I'm sure that they will appreciate the help and you can feel good in contributing back to the Agile community!

Video: Agile Testing

2007-05-10T10:12:00.000-07:00

Here is another great video. This time, the topic is Agile Testing. I met the speaker, Elisabeth Hendrickson, a few months ago at Agile Open Northwest here in Portland. She led a couple of sessions, and they were some of the best attended and interesting ones of the bunch. This lady has ENERGY and a PASSION (BIG CAPS HERE) about testing. Before meeting her, I didn't know somebody could care so much about testing software. This video is from December 2006, but is still very relevant. If you are transitioning to Agile and have been used to having a separate testing group, this video is for you. If you are still struggling with how testing should be done on an Agile project, you will get something of value. Elisabeth does a great job helping understand the shift that needs to take place to do testing differently within the context of Agile. Here is the video:

Changing the way you do long term planning

2007-05-09T08:28:00.000-07:00

We are in the process of implementing a pull scheduling system for each of our product development teams. No longer do we think in terms of projects, but in terms of releases and iterations. No longer do we figure out the entire scope in detail of what will be contained in the product, but we determine the details when we are ready in a particular iteration in a release. No longer do we have rigid control of scope for the entire project, but we try to contain what we do in a 2-week iteration once it has been planned and committed by the team. No longer do we try to figure out exactly what we will do in the next 1-2 years, but we "pull" enough work to determine the next release when we are ready to think about it. No longer do we wait for customer feedback only at the end of the project through alpha and beta cycles, but we get internal and external feedback through demos at the end of each iteration. No longer do we conduct vague and long weekly project status meetings, but we conduct quick and concise daily standup meetings. Our world has definitely changed, but in the minds of our executive stakeholders they have struggled a little with it. In particular, if you are only planning within short iterations and the current release cycle, how do you do long-term product planning for customers that want to have some idea of a delivery date for something "down the road"?

I have no problem in planning long-term if the planning is focused more on priorities than dates. Though the team is pulling just enough work for the next release, the more work that is prioritized and thought out (at least to a high level) will make the release and iteration planning go much smoother. And, actually I don't even have a problem committing to a particular date for the next release or any iteration in the release because those are fixed. However, committing to particular scope outside of the current iteration is something that I find challenging. Why?

Because we know only look at the details of the release within the current iteration, we haven't given enough thought about the remaining stories in the release. Sure, we have done some high-level estimates but we have learned that what you think you are going to do after thinking about it for a few minutes is much different when you think about it for a few hours in iteration planning. Until you really start looking at particular tasks and tests that the team will do in fulfilling each story, you don't really have an accurate estimate. And even with that said, you may find surprises within the iteration after planning.

The other challenge is that you are dealing with change in a whole different way. When you find a defect, you don't "log" it in a repository automatically because you don't have the time to fix it during the project. Instead, you deal with defects as they happen. When you show functionality in a demo, you don't log future enhancements automatically to be considered in future projects but you make appropriate changes if they are valuable and while they are fresh in the team's minds. When new issues come up, you create new stories to deal with them if they are deemed the right time. All of these things change scope, which changes what is done in a particular release. This changes the expected delivery dates many times. Which makes it very difficult to predict the scope of a release, and makes it almost impossible to predict when new functionality will be available in 12-18 months.

Now, this may sound bad initially to your stakeholders, because they want to make promises to customers as soon as possible. So, this may sound like Agile and Lean is the wrong way to go right? While I can't predict what will be done in 18 months, I can show you progress every two weeks. This progress will have higher quality because it has been tested and could be released to customers if desired. What we work on has been determined the most important to our customers, so this is a good thing. The best thing of all? The fact that we make sure that every deliverable, minimal useful functionality, really is what the customer wants because they have been more involved more frequently. So does it really matter if we try to fix scope, resources and schedule but having such as strict change control process if in the end we don't deliver what is really wanted?

So, what I have agreed on with the stakeholders is that we will go out beyond a release but strictly from a planning and goals perspective. At the point where it is determined that the goals are not obtainable in the process, I will make sure that the stakeholders are notified and can adjust the scope or define new goals based on this information. What's interesting is that I will never have to have that kind of conversation because the stakeholders will be involved in every part of the process. They will see progress being made, they will set priorities of work, they will attend demos and make changes, and they will see the impact of those changes. In the end, they will learn that they can draw the "line in the sand" when they feel most appropriate. Based on the confidence that they are gaining with the teams, they will change the long-term discussion with the customer to what is being developed than what is being planned. When that isn't enough, they can give the customer an idea of the priority of work being considered. If the customer feels that the priority isn't meeting their needs, they can voice their opinion. Then the stakeholders can determine what to do with that feedback and make priority changes as necessary. As long as customers see steady progress with frequent releases that are meeting needs, they should be satisfied with this approach.

Video: Agile Estimating and Planning

2007-05-08T15:51:00.000-07:00

I guess this week is the week of video postings, as I have discovered some more great ones to share with you. Today's is from Mike Cohn. I went to a seminar a couple of years ago on the same topic and presenter as this video. There is great content here, especially if your team is struggling with estimating and don't know about story points and planning poker concepts. Mike's presentation style is also engaging and entertaining. This presentation is is two parts, click the videos below to watch each part.

Part One

Part Two

Video: Scrum at Google

2007-05-07T08:32:00.000-07:00

Jeff Sutherland provides an excellent video discussing how the Google AdWords team adopted and adapted Scrum to fit their team and the Google environment. As my organization has been going through adoption of Agile, I saw similar "lessons learned" as described in the video. I think there is a lot of valuable knowledge to be gained listening to Jeff. Enjoy!

Another View of Lean Leadership

2007-05-04T08:34:00.000-07:00

Joe Little from Agile and Business has an excellent post today called "Leaders, Managers, Bosses and Administrators". Actually, he focuses on what leadership means in the Lean and Agile world. He refers to the Poppendieck book, "Implementing Lean Software Development: From Concept to Cash" (If I had to pick one book to understand Lean as applied to software, this would be the one!). Here's his take on leadership after reading the book:

They [Poppendiecks] raise several excellent points.

1. A team needs leadership. Which is to say, vision. Someone to inspire and someone to help them put their hearts in the game. And keep it there.

2. The project needs decisiveness. If the team has too many leaders, and the leaders squabble about decisions, then the team wastes time. The team needs to know how it will make decisions. There is a trade-off between making the right decisions, and making decisions quickly.

3. Generally, the team needs to learn to make decisions only at the last responsible moment. So, much of the decision-making is about when to make a decision. At what point have you learned enough to make a good/better decision?

4. The project needs business decisions and technical decisions. This is very true. So the team needs business people and needs technical people who are ready and able to make those decisions. And, preferrably, business people who understand technology and technology people who understand business (and the customers).

5. And there are many other types of decisions to be made too. People decisions. Decisions about whose insights to go with on specific areas. Process decisions. Decisions about who is working effectively and who is not. Decisions about how to get the team to communicate better and learn faster.

6. In Scrum, we have ScrumMasters and Product Owners. These roles are endowed with certain leadership aspects. This is different than the leadership of the Chief Engineer, which is a role Toyota uses.

7. Project managers have also provided leadership. (And we have the whole PMP, PMI thing, too.) PMs have also provided managership, bossiness, administration and other things.

8. We know that no bosses are wanted. We want all the best from every person, and a boss will only inhibit that. A boss wants to command others, and thinks he knows all. These are not helpful traits in a learning situation. (So, semantically, we are using "boss" here to represent all the bad things that a boss can be. Of course, few managers or leaders are as bad as a boss, but we all can be that way sometimes.)

The Agile Team's Dashboard

2007-05-02T11:27:00.000-07:00

Last summer, I shopped for a new car. I decided with rising gas prices to purchase a Hybrid. My goal in buying this car was to improve my miles per gallon (MPG) so I didn't have to spend as much as often at the pump. With this car, I have two gauges that I didn't have before. The first is next to my speedometer and shows me my MPG consumption (or electric if no gas is being used). The other gauge will show me the Tank average for MPG.

At first, I wasn't used to these measures and didn't pay much attention to them. Therefore, my driving habits were similar to what I had done before with my other car. I drove the car as I had driven the previous one. As I got familiar with my car, I started to explore both of these gauges. I started experimenting with how to increase my MPG usage. I discovered that if I accelerated a little slower after being stopped it made a difference. I discovered that I could coast more often when going downhill instead of hitting the gas and still maintain the same speed (but save on MPG usage). As I result, I started to see the MPG increase.

Then, I decided to not pay attention to either gauge for awhile. Just try to drive the car using what I have learned and see if I got the same results. Though I did better than when I first started using the car, I still was getting lower MPG by not paying attention to these gauges. These gauges were a good visual tool to help me accomplish my goal of higher MPG, and were a good reminder when my bad habits would come back and cause my MPG to drop.

So, why all of this talk about cars? Well, think about Agile teams. Their goal is develop as much working (deliverable, quality, meets customer needs) software as possible in set increments of time. How do they measure this? There are two gauges that most Agile teams use to ensure they focus on the goal of delivering working software.

First, the team calculates their velocity from Iteration to Iteration. Velocity measures the total amount of work that was accomplished to deliver working, completed stories by the end of the iteration. If a story is partially completed, the team does not get credit until the iteration in which the story has been completed. This encourages teams to focus on completing stories than completed tasks leaving incomplete stories. If the team is focusing on the completion of stories, as is a dedicated team of individuals, you should be able to sustain a certain range of velocity from iteration to iteration. Where Velocity gets lower, you should see why things are getting done. Where Velocity is higher, you need to see why the team was able to accomplish more. If the team pays attention to Velocity, they will begin to see better development practices (much like my driving habits improved after paying attention to MPG).

Second, the team uses a chart to track progress at both a Release and Iteration Level. The burndown chart shows how work in being completed on a "trip" by "trip" basis. At a Release Level, the chart shows how each Iteration is progressing and the team can get a good idea of what amount of scope is possible based on this information. At an Iteration Level, the chart takes the daily remaining hours for the team to be used in the Daily Standup to see progress. If the chart isn't going down as quickly as expected, this is an indication that the team has impediments that must be addressed. These impediments could be thrashing (team isn't getting to a resolution), dependencies to other stories/tasks (team needs to remove dependencies where they can), additional scope (additional stories and/or tasks have been determined necessary to get complete). In any of these cases, the team can quickly see the impediments impact to the iteration and make adjustments.

Both Velocity and Burndown charts are good gauges to ensure that you are doing what you can to deliver working software. These measures should expose you to some problems (waste in Lean thinking) that need to be resolved. Most of the time this requires a change in how you do things in order to accomplish the goal better. If Agile teams aren't using these gauges, much like my driving, they can get back into bad habits and find themselves not having great results after each Iteration.

Thinking inside the Agile Box

2007-04-30T14:19:00.000-07:00

Pretend you are taking a week long trip with a friend. You start packing and end up with a couple of large suitcases. Your friend arrives with one medium suitcase. You start thinking, "What are they thinking? They don't have enough clothes to last the week." You arrive to your destination. When you start unpacking, your friend's clothes are all nicely folded and no wrinkles. Your clothes look like a mess and will need to be ironed. Your friend is able to put things into drawers quickly as well as the bathroom items. You have to go through your bag and it takes you longer to find things and put them in their place. However, your friend has less clothes than you. As the week progresses, you begin to see that your friend has found ways to reuse what they wear - different shirts with the same pants - yet still appear like they aren't worn. You on the other hand, find that at the end of the trip you had clothes that you didn't wear, but had to carry around. Now you start thinking, "My friend has a better way. I've got to learn from them for the next trip!"

Was the friend always this organized? Probably not, they probably learned by somebody else. Experienced travelers have tricks on how to properly fold clothes and how to maximize every spaces in a suitcase. They have also learned what's "good enough" and not to take more than they need. Even if they end up needing fresh clothes, they will end up using a local cleaners or wash their own clothes instead of taking too much. They have learned how to best put clothes and other accessories together to maximize their use.

As my development staff is asked to reduce cycle times through iterative development using Agile best practices, their initial reaction is "That sounds all good, but we have found things to take weeks or months and that won't work with this approach." They are correct, it won't. But, just because you have done things a certain way doesn't mean that it is the ONLY way to do things. We have learned how to do things as individuals. We will now need to do those things together as a team. We have learned that each person plays a particular role and has their own specialization. We will now need to remove those specializations and have people wear multiple hats. We have learned to think about everything up front is much detail. We will now need to learn to do just enough for now and think about details when it is appropriate later. We have learned to break the work and requirements down just enough to make sense for a 6-8 month project, we will now need to refine that work to make sense for 2-4 week iterations. We have learned a way to deliver software after 8 months, we need a way to deliver that same software in a portion of that time and possibly in weeks instead of months. We need systems that get tested and built at least every day if not several times a day, the systems we have used cannot handle that load.

What's great about Agile is that it isn't just "theory" anymore, people have found it to work and improve the flexibility and responsiveness of the team to provide working solutions to customers. We can tap into these "experienced travelers" and figure out how they have done it a different and better way. It just seems strange to us because we have learned how to do things differently, not necessarily better. However, if you think that Agile is just about doing things faster you will miss the point. Agile is about doing things differently in order to deliver working software more effectively. That means we need to relearn how to build, maintain and support software under this new way. Relearn skills, tools, knowledge, processes and essentially how you do your jobs differently.

You might have thought you were a great traveler and packed accordingly, until you realized that there is a better way. However, you can't just copy the better way like it was out of a cookbook. You must understand why this experienced traveler didn't to make the change. Then, you must understand how he made the change work for them. Then, you must make the changes work for yourself. Same with Agile, you don't just adopt a methodology and away you go. You must understand the principles behind it, then determine what set of best practices fit your organization. When people ask us if we are using XP or Scrum, I respond with "Yes". "Which one?", they ask. "Both" I respond. There are practices within each that we have adopted to support Agile principles. There are other practices that we have come up with that perhaps nobody else is using. That's ok, as long as we are staying true to the goals of Agile development. I will say that it has required EVERY person to think about their jobs differently. In some cases, much differently. In the end, it required each person to "think inside a different box". An Agile Box!

Where does QA fit within Agile?

2007-04-25T08:16:00.000-07:00

Like many development shops that have been around doing Traditional Waterfall development over the years, we have a separate department called Quality Assurance. This team consisted of non-developer types that focused primarily with ensuring quality of the product through testing activities. In our case, because we have legacy products that have been developed in technologies that don't lend itself well toward test automation, these testing activities were largely done manually. Unlike the unit-level testing that was done by the developers, the QA team primarily focused on overall system testing. This included test verification of current requirements but also a fair amount of regression testing (again manually done) to ensure that
past functionality continued to work.

As with Waterfall development, QA was largely involved late in the process. Once developers felt their code was completed, they would "throw in over the wall" to QA for testing. QA would do their initial testing, and submit bugs for things not working. Until the overall product was stable enough to give to our customers (through several release cycles), it would go through several fix/build cycles. How long this would go was highly unpredictable, and therefore would cause much frustration if our initial test estimates were too far off.

A couple of years ago, I attended the local Northwest Software Quality Conference here in Portland, Oregon. This is an excellent conference and while anybody in the software industry can get something useful out of it, the primary audience are people involved in QA. As Agile started to heat up in the software industry, presentations started to show up in every conference. Given this audience however, very few had even heard much of Agile and for those that did, weren't using Agile practices. Why? Because Agile assumed (at least at that time) that the testing role and other QA activities were just another hat that developers wore. So for those that had heard of Agile, they were resistant because they feared their jobs in the long run.

I was seeing that same reaction from my own QA group, and was dealing with how to best transition that group into a valuable role of an Agile team. After many sessions with both developers and QA, I started to see the light of how QA could fill some holes that the teams were having. So, here's how QA will look in our organization:

1) QA integrated into every team - Each member of the QA team is now co-located with the Development instead of being in a separate department.

2) QA plays Testing Manager role - QA is responsible for helping the team identify how we know when a story or task is "done". They define done by developing tests with the team that ANYBODY can run including the QA person. They also determine how best to implement that test (manual or automated, which tools, etc.)

3) QA plays Process Improvement Manager role - QA is now going to lead retrospectives at the end of iterations and releases with the team. They will ensure that there is just enough process for the team to ensure quality but not too much that the team doesn't see value in the process. They are also to ensure that all action items from the retrospective get reflected in the work in future iterations and releases.

With these three implementations, we hope to build more quality into the entire process and ensure quality earlier and often in the process. While anybody can wear the Tester hat, now QA has a much broader role in ensuring quality beyond just testing.

The Top 5: Feb - April 2007

2007-04-24T15:22:00.000-07:00

Here are the top 5 posts most visited from February 5th through April 24th:

#1: What does it mean to be Lean? - Lean is a paradigm shift from normal thinking. Learn what it takes to think in Lean terms.

#2: An Introduction - Just some background of why I created this site and who I am.

#3: The Predictability Paradox - A link to a wonderful white paper written by Mary Poppendieck discussing predictability in software projects.

#4: An Agile Approach to adopting Agile - Some advocate taking the big bang approach to adopting Agile. Not me! If Agile is all about incremental and evolutionary delivery, why can't your adoption be the same way?

#5: Deming Revisited - I tackled Deming awhile ago from a management perspective, now I update Deming's list of Fourteen Points from an Agile and Lean perspective.

Growing Your Leaders

2007-04-23T11:20:00.000-07:00

Alan Shalloway from the Net Objectives Thoughts blog challenges how Agile has viewed management and leadership up to this point in his post "The Need for Leadership in Scrum". Here are some highlights:

There underscores the notion that self-managing teams and direction from the top are somehow opposed to each other. In practice, this may be. But that is because in practice, leadership is missing. Direction in the form of command and control is bad. Direction in the form of vision and matching people to their abilities is not. This points to one deficiency I have seen in the general Agile community, and one that, I believe, is a serious impediment to having the Agile community have success at the enterprise level. I am referring to the fact that most Agilists do not trust management to do what is right – they feel the team must be protected from management. I have heard this opinion voiced in several books and from several leading agile consultants/practitioners – and I disagree with it.

Unfortunately, this attitude has been pervasive from the beginning of the Agile community. Given the heavy focus on over-bearing process that has gripped this industry for years, this is not surprising. However, it is time to include leadership and management in the transition to agile methods. We must see how we can marry respect for people (and hence the team) along with process. This requires leadership and proper management.

Buckingham, in his excellent book, The One Thing You Should Know describes leadership as the “ability to create a solid vision of a better future for those people he/she is leading. ” A leader must have a compelling desire to move towards that vision. Buckingham defines management as the “ability to match people’s tasks with their skills. ” A good manager is liked a skilled Chess Master. As opposed to checkers, where all the pieces (until one gets kinged) are the same, in chess, different pieces have different strengths. A good chess player knows how to use these strengths. A good manager knows how to maximize the strengths of his staff. Hence, leadership and management are not opposed to each other – they just have different functions.

Keeping Buckingham’s views in mind, the role of leadership and management then is to create the vision for the Scrum teams to be effective and to staff the teams with the appropriate resources. But just creating the vision is not enough. Leadership and management must help create the structure within which the team works to assist the team. This is something Scrum teams cannot do on their own. Ignoring this issue is what often has Scrum teams feel they must protect themselves from the organization when what they need to do is help the organization see how to better support the team.

If Scrum can’t help us here, what can? This is where Lean Thinking comes in. In a nutshell, Lean Thinking tells us to set things up so we can have fast-flexible-flow. That is, we can get customer requests in and get business value out – quickly and repeatedly. At Net Objectives we base our entire software development philosophy on this (which is why we call it SPeeD: Sustainable Product Development). The issue is – “how can I develop software quickly now (to deliver business value quickly) while retaining the ability to add business value quickly in the future.”

Read more in Alan's post.

In the book The Toyota Way by Jeffery Liker, one of the 14 principles outlined says "Grow Leaders Who Throughly Understand the Work, Live the Philosophy, and Teach It to Others". At the beginning of the chapter, there's an excellent quote that parallels Alan's frustration mentioned above:

Until senior management gets their egos out of the way and goes to the whole team and leads them all together...senior management will continue to miss out on the brain power and extraordinary capabilities of all their employees. At Toyota, we simple place the highest value on our team members and do the best we can to listen to them and incorporate their ideas into our planning process. -- Alex Warren, former Senior VP, Toyota Motor Manufacturing, Kentucky.

In summarizing this principle, Liker says:

If we look at all of the great leaders in Toyota's history we see they share several common traits:
1) Focused on a long-term purpose for Toyota as a value-added contributor to society.
2) Never deviated from the precepts of the Toyota Way DNA and lived and modeled themselves around this for all to see.
3) Worked their way up doing the detailed work and continued to go to the gemba - the actual place where the real added-value work is done.
4) Saw problems as opportunities to train and coach their people.

This doesn't sound like the ol' command and control type of manager works in this paradigm does it? However, managers are needed to provide Agile teams this kind of guidance. To do so, managers must respect team members as well as the other way around. Each have their place and need to understand their interdependencies in order to be a successful organization.

Now, Rules for Self-Leadership

2007-04-19T15:44:00.000-07:00

A few posts ago, I talked about a great post by Rosa Say around self-management. Now, she has followed this up with a separate list this time focused on leadership qualities. I agree with Rosa that there is a difference between management and leadership qualities and that everybody should strive to accomplish both. Here, Rosa elaborates on this as well as presents the 12 Rules of Self-Leadership. Enjoy!

Management and Leadership are not interchangeable words for me. We need both of them, for in part, management tends to be more internally focused (within a company, within an industry, within a person) whereas leadership is more externally focused on the future-forward actions you will take in the greater context of industry, community, or society. They have commonality to be sure, for instance, both are about capitalizing on human capacity, however they are defined by the differences we value in them: Management tends to be about systems and processes, whereas Leadership is more about ideas and experiments.

I believe there is both art and discipline in each, and I think of these rules as the discipline which helps reveal the great capacity of the art. Thus last time, twelve suggestions to help you self-manage, with a more disciplined you newly able to reveal your art. Now, twelve to help you self-lead, so a more disciplined you is newly able to reveal the art in others, those who choose you to lead them.

12 Rules for Self-Leadership

1. Set goals for your life; not just for your job. What we think of as “meaning of life” goals affect your lifestyle outside of work too, and you get whole-life context, not just work-life, each feeding off the other.
2. Practice discretion constantly, and lead with the example of how your own good behavior does get great results. Otherwise, why should anyone follow you when you lead?
3. Take initiative. Volunteer to be first. Be daring, bold, brave and fearless, willing to fall down, fail, and get up again for another round. Starting with vulnerability has this amazing way of making us stronger when all is done.
4. Be humble and give away the credit. Going before others is only part of leading; you have to go with them too. Therefore, they’ve got to want you around!
5. Learn to love ideas and experiments. Turn them into pilot programs that preface impulsive decisions. Everything was impossible until the first person did it.
6. Live in wonder. Wonder why, and prize “Why not?” as your favorite question. Be insatiably curious, and question everything.
7. There are some things you don’t take liberty with no matter how innovative you are when you lead. For instance, to have integrity means to tell the truth. To be ethical is to do the right thing. These are not fuzzy concepts.
8. Believe that beauty exists in everything and in everyone, and then go about finding it. You’ll be amazed how little you have to invent and much is waiting to be displayed.
9. Actively reject pessimism and be an optimist. Say you have zero tolerance for negativity and self-fulfilling prophecies of doubt, and mean it.
10. Champion change. As the saying goes, those who do what they’ve always done, will get what they’ve always gotten. The only things they do get more of are apathy, complacency, and boredom.
11. Be a lifelong learner, and be a fanatic about it. Surround yourself with mentors and people smarter than you. Seek to be continually inspired by something, learning what your triggers are.
12. Care for and about people. Compassion and empathy become you, and keep you ever-connected to your humanity. People will choose you to lead them.

One Story Flow

2007-04-18T12:06:00.000-07:00

One of the practices in Lean is this idea of One Piece Flow. What does this mean exactly? It focuses on getting more components completed more quickly by focusing on the completion of each one instead of having too much work in process. The thinking is that if your team can produce one component quickly with quality, you will have peak efficiency for future components.

So, where does this fit into Agile? In iteration planning, the team is working with a set of Stories. Here's where you can go in two different directions and has been a constant challenge in our organization. Where you go from here can potentially have two very different results.

First, you could ask the team who wants to work on each Story. Each individual responsible for the Story can then go off and determine the Tasks and Tests necessary to complete the Story. So, if you have a team of five developers, you will have five Stories being worked on (at a minimum, they may take on more Stories if they believe they have time in the Iteration and the Stories are small enough).

Or, you could have the entire team in Iteration Planning determine both the Tests and Tasks necessary for the first two or three Stories (more if you have a smooth process ). Then, have the team implement the first Story together (or as many team members as possible). If there isn't enough work for everyone, some go and start implementing the second Story. If the team members responsible for the first Story need help, others working on the second Story join the effort. The goal here is for the team to get each Story implemented as quickly as possible before they go on to the next Story. This is what I will coin "One Story Flow".

As I mention, we are still really struggling with this because it has been the nature of developers to have each person work on their individual piece and to bring the pieces together during some time of integration. However, this way of thinking starts to produce mini-Waterfalls within each iteration. Also, I have seen more unfinished Stories as a result. I would rather have more Stories that have been carried over to the next iteration at 0% than at 50%. I would also want more Stories that have been completed by the team at 100%.

If you are struggling with the team's performance and completion of Stories, try the "One Story Flow" approach. You will have more flexibility as a team and the team will be more in sync on how each Story has been implemented and how it contributes to the big picture of the product. More discussions up front will resolve conflicts and turn assumptions into facts. Finding ways to implement one Story after another will address issues such as specialization of skills, knowledge, and bottlenecks in the development process. These are painful but good things to discover. Because if the team can find ways to implement one Story better, they will be able to implement all Stories better!

Rules for Self-Management

2007-04-17T13:13:00.000-07:00

If you want Agile to be more successful for your organization, you need each individual to think differently about their role and expectation from management. For the teams to be self-organizing, you need each individual to be self-managed. This isn't to say that management in an Agile organization isn't necessary, as they need to be there to provide direction and remove roadblocks for the team if the team cannot do it for themselves.

However, the more the team can manage for themselves, the greater the productivity and efficiency. After all, part of Agile is to be able to deliver working software with better quality and response time.

Rosa Say has always been one of my favorite bloggers on the top of management. She has a recent post on her Talking Story blog that as I read it realized how well it would work on an Agile team if the team adopted her 12 Rules of Self-Management. Here they are, read it for yourself and imagine if each individual followed these rules:

1. Live by your values, whatever they are. You confuse people when you don’t, because they can’t predict how you’ll behave.
2. Speak up! No one can “hear” what you’re thinking without you be willing to stand up for it. Mind-reading is something most people can’t do.
3. Honor your own good word, and keep the promises you make. If not, people eventually stop believing most of what you say, and your words will no longer work for you.
4. When you ask for more responsibility, expect to be held fully accountable. This is what seizing ownership of something is all about; it’s usually an all or nothing kind of thing, and so you’ve got to treat it that way.
5. Don’t expect people to trust you if you aren’t willing to be trustworthy for them first and foremost. Trust is an outcome of fulfilled expectations.
6. Be more productive by creating good habits and rejecting bad ones. Good habits corral your energies into a momentum-building rhythm for you; bad habits sap your energies and drain you.
7. Have a good work ethic, for it seems to be getting rare today. Curious, for those “old-fashioned” values like dependability, timeliness, professionalism and diligence are prized more than ever before. Be action-oriented. Seek to make things work. Be willing to do what it takes.
8. Be interesting. Read voraciously, and listen to learn, then teach and share everything you know. No one owes you their attention; you have to earn it and keep attracting it.
9. Be nice. Be courteous, polite and respectful. Be considerate. Manners still count for an awful lot in life, and thank goodness they do.
10. Be self-disciplined. That’s what adults are supposed to “grow up” to be.
11. Don’t be a victim or a martyr. You always have a choice, so don’t shy from it: Choose and choose without regret. Look forward and be enthusiastic.
12. Keep healthy and take care of yourself. Exercise your mind, body and spirit so you can be someone people count on, and so you can live expansively and with abundance.

Iteration Planning for the Unplanned

2007-04-16T12:28:00.000-07:00

On any project, you will have both planned and unplanned tasks. Planned are those tasks that you knew ahead of time and scheduled some of your work towards those efforts. Unplanned are everything else. In many software shops including mine, the people doing the development work also help internal operations with ongoing technical support for internal and external customers.

Though Agile helps adapt to change by accommodating unplanned tasks in future iterations, there is still a challenge of what happens in the current iteration. If you are doing Scrum, it is expected that at the start of each iteration that the team commits to the scope of that iteration and that any additional work requested by customers should be considered in future iterations. Sounds good, but most technical support is needed on a much frequent basis and turnaround must be at a minimum. Therefore, you can have support issues to resolve on an hourly basis at times and you must respond within the next 12 - 24 hours. So, how do teams address these issues? Mishkin Berteig from Agile Advice has identified a few in his latest post:

1) The part-time allocation to the iteration's work is most common. There are a couple things that need to be done to make this work well in the long term. The main thing is to make the allocation of time inflexible: if you allocate 50% to the iteration and 50% to support, then you should never be flexible about that allocation. This is necessary in order for the team to make a commitment at the start of each iteration.

2) Another common method is the "fluorescent note card" method which requires stakeholder negotiation around the impact of interruptions. With this method, any time a stakeholder comes to the team with a request, the Process Facilitator writes the request on a bright colored note card. The team then does a task breakdown on the card and using their normal process (whatever that is) estimates the work. The requesting stakeholder then has to negotiate with any other stakeholders about what work to remove from the iteration in order to make room for the new work. The trick here is that the team has to be involved because they have already started on some of the work and it might be difficult to dis-entangle things enough. This process works well primarily because it makes the tradeoffs visible. It does not work so well with letting the team make their commitments.

3) A third common method is to form two separate teams: one doing new work, one doing support work. This is simple, effective, and annoying for the people on the team doing the support work! Please don't consider a rotation system since this destroys the process of team development and makes it nearly impossible for the team doing new work to learn their capacity for the purposes of commitment.

4) A less common, but fourth method is to have extremely short iterations. In this method, choose your iteration length to be so short that you can always start work on urgent interruptions before anyone gets impatient! This can be exhausting, but it is one of the best ways to get the team and the organization to understand the large toll that these interruptions take.

We have tried both #1 and #3. The problem with #3 is that developers that are put on the support work would love to do some development work. Plus, it is often difficult to have the knowledge (at least in our organization) to have a select few people who can address all of the support issues. However, if you try and combine support issues with ongoing development, you will find yourself trying #1 next. Unfortunately, it is very difficult to estimate what percentage of time should be put towards Support vs. Development as Support issues will come and go at an expected rate.

I don't think we can manage one week iterations (idea #4) effectively at this point at least until we become more mature at Agile and have the right infrastructure in place to turnaround iterations this quickly.

This leaves us with #2, which is what we are trying to do at this point. On our current Iteration Schedule board, we will have a Story called Issues that will be at the top of the list. Then, every issue will be identified as a Task and will have a color designation to help it stand out against "Planned" tasks and stories. I like this idea because it helps the team address what is reality but also balance the priorities of the Issues against Development stories. It also makes since that the Development team OWN the support issues since they have the most knowledge and skills to address issues in their own product and technical domain.

The Ultimate Measure of Agile?

2007-04-12T14:12:00.000-07:00

I attended a local lunch meeting where a panel of people talked about Agile development and how they addressed challenges in each of their organizations. Two of the panelists were managers in software product environments. As they talked, it was obvious that each person was at a different stage of maturity in their adoption of Agile. The first speaker seemed to think they were doing Agile development, but his description of their lifecycle seemed to indicate that they were just doing Iterative development with mini-waterfalls. The second speaker seemed to be much further along as they were committed to short, timeboxed iterations of 2 weeks with co-located teams serving both testing and development efforts.

When it came to Q&A, there was something that was bugging me so I asked them this question, "How has the adoption of Agile impacted the Release Cycle time?" The answer from both was what I expected after hearing the earlier discussion, "Not much at all". Because of marketing and other budgetary reasons, they felt they could only do a release or perhaps two a year.

And yet, especially that they are managers, I think they are missing perhaps the ultimate measure of Agile - How quickly are we giving our customers working software? Perhaps this is what I have learned from Lean and the emphasis on Waste and Value. Within Lean, the emphasis is to always improve the cycle time to the customer by providing the most valuable functions and minimizing waste. Part of waste is partially completed functionality.

I think as people implement Agile, they will realize that they need to mature to a point where they are incorporating Lean. After all, shouldn't software be about maximizing ROI from both the customer and shareholder point-of-view? You cannot improve ROI if you aren't improving the cycle time. You cannot improve cycle time without improving quality and removing waste.

While each of these individuals have gained some benefits with Agile such as better adapting to change, getting frequent feedback, and better ways to track progress, they still haven't realized the greatest reward of all - getting working software out to customers as quickly as possible. This goes beyond budgets and marketing plans, and turns the organization 180 degrees to fully accommodate customers. They don't want to have to wait a whole year for another product, they want changes as quickly as possible. If you can't find a way to make it happen, it's possible a competitor will.

Maximizing Customer and Shareholder ROI by provide working product that meets both business and customer value as quickly as possible. This is the ultimate measure of being an Agile organization!

The Agile Zealot's Handbook

2007-04-11T08:36:00.000-07:00

Simon Baker from Agile in Action has updated his Zealot's Handbook. Here's what it says:

VALUE
IF you don't repeatedly release software
into a production environment
at least once every month
that realizes business value
for a real customer...

QUALITY
IF you're not paying constant attention to technical excellence
with simple, effective, incremental design
driven by continuous, repeatable automated testing
with at least 95% coverage...

LEARNING
IF you're not learning
by inspecting and reflecting every iteration
and you're not re-planning, adapting and improving
all of the time based on what you've learned...

TEAM
IF your team is not empowered to self-organize and be creative,
does not sit together and engage in face-to-face communication,
does not include your customer
and all the necessary skills to make its own decisions and take immediate action...

THEN YOU HAVE COMPROMISED YOUR AGILITY

Very nice! Short and sweet and incorporates all part of Agile. Thanks for the update Simon!

Deming Revisited

2007-04-09T09:50:00.000-07:00

Note: This post originally appeared in my other blog, Random Thoughts from a CTO in February 2006. I have updated it based on what I have learned and tied it more directly to Lean development.

You can't talk about management without eventually bringing up W. Edwards Deming. Deming is a fascinating individual and his ideas have challenged the future of management. One of his contributions is fourteen key principles of management for transforming business effectiveness. As I have been learning more about Lean, I am finding that there is significant overlap between Lean principles and these by Deming. This is no coincidence, as Deming learned a lot from Japanese companies include Toyota. Here are each of the 14, with my comments following each one:

1. Create constancy of purpose for the improvement of product and service, with the aim to become competitive, stay in business, and provide jobs.

Skip: Too many times in software development, we fall victim to the "kitchen-sink mentality". The focus is on what the technology can do more than what is important for business. We lose track that we need to get something out the door that allows us to stay competitive and in business. Competition has shown us that things can get done more quickly. Customers expectations are higher than ever in wanting quick responsiveness. You must figure out what you absolutely have to produce and then get it out as quickly as possible without compromising quality. Is your company aligned around your particular products or do they have various project teams that can players every few months? Is it easy or difficult to understand why products are going because initiatives don't align well with the progression of each product?

2. Adopt a new philosophy of cooperation (win-win) in which everybody wins and put it into practice by teaching it to employees, customers and suppliers.

Skip: Partnerships, this is what we are talking about. Let's work on the solution together. Let's get rid of our own political agendas and figure out what we can make together. We need each other, so let's recognize that. We are moving more towards this model but it is slow going. There are still customers, employees and suppliers who just don't "get it" and are only looking out for number 1. I wish those people could realize how much easier life would be if you think of these relationships as cooperative instead of "give-and-take".

3. Cease dependence on mass inspection to achieve quality. Instead, improve the process and build quality into the product in the first place.

Skip: Years ago, if you did think about quality you usually had a testing group that would manually review the product before it went out the door. Then, if you took it a step further, you would have customers "test" your product in an "alpha" or "beta" release. However, organizations are finding out that it is best to be proactive instead of reactive when it comes to quality. Automate testing, create tests before design or implementation, make people think about the quality of the solution as requirements are being gathered. Those organizations that build quality in by checking early and often are going to be more successful in the long-term. Why? Because while the competitors are having to go back and fix past mistakes that could have been caught, you are able to focus on better meeting the customers' needs right now!

4. End the practice of awarding business on the basis of price tag alone. Instead, minimize total cost in the long run. Move toward a single supplier for any one item, based on a long-term relationship of loyalty and trust.

Skip: I have learned a long time ago when it comes to looking at financials and success in any business, you have to focus on the long-term results and impact of your short-term decisions. You may think your short-term decisions have merit, but if you choose to ignore (or understand) the long term negative effects you could get yourself in trouble. On the other hand, if you are focused on long-term partnerships that work for all of those concerned, you will be better off.

5. Improve constantly, and forever, the system of production, service, planning, of any activity. This will improve quality and productivity and thus constantly decrease costs.

Skip: For many organizations, this is a challenge. Why? Because they like to move on. Fix something, it works, leave it only since it works (the ol' adage, "If it's not broke, don't fix it"). So, even if it does seem to work,it may not be the most efficient or it may tend to have problems as time going on and the world changes around it. The best organizations are those that recognize that change is always happening and can adapt to those changes quickly. When they realize that there are always improvements, and the company rewards those who make improvements to improve quality or productivity or reduce costs, then you will have an efficient organization. Now, I encourage others to have the attitude of "Is there a better way?"

6. Institute training for skills.

Skip: Does your organization have formal training? Is it for new people only? Or does your organization expect the skills to be developed before they are hired? Or does your organization "train by fire". Organizations need to evaluate their training programs and look for consistency. In other words, are people being consistent in their performance based on their knowledge? Are they getting the same training? If they need refresher courses, is that available? If everyone isn't "sharpening their blades" on a regular basis you can quickly find that you aren't able to adapt to changing conditions or have as much flexibility in responding to change as you used to. The world is changing around us, don't you think each of us needs to change and improve as well? How can you do that without some time around training?

7. Adopt and institute leadership for the management of people, recognizing their different abilities, capabilities, and aspiration. The aim of leadership should be to help people, machines, and gadgets do a better job. Leadership of management is in need of overhaul, as well as leadership of production workers.

Skip: I am a firm believer that leadership isn't something you are born with, isn't part of your DNA, isn't part of a job title, and that each person is capable of becoming a leader. Some will take on more responsibilities than others, but each person need to become an expert of their own domain. You can't be an expert in my opinion if you aren't leading others with your expertise. Organizations promote people every day into management roles that they are prepared for. We should constantly encourage the building of leaders within the organization. I have told many people that one of my primary job duties is to create leaders for the organization. It is my legacy. I truly believe that!

8. Drive out fear and build trust so that everyone can work more effectively.

Skip: Mutual trust and respect are the heart and soul of a team's productivity. If you don't have one or both towards a teammate, you will find extra work is produced to work around this. Extra work means less productivity...period. Also, you also need to be able to take risks and make decisions that could have a negative impact. Of course, you should do your homework if the risk occurs. On the other hand, you can't be scared of making decisions by always going the safe route.

9. Break down barriers between departments. Abolish competition and build a win-win system of cooperation within the organization. People in research, design, sales, and production must work as a team to foresee problems of production and use that might be encountered with the product or service.

Skip: I have seen this happen in both large companies I have worked for as well as smaller ones. Why does this happen? Easy....departments have different goals that are in conflict with each other. Processes or work doesn't seem to flow well between departments, or work seems to contradict each other. This causes much frustration and the feeling of "us" vs. "them". I have even seen organizations with there are contests between different development groups to see who is better. While this may sound like healthy competition, it often creates "classes" of groups within the organization that is anything but healthy. The entire organization should look at the processes across the organization and determine if everyone is working towards customer needs in the most efficient way possible. We can't forget that the "us" is the entire organization and the "them" is our customers, vendors and even competitors. We can't forget to serve customers, support vendors and go after the external competitors for market share.

10. Eliminate slogans, exhortations, and targets asking for zero defects or new levels of productivity. Such exhortations only create adversarial relationships, as the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the work force.

Skip: This is SO true. The cost of trying to support zero defects goes up exponentially as you get closer to zero. There are always acceptable defects that can wait on being fixed. The opposite is also true, in that there are defects that could cause data corruption that are not acceptable. You need to decide what is and is not acceptable and make sure that employees, customers and vendors are on the same page. If customers expect zero defects, and internally you know that is impossible, you will only have frustration on both sides.

11. Eliminate numerical goals, numerical quotas and management by objectives. Substitute leadership.

Skip: I hear from people that you "can't manage what you can't measure". However, I really haven't had any particular metrics for many years....yet, I know that I have been successful with the groups that I have managed. Whenever I have tried to implement metrics...it seems the focus has been more on keeping up with the metrics than focusing on the work itself. Isn't it the work that should be more important? In the end, I'll pick a good leader that motivates people to perform over a manager that is making sure his people are conforming to the metrics any time! I'm not saying metrics are bad but that they need to measure true value and support the right goals and behaviors of the organization.

12. Remove barriers that rob people of joy in their work. This will mean abolishing the annual rating or merit system that ranks people and creates competition and conflict.

Skip: Dilbert and The Office typically portray the workplace as a terrible place to be. Though it's done in a humorous fashion, it's still a testament to how people view most organizations. Managers from all levels should pay attention to this one, as they create programs, policies, procedures, etc that seem like good ideas but cause more conflict than they resolve. Are these things going towards improving morale and productivity or against it?

13. Institute a vigorous program of education and self-improvement.

Skip: Education and self-improvement seem to take a second seat to getting the work done. They are either used as recruiting perks, or rewards for hard work. I haven't been with an organization that puts a high enough value on these activities. Most people don't realize that if the company invests in people this way, and encourages self-improvement....that will lead to organizational improvement with better skills, higher knowledge and overall desire to learn.

14. Put everybody in the company to work to accomplish the transformation. The transformation is everybody's job.

Skip: Imagine a company where everybody has a particular role, everybody knew what was most important or not, everybody worked towards the same goals. I haven't seen it yet, but would love to be part of an organization like that! This is the ultimate goal of a Lean and Agile organization.

A good resource on code reviews

2007-04-06T10:24:00.000-07:00

Code reviews are an essential part of ensuring that quality is properly built-in to the product development. However, for many it's been difficult to figure out how to make them effective without weighing down the team with more meetings, process and time it takes to prepare and conduct reviews. Especially, in a Lean and Agile product development environment.

Brad Appleton from his ACME blog has a good post today of a resource with lots of good information on this topic. Plus, if you act now you will get a book sent to you entirely free (shipping/handling included)!

Can you really plan for the long-term future?

2007-04-05T14:59:00.000-07:00

I attended a "planning conference" last night as part of my involvement in Cub Scouts with my son. The purpose of this meeting according to the leader was to "knock out the schedule" through summer of 2008 and we had the next 4 hours to do so. Initially, as we focused on the next few months things went pretty well. However, things begin to break down quickly the further out we went. One of the primary reasons is that the next scouting year would begin in the Fall of 2007 and that will cause some changes with kids, parents and leaders in that process. Therefore, it was difficult to firm up dates because the people involved by that time will be different. Yet, we were told by the leader to just give it our best shot and go on. The other reason that became frustrating is that the leader wanted particular dates and times that we would have different events. It was easy for the group to determine particular months, but much harder to nail down days and almost impossible to determine times. Why? Because who knows their own schedules a year from now (much less the next month). In the end, we created our "project plan" of activities but it left everybody a little frustrated, very tired, and most everybody skeptical that the plan would reflect reality a year from now.

I don't place blame on the leader of this because she was doing what had been done before. It was my first time, and knowing the problems I have had in the past with long term software project planning I was perhaps a little more sensitive. But, I had to ask myself, was it really worth spending hours on something that was probably only good for a few months of accuracy? Sure, there are probably some events we need to reserve well in advance because others are asking for it. However, we really didn't know if we would do these events until we get closer. There were just too many unknowns in the equation and things that could happen between now and summer of 2008. Despite our best planning, we can't really predict what will happen to the day and time. Plus, we already know that the people will change and they may want to do some things differently later on.

That's the beauty with Agile. It puts a shorter "time box" on everything. Let's focus on the next few months with a greater precision in the short term, and intentionally leave the future as fuzzy as possible. When we absolutely have to make decisions, let's make them at the last responsible moment. All the time changing the plan to better reflect the current state of things. Then, when people change (or change their minds), you can change the plan with it. I don't know what I will be doing at 10 am on June 1st of 2008 but I can give you a better idea of what I will be doing in the next six weeks.

YAGNI is very lean

2007-04-04T09:04:00.000-07:00

In Extreme Programming (XP), there is an accepted principle with the acronym of YAGNI. What is it? Here's a good summary from Wikipedia:

The YAGNI principle of software development says that when there's a good chance that "you ain't gonna need it", it's better to postpone adding a program feature.

Even when it turns out that the functionality really is needed, the habit of writing code that is not necessary at the moment has some overlooked disadvantages:
* The time spent is taken from necessary functionality.
* The new features must be debugged, documented, and supported.
* Any new feature imposes constraints on what can be done in the future, so an unnecessary feature now may prevent implementing a necessary feature later.
* Until the feature is actually needed it is not possible to define what it should do, or to test it. This often results in such features not working right even if they eventually are needed.
* It leads to code bloat; the software becomes larger and more complicated while providing no more functionality.
* Unless there are specifications and some kind of revision control, the feature will never be known to programmers who could make use of it.
* Adding the new feature will inevitably suggest other new features. The result is a snowball effect which can consume unlimited time and resources for no benefit.

This principle very much supports both the concepts of Eliminating Waste and Defer Commitment. If you are not sure that code or functionality is really needed, you need to do some more homework. This homework could be additional research, talking with customers, or just waiting until you deliver higher priority functionality to the customer and really see if they need something else. In Lean, if there isn't direct Customer Value it is considered Waste. That is a bad thing. As mentioned above, it also leads to more complexity which can impact the long-term maintainability and quality of your code. Therefore, why not keep things as simple as possible until you absolutely need to put more into it?

I have seen in many design sessions where things get into a frenzy because the developers have figured out something cool with the technology they are using and are very eager to implement their ideas. This is fine, as long as the technology and their ideas translate into something the customer really wants or needs. Too many times, customers are usually forced into having something new that the development team thought they needed only to find they hardly use it. What happens next? Well, usually both the customer and the development team live with this unused, most likely complicated functionality forever. In the end, nobody is really happy about it.

However, when you find yourself in such a meeting (and you will at some point), remind the team about YAGNI. This will be a good reminder to them about the important of value and to wait until it is the right time to implement such functionality.

Go to the Source

2007-04-03T08:51:00.000-07:00

In Lean, a key practice is one called "Go to the Gemba". Here's a brief description according to Wikipedia:

Gemba is a Japanese term meaning "the place where the truth can be found." Others may call it "the value proposition."

A Gemba visit is often simply called a customer visit. The hallmarks that make it uniquely useful are:
1) The purpose is firstly to observe, occasionally to question, rarely to guide or direct.
2) The visit occurs in the context where the product or service is used, which allows direct observation of problems that arise, workarounds that are applied, and capabilities or services that are never used.
3) Sometimes the customer (or client or user) is asked to describe what he is doing while he is doing it; this provides insight into the thought processes, which often reveal differences between the customer's mental model and the model of the developers or providers of the product or service.
4) The customer will often express wishes or needs while working in context that would be forgotten or suppressed in a different context such as a structured interview or sales meeting.

Common cases for a customer visit include:
1) Enhancing the features or usability of products (especially software) or devices (especially ones aimed at very broad or very niche consumers).
2) Improving processes or tools.

I think this describes the primary use of this practice, but I can see other uses as well. Here are some other sources that should be considered as a manager:

Individual Worker - Look at each person. Do they have a significant role? Do they understand their role? Have they developed the right skills for the role? As a manager, it is your job to ensure that each person understands their purpose in the organization. It is also your job to encourage that person to continually improve how they do their work as well as increase their value to the organization. Management should make sure that the culture is such that each individual once they understand their role, have autonomy to improve it as they see fit.

Development Team - How is the team functioning? Are the processes and tools supporting the team or getting in the way? Are they communicating well? Working well together? It is crucial that the team are working in a way that work flows smoothly, there is little or no waste and everything the team does goes towards providing value to the customer.

Management - Does your management style support Lean? Do you understand what Lean is all about? Are you establishing a culture that compliments Lean? Are you a champion of Lean? Are you helping others understand it? As a manager, you need to be the experts of Lean and do all you can to understand what it takes for an organization to become Lean. If things are falling apart because people don't understand what they are doing or have gone back to their old ways, it is up to managers to educate and instill the self-discipline each person needs to have to make it successful.

This is an area that we are still struggling with but realize that it is critical to our success moving towards a Lean organization. We must "Go to the Gemba" and find the problems (and solutions) where the root source is to properly address. Otherwise, we are making a lot of assumptions that may not prove accurate. As the old saying goes, "When you assume, you make an ass out of you and me." It's important to base decisions on real facts than assumptions. Going to the source will get you to the right place to identify the facts. It's then up to you to figure our fact from fiction.

Certification and Agile: Are We There Yet?

2007-04-02T15:53:00.000-07:00

I've always been a little skeptical around certifications. Why? Because I am not sure that the certification process is focused on what counts (experience) but what is easily obtainable (knowledge). Seems I'm not alone in that feeling. The Agile Alliance came out with their own viewpoint last week. Here's what they said:

Now that Agile software development is becoming a mainstream practice, more and more employers need to staff teams that will perform it well. How can they know if a particular person will be an asset? One way might be to favor employees who are vouched for by some certification body. It is the position of the board of the Agile Alliance that employers should have confidence only in certifications that are skill-based and difficult to achieve. We also believe that employers should not require certification of employees.

Certifications in our industry usually tell you that a person has been exposed to particular knowledge. Some certifications additionally tell you that she has passed a test on that knowledge. Knowledge is a wonderful thing, but businesses pay for performance. Performance requires skill. A skill is not as simple to acquire as knowledge: the learner has to perform the skill badly, recover from mistakes, do it a bit better, and keep repeating the whole process. Especially for the interrelated and interpersonal skills required of Agile software development, much of the learning has to take place on real projects. It is that learning that a certification should vouch for. Vouching for someone else’s skill requires close observation or questioning by someone already possessing it. For anything other than uninterestingly simple skills, that’s a lot of work–which means it’s expensive. Therefore, the only skills worth formally vouching for are those that require substantial effort to learn.

While a skill-based certification can shorten the hiring or promotion process, there are many skilled practitioners who are not certified. Excluding them from consideration would be a poor business decision. Moreover, the state of the practice moves on. Skills decay when unused. The question is not whether an applicant once possessed appropriate skill; it’s whether the applicant can do what’s required today. A certificate cannot substitute for the hard work of individual evaluation.

Certifications such as Certified Scrum Master and DSDM Foundation are knowledge-based and easy to achieve. We believe the courses that lead to them are good ones. We believe people who attend them get their money’s worth. But while the certifications may be evidence of good faith, useful knowledge, and a desire to learn, they are not in themselves evidence of skill. Higher levels of certification, such as DSDM Practitioner or Certified Scrum Practitioner, require project experience, a written project synopsis, and an oral examination. They are skill-based. Other organizations like the Agile Project Leadership Network are working on skill-based certifications. We applaud their efforts. Although, the position of Agile Alliance remains firmly that employers should not require certification of employees and that skill needs to be acquired by practice on agile projects not by training alone.

I think there is a place for certification in Agile as it has worked for other technical certification processes such as PMI, Sun, Microsoft, and others. It's good to see that Agile is maturing to a point where certification is becoming important in hiring skilled people. However, it's just too easy for anybody to become a Certified Scrummaster. Just a two-day course and viola....you are certified. This does not guarantee that you are skilled in Scrum, have experience with teams, and understand Agile principles and practices. Perhaps some of the higher levels discussed above will demonstrate those skills.

Out for a week

2007-03-23T10:44:00.000-07:00

Thanks to everyone who has been supportive and reading this blog. I will be out until Monday, April 2nd for vacation. However, I will be back and ready to provide more posts for your reading pleasure.

Games that you can plan with

2007-03-23T10:36:00.000-07:00

Mike Griffiths from Leading Answers has a great post called "Release and Iteration Planning with Innovation Games". Here is his introduction:

In this post I outline some really useful techniques for planning releases and iterations. They are adapted from a great book called “Innovation Games: Creating Breakthrough Products through Collaborative Play" by Luke Hohmann

I have never been a fan of suggesting the use of “games” in the enterprise workplace, as in XP’s “Planning Game”. The term does not sit well with some traditional-type folks; to them it sounds unprofessional and not serious enough for important work. Yet the Innovation Games described by Hohmann are high performance facilitated workshop exercises that produce great results. If the project is serious enough to engage busy stakeholders then I think we owe it to the business to use the most effective tools at our disposal. If calling them “facilitated workshop exercises” eases their acceptance then I’m all for it, because it is the results I’m really interested in, not so much what we call them.

In Luke’s book, he outlines a number of games (exercises) that can be used in a variety of settings. Some like “Design the Product Box” and “Buy a Feature” are probably familiar to many people working on agile projects, others will likely be new. To keep this post a reasonable length I will focus on adapting three exercises for use in release and iteration planning.

The three we will look at are “Remember the Future”, “Prune the Product Tree”, and “Speedboat”. I use slight variations on the last two, I call them “Shape the Product Tree” and “Sailboat” and I will explain the differences when we get to them...

Read more in his post on these three tools. Personally, I like tools like these especially for people who think more visually. Plus, by using stickers you have plenty of flexibility as you brainstorm with others and determine what the release will look like. I definitely plan on buying the book.

Agile Project Leadership Network (APLN)

2007-03-22T14:29:00.000-07:00

Leaders from the Agile community decided that something more should be said for those that manage agile and adaptive teams. Thus, Agile Project Leadership Network (APLN) was born. Here some information provided by their web site:

APLN was founded in 2004 by a group of people who are active in writing about, practicing, and evangelizing the movement towards fast, flexible, customer value driven approaches to leading projects of many types. Although this organization is separate from the Agile Alliance, our intention is to work closely with that group within the software community, but also work with people and companies outside of software and IT to help them become better Project Leaders.

This group was also responsible for their own manifesto, called the Declaration of Interdependence (DOI). A group of managers, authors, consultants, and team members from different project and product domains came together to discover common ground with respect to Agile and Adaptive Management (Similar to the Agile Manifesto meeting of 2001). This group represented people from the software development community (Alistair Cockburn, Mike Cohn, Jim Highsmith), the product development community (Preston Smith), and the general project management community (Doug DeCarlo, Robert Wysocki). Six core values emerged from that collaboration:

1) We increase return on investment by making continuous flow of value our focus.
2) We deliver reliable results by engaging customers in frequent interactions and shared ownership.
3) We expect uncertainty and manage for it through iterations, anticipation, and adaptation.
4) We unleash creativity and innovation by recognizing that individuals are the ultimate source of value, and creating an environment where they can make a difference.
5) We boost performance through group accountability for results and shared responsibility for team effectiveness.
6) We improve effectiveness and reliability through situationally specific strategies, processes and practices.

From a executive management perspective, this is great material to sell upper management on the business advantages of moving to agile or adaptive development. They think in terms of management, and understand less about the inner workings of an agile team (either because they don't relate or don't desire that level of detail). This still speaks to the core values of agile, but in terms management likes to hear (I highlighted those terms above). If you are a manager or trying to sell management on agile, check out the APLN. They have some excellent resources. If you want to attend a local chapter of APLN, you can find out if your city has one here.

What does it mean to be Lean?

2007-03-21T13:12:00.000-07:00

Note: I wrote this post originally on my first blog, Random Thoughts from a CTO, but thought it was good enough to share with the readers here.

Lean can mean different things to different people.

For some, lean can refer to how you maintain your physical shape through dieting and exercise. For those that have been successful in staying lean, they will say that it takes a paradigm shift of how you take care of your body through what you eat and how you exercise. When you start the process of becoming lean, you begin to cut out those things that are unhealthy and make you fat. You begin to eat better. You begin to feel better. You start seeing results in the mirror. You also are constantly monitoring your weight, heartbeat, blood pressure, calories, etc through the process. For some of us, myself included, we get lazy once we reach this point and start falling back to bad habits -- at first, a mistake here and there on what we eat, or missing a workout or two. Then, you stop getting on the scale and recording your results. Then, before you know it you are not lean again!

For others, lean can refer to how you maintain your financial status through budgeting and tracking your expenses. For those that have been successful in saving money and making more through investing, they will say that it takes a paradigm shift of what you do with your money and where it goes. When you start the process of saving money, you begin to determine where your expenses are going and which of those expenses you can eliminate that takes you from your goals. You also look at ways to increase your income with those savings through investments. You also look to remove your debt. You begin to save money. You begin to feel better. You start seeing results in your portfolio. You also are constantly monitoring your income flow, expenses and net worth through the process. For some of us, in this case myself not included, we get lazy once we reach this point and start falling back to bad habits of spending excess money, getting into debt, and eventually not saving money. Then, before you know it you are not financially set again!

In the software engineering industry, lean is now referring to your cycle time - how quickly you get to a working solution for the end customer without losing quality. For those that have been successful with this approach, they will say that it takes a paradigm shift of how you development and manage your software solutions. When you start the process of reducing cycle times, you begin to determine where communication and collaboration need to happen, what activities or processes take away from reducing cycle times, how we do our jobs and the way we interact with other areas. You will begin to see results. You will see the customer is more satisfied. You will find better ways to get your work done. You also are constantly monitoring your estimates, work completed, costs, quality much better. For some of us, myself a little bit, it is very easy to fall back into old habits of traditional and formal processes because you are so familiar with them and may get discouraged early on in not seeing immediate results you are hoping. You then begin to see things slip, take longer and less of a feeling of accomplishment either by the team or the end customer. Then, before you know it you are feeling that things aren't getting done as they should!

There are many other scenarios that this lean approach can take - time management, stress management, strategy, project management, the list goes on and on. What is getting in the way of your goals? How do you improve on those goals? What do you do to maintain those goals? How do you measure the process of those goals?

Focus on the goals and maintain discipline to see it through! This is what lean should mean!

The Predictability Paradox

2007-03-19T14:17:00.000-07:00

We had just several months earlier started our implementation of Scrum to all of our project teams. As we were learning what to do, more importantly we were learning things not to do. One of the things I found challenging in our adoption of Agile was that many felt that the traditional (Waterfall) way we were delivering software before was more predictable and that Scrum would be totally unpredictable given its emphasis on focusing one iteration to the next instead of heavy upfront requirements, design and planning. I knew in my heart that Waterfall really couldn't predict any better but really couldn't put my finger on why. It was then that I read this paper called "Lean Development and the Predictability Paradox" that I got my answer. It was this paper that changed my thinking and introduced the Lean concepts to apply to what I had already learned with Scrum. Here is the introduction (read more including more details on each aspect of Lean by downloading the paper here):

Wall Street has little sympathy for companies that can’t meet their forecasts every quarter. In turn, senior management expects department managers to make and meet forecasts. By the time these expectations arrive at an IT department, meeting forecasts often becomes a significant challenge. Unfortunately, it seems that a large fraction of software projects fail to deliver on their promises for one reason or another.

Why is it that software development projects seems to have so much difficulty delivering predictable outcomes? It seems that all too often projects are late or over budget or they deliver the wrong system or all of the above. How can the predictability of software development outcomes be increased? This executive report discusses a dilemma: in our zeal to improve the reliability of software development, we have institutionalized practices that decrease, rather than increase, the predictability of outcomes.

The best way to achieve predictable software development outcomes is to start early, learn constantly, commit late, and deliver fast. This may seem to cut against the grain of conventional project management practice, which is supposed to give more managed, predictable results. But predictability is a funny thing; you cannot build with confidence on a shifting foundation. The problem with conventional approaches is that they assume the foundation is firm; they have little tolerance for change.

The paradox is that trying too hard to create predictability creates opposite effect. Conventional practices are fragile in the face of change, and even in the face of learning. And yet, the more complex the system, the more necessary learning becomes. What is needed is an approach that encourages learning, and does not commit until learning is complete.

It should be obvious that decreasing the amount of speculation involved in making a decision increases predictability of the outcome. If you can make decisions based on facts rather than forecasts, you get results that are more predictable. Lean development is the art and discipline of basing commitments on facts rather than forecasts. It starts earlier, encourages change, freezes decisions later, and delivers faster than traditional practices, but nevertheless lean development produces outcomes that are more predictable.

The paradox of lean development is that you have give up some of the trappings of predictability in order to get true predictability. You have to abandon some conventional wisdom to gain the benefits of making decisions with more certainty. Fundamentally, you have to develop the capability to respond to events as they unfold, rather than hold dear the capability to orchestrate events in advance.

What part does Credibility play?

2007-03-16T10:27:00.000-07:00

In both lean and agile cultures, it is important that teams are self-organizing, self-improving, and self-managing. To be able to do that requires a certain amount of autonomy and latitude to "do the right things". This in turn requires a certain amount of trust that the right things will happen and respect that the team and individuals on that team have the right knowledge and skills to accomplish the work. All of this requires establishing credibility.

What is credibility? I see credibility as a track record. Each person can have a good track record or a bad one. A good track record is one where the person says what they are going to do (integrity), their opinions are proven through implementation (knowledgeable), and their actions are in the best interest of the company (commitment). A bad track record is one where one or more of these components are consistently being challenged by others. "Is this person representing the company?" "Do they know what they are doing?" "Will they really follow through and deliver?" If these and other questions continue to happen, this will impact the credibility of not just particular individuals but eventually the team as a whole.

Why all of this talk around credibility? Because if you want good communication between customers and the development team you need credibility. If you want the team to think in Lean concepts and perform in an Agile manner, they must have an appropriate level of credibility at some point. This isn't just an Agile or Lean thing, the same can be said about any employee, department or project team. It's just much more apparent and critical with Lean and Agile since the focus is on teamwork and the role of teams.

Whenever you have your next brainstorming or design session, look around and listen to what people are saying and how others are reacting. If you hear things that are based on little fact, make sure that work is done afterwards to experiment and prove out the theories and assumptions. If you see that others are showing through their words (and more importantly their body language) that they don't believe individuals or even worse the team, talk to those people and coach them on credibility. Make sure they see examples that hurt credibility as well as those that improve it.

As a manager, I am highly sensitive to the reputation of people, teams or the entire department. Typically, credibility and reputation go hand-in-hand. I want individuals to be successful. I want trust and respect established. I want them to have the highest integrity. Each of them should want it just as bad. Many times, they may not know why they aren't getting it. Credibility isn't something that is easily given, it must be earned. It is something that is earned over your lifetime. Think about that for a moment...

The Prime Directive of Retrospectives

2007-03-15T11:00:00.000-07:00

Retrospectives are a core element of both Agile and Lean. The need for constant improvement through feedback throughout development is essential for success. I found out that there is now a Prime Directive for Retrospectives and I really like what is said:

Holding a retrospective ritual is a very old idea. It has served the human species well, as the stories of hunts are retold around campfires and the stories of child rearing are shared as baskets are woven. It has survived this long because it works. It’s a fundamental vehicle to discover, share, and pass along the learning from experience—something we also call “wisdom.”

I have one question: Is your organization good at acquiring and using its wisdom in creating software?

Maybe it’s time to try a retrospective. It’s something that every true learning organization has as part of its culture, and it’s one of the best ways to grow—project by project—into a smarter and increasingly successful organization.

One of the most obvious fears people have when first trying a retrospective is that the ritual will become a negative gripe session, interspersed with blame and counter blame. Clearly such an event will not contribute to much learning.

The key to a constructive successful ritual is assuring that all the participants adhere to the Retrospective Prime Directive.

Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.

At the end of a project everyone knows so much more. Naturally we will discover decisions and actions we wish we could do over. This is wisdom to be celebrated, not judgment used to embarrass.

While there are many possible interesting questions to address during a retrospective, I have found four to be key to focusing a community on learning and improvement.
1) What did we do well, that if we don’t discuss we might forget?
2) What did we learn?
3) What should we do differently next time?
4) What still puzzles us?

While this can apply directly to those using Agile and Lean practices, what's great is that you can apply this technique to any form of feedback. Ask yourself these questions on tasks you perform. Ask these with your teams. Ask these with your family. All of these areas can help these relationships through communication, learning and improvement.

Transparency is the key

2007-03-14T09:03:00.000-07:00

Recently, we did a retrospective at the end of one of our iterations. As part of this process, we show a working demo of functionality that has been completed as part of that iteration. Usually, this goes pretty well but this one didn't end up as well.

During the demo, things started out smoothly. We demonstrated the functionality and everybody was pleased with the results. It was the conversation that happened afterwards where things went downhill.

Those that were representing the customer wanted to know about some functionality that had been talked about but hadn't been completed yet. The Developers assumed that the Customers were accusing the team of not delivering something yet. They got on the defensive quickly saying such things as "That functionality wasn't part of the scope of the project" and "I wasn't tasked with that work so I know nothing about those requirements" and "I don't remember those conversations" and "I thought it was part of a future project". What's was interesting is that none of these conversations were really true, but they were used as a defense mechanism. Once these things were uttered, the Customers also became defensive. "What else have you forgotten?" "Have you even thought about these things?" "Will the current solution accomodate this new functionality that is added later?" "When are these requirements being considered, if at all?"

After we ended the meeting, I met with the Project Manager/Scrum Master who had also attended the meeting to recap. She was as surprised as me to the statements that were made and was a little upset that the Developers had acted that way. I went to talk to the Customers and she went to talk to the Developers. Here's what we learned...

The Developers knew the right answers but felt they were been questioned for things that they weren't focusing on at the moment. So, they tried to do a little CYA for their lack of knowledge with the questions. Instead, they could have easier said to the Customers, "Though we have talked about that functionality, we haven't really focused on it because the functionality that you saw today needed to be completed first (because it was core and highest value). The Scrum Master could have talked about when those discussions would happen. In my talking with Customers, they knew what was being shown and understood the team was to deliver what they saw but just wanted to know how much thought had been given to the design and implementation so far to the next functionality that is being considered. It was an honest question that should have had an honest question. Instead, the Developers interpreted the question as blame for not having more to show in the demo.

I realized that there is still much work to be done to encourage transparency between Customers and Developers. It is ok to discuss roadblocks and issues between groups as they happen. Bad news shouldn't be hidden. It is ok to discuss that there are things that haven't received the focus on the team yet. It is ok to discuss changes and new ideas that weren't part of the original conversations. We should encourage feedback (good or bad). We should encourage honesty in where we are at. If this is done, it will establish better trust and respect between these groups and a healthy relationship will grow. Perhaps we are dealing with years in which Customers and Developers didn't spend much time together and the relationship was strained and distant as the result. When they got together, it was more like contract negotiation across the table than a partnership with much collaboration.

In the end, I was glad to see that this happened and know how to coach others in better communication in the future. Retrospectives are to be a checkpoint on where you are at in the delivery of work. Not just a checkpoint on what you deliver but also how you deliver it. The process is just as important as the deliverables. Both need feedback and constant improvement based on each "checkpoint". This is one of the key benefits of Agile -- to embrace changes based on constant feedback towards providing better customer and business value. How you work with others is a big part of that.

The Agile Elevator Speech

2007-03-13T13:26:00.000-07:00

I was going to also call this post, "How to talk about Agile to the non-Agile" but thought the title above was more appropriate. Joe Little from Agile and Business blog has a post was trying to figure out how to tell friends and family about what Agile and Lean are without having to spend a lot of time getting them up to speed. In his post, he ends up summarizing it as:

* Focus on delivering small slices of concrete business value frequently (1-4 weeks)
* Work with a small team of people, to enable them to be as productive as possible
* Use concrete, understandable results to allow all parties to adapt and move forward
* Arrange things so that change and learning are benefits
* Tell the truth, and make the important things as visible as possible
* Minimize waste and extra effort; maximize customer value, don't impress with hard work

I think this is an excellent introduction to both Agile and Lean that can be accomplished in a few minutes and give the listener an idea without having to know much about software development, product development or manufacturing. Read more in his post on how he led up to this list.

What kind of person do you need for Agile?

2007-03-12T09:11:00.000-07:00

What kind of people does it take to be successful with Agile? I have learned that there are some attributes that are critical for success. If you don't have many if not most of these qualities, you or your team will suffer the consequences. As you are developing your team (and perhaps hiring people outside of your company) think about these things as you are determining who will be on your Agile teams:

Team Player - Agile is focused around communication and collaboration with other team members. If you can't play well with others, you will have a difficult time. If you are uncomfortable communicating within a team, you need to get out of your comfort zone. You are not doing Agile if you sit somebody in a corner to code.

Disciplined - Many people when seeing Agile for the first time think it is very informal and therefore undisciplined. I have experienced first hand that it takes a lot of discipline. There is never really down time. With shorter iterations, every minute really counts. You must be organized with your time. You must think how to be more efficient. You must constantly be thinking about the architecture, how to test, how to redesign, integration, builds. These can happen on at least a daily basis if not more frequently. This takes discipline!

Adaptable - As Agile requires the flexibility in the process to embrace change, so should each individual have the same approach. Processes will be constantly improving. Code and design will be changing. Customers will have the opportunity to change their minds more often. If you don't like change, you will have difficulty with Agile. You must accept that change will happen and prepare for it mentally.

Takes Initiative - Agile requires that the team be aware of not only what the customer needs but always looking for ways to improve the quality of the deliverables. Creating tests, looking at design, code review, refactoring takes initiative. Each person is expected to tell the team what needs to happen and sign up for tasks along the way. That takes initiative!

Self-managed - Though most Agile teams have some kind of manager or coach present, whether that be a Project Manager, Scrum Master, or Agile Coach, the ultimate goal is for the team to be self-organizing. If management is present, it is only there as help from the team to remove impediments and may be involved to report progress to stakeholders outside of the team. The team is responsible for taking what the customer wants and delivering it in increments. They must figure out how best to work as a team, who is going to take what, and know that each person is responsible for the team's success. In order to do this, each person cannot require somebody to assign tasks and have that person check progress. They can't be micro-managed but need to manage themselves against the team.

Experienced - Though Agile can work with some members of the team being junior-level, it does require experience from most of the team. Experience will technical skills. Experience with "soft" skills. Experience with Agile. Experience with the domain (customers, industries, products, services). The team will do better with the confidence that comes with experience. It is also crucial that junior-level people get up to speed quickly so they don't slow the team down. By developing in small increments and using techniques such as pair-programming, this can make the learning curve smoother.

Customer/Business Minded - Though most Agile teams have a dedicated customer, it doesn't free the team up to not worry about what the customer wants. In fact, I would say that Agile forces you to put yourself in the customer's shoes. "Am I doing what the customer wants?" "Does what I provide help the customer's business?" "Are there ways to help the customer be more successful?" I have seen people on teams who could care less about the customer but just want to go do their work. In the long term, this really won't work with Agile. Since everything is around prioritizing the work based on customer value, you can't be effective if you aren't thinking about the customer.

Did I cover them all? What other attributes can you think of that individuals need to possess?

Challenging how some are using Scrum

2007-03-09T08:46:00.000-08:00

Tobias Meyer is challenging how some are using Scrum and has proposed some different ideas based on his experience. His post called "When Scrum is not Scrum" is bound to generate some discussion in the Scrum community. I have summarized below each of his points, followed by my own experience around each:

1. Product Owners are part of the team. Product representative (not owner) should attend the daily meetings, retrospective, planning and review meetings.

Skip: There are a couple of thoughts in this bullet. Is the term Product Owner accurate? Do they really OWN the product? I have never liked the terms Product Manager and Product Owner because I don't believe that one person truly can own or manage a software product. Especially in an Agile environment where there is such emphasis on teamwork. In the training I recently attended, a new term called Product Champion was introduced. While this person does represent customers, their main role is to provide vision, passion and support of the team as the product is being developed. The other thought is How often should this person be involved in the team. People who play the customer role typically have limited time with the team (I would love to have a full-time customer available!), so we have tried to include them in just the planning and retrospective meetings. We have had some success but have also realized that something is definitely missing. We end up spending time discussing decisions that the team made on a daily basis on behalf of the customer. Not only does this take additional time (I would consider Waste in Lean terms) but also some rework if our assumptions were wrong. Perhaps there is a point that more involvement needs to be made.

2. Two-week Sprints. Teams are incapable of planning a thirty-day sprint effectively. It is overwhelming. Most teams I have tried it nail the first part of the sprint, but allow the second part to be vague and muddy.

Skip: I totally agree. In fact, it seems that most people using Scrum are doing 2 or 3 week iterations and those that are experienced are going with 1 week iterations. We tried one month iterations and experienced the same issues. We have found that 2 weeks provide the right amount of scope balanced against the right amount of workload/urgency needed.

3. Tasks are not measured in hours. Insisting on hours breakdown and having each team member continually update hours remaining on a task is often perceived as a sneaky, micro-management approach. In my experience team members find this practice frustrating and meaningless. Long ago I moved towards encouraging team members to break all tasks down as small as possible. A task must be completed in a single working day or it is considered an impediment, and should be marked as such.

Skip: We tried at first to use points instead of hours for tasks. And by we, we are talking Management not the team. It was actually the team who felt more comfortable with looking at tasks in hours. So, at least our experience is the opposite of Tobias. I also look at tasks as a way to identify how the team is going to implement a story. Stories typically are defined between 1/2 to 3 days. We find it has been easiest many of these times to define just one task per team member who is contributing to the story because the size really is manageable. I'm just don't agree that these tasks have been completed as part of a day.

4. Use of Taskboards rather than spreadsheets. Spreadsheets hide information, and they lie. The best way to create transparency is to display everything on Big Visible Charts. The interactive, collaborative nature of taskboards lends itself to the Scrum process, like no electronic tool ever can.

Skip: We have been using an electronic solution and are finding ourselves liking the Taskboard approach better for some things and more difficult for others. It's better to ensure that the team is collaborating and taking ownership of the board. If it's in another format the responsibility has fallen more on the Scrum Master to maintain. What we have liked about the electronic format is that it is more portable (we don't have a dedicated room for each team - yet!) and it provides a good archive and easier ways to calculate burn down charts than the taskboard approach.

5. Backlogs on the wall. At least the next 3-4 sprints worth of stuff should be displayed on 3×5 cards on the wall of the team room so the team can get look-ahead time. Backlogs much longer than that, containing anything more than placeholder items (reminders) can be thought of as wasteful, in any case.

Skip: We have talked about this, and may go in this direction but again it's a matter of having dedicated space to have this much information on the walls. Having backlogs for the product has been helpful storing in an electronic format so that we have records of things that we aren't doing in the next couple of iterations but plan on looking at beyond that. We don't release every iteration, yet we also don't plan the entire release at this point. We also need a repository for discussion of future releases at least at a high-level as a roadmap of where we are taking our products. Not sure all of these things can be accomplished with backlogs on the wall.

6. Estimation Meetings. Estimation is done before the first sprint, and then on a regular basis during each sprint. I have found that a 1-2 hour meeting about 2-3 days before the end of a sprint is the ideal time. Estimates are done in Story Points, as described in Mike Cohn’s books (not “ideal developer days” as described in the Scrum books).

Skip: Totally agree but we have not been consistent with this approach. I love the idea of using points to provide relative size of features or stories against each other. Better than t-shirt sizing, but not requiring the depth and time of getting to an hour estimate especially as you are looking at the next couple of iterations. I have also liked Mike's approach of using Estimating Cards with the team as a way for the team to come up with these estimates instead of individuals.

7. Insistence on Agile Engineering practices. It is essential to introduce some core practices such as unit testing, early acceptance test specification, componentized design, continuous refactoring and pairing from the start. Scrum itself says nothing about such practices, and that tends to give organizations a sense that Scrum is easy to do, and can simply (as many of it’s proponents state) wrap around existing engineering practices.

Skip: I never really thought Scrum was to address engineering practices. I'm pretty sure Tobias isn't saying that it was, but the point that is that is all you are doing with a software product you are NOT fully taking available of the benefits that Agile can provide. We are seeing this first hand as we have much work on the engineering practices side to allow us to be more successful with our iterations. So, I TOTALLY agree with Tobias on this point.

8. The Scrum Master role is not always necessary. An effective self-organizing team is exactly that: self-organizing. Leadership emerges from the team when the key Agile principles are adhered to. The Scrum Master role becomes superfluous in a healthy Agile organization.

Skip: I have been wondering this myself. I don't know if we are experienced enough to do this on a full-time basis, but I have seen evidence that this can work. We have had our Scrum Master out for periods of time either sick, training or vacation. On a couple of times, she has been out for at least half an iteration. So you would think that the iteration would have problems, right? Wrong! The team conducts their own daily meetings and figures out how to remove impediments and keep committed to the iteration. It's not the perhaps the Scrum Master role isn't necessary, it just means that it doesn't necessarily have to come from a particular person playing the role.

List of Lean Resources

2007-03-08T13:10:00.000-08:00

In my constant search for more information on Lean and Agile, especially around Software development, I found an excellent list put together by Brad Appleton and his ACME blog. The list is broken down by books, presentations, articles and websites/blogs (mine hadn't started yet or I'm sure Brad would have included me on the list).

You can find the list here. Also, Brad has accumulated quite a blogroll on various topics (which does have my two blogs and many others that I recommend and read often).

This should keep you busy for some time. I've got some reading to do now...

Myths about Developers

2007-03-07T09:05:00.000-08:00

In our process of implementing Agile and Lean, I have realized that there are things said or thought of about developers that Agile/Lean are showing as myths. If you still believe these statements are correct (or have developers believing them as well), you need to change your thinking if you want to adopt Agile/Lean. I used to believe some of these myself until I was surprised to find something much different.

Myth 1: Developers make poor testers. We established an entire QA department based on this myth. Developers don't like to test. Developers don't know how to do it. Developers are too close to their code. Developers should not test their own work. We said these things thinking that testing was a unique skill that could only be taught to non-developer types. What we realized is that these skills can be taught to anyone but few developers really know how to think about tests. The value of QA (or other testers) is to make sure that multiple people validate the test results. Two heads are better than one.

Myth 2: Developers do not work well in teams. It is thought that most developers by nature are introverted, and would rather go off and work in the corner than interact with others. Therefore, they we thought they would have a hard time with the teamwork that is required of Agile and Lean. What we realized is that developers just want to make sure there is value working in teams and they will come out of the comfort zones if it makes sense to accomplish more as a team. I will agree that for some this is an easier process than others, but eventually I have seen all developers warm up to working with others on a team.

Myth 3: Developers are not good in front of customers. Developers are too technical for customers. They talk above their heads. They don't have great communication skills. They are too brash and abrupt. They don't really care about what the customer wants. These are things that I have heard (and perhaps even said). However, just like testing Developers just haven't received the proper training and experience to work with customers. For most, it isn't natural to them. But, if you have them working often with customers they learn quickly. The customers also learn how to adapt to working with technical people. In the process, they learn the value of some of the "technical talk". Agile/Lean forces everyone to think in terms of the customer and working more directly with customers (or customer representatives).

Myth 4: Developers only care about coding. I just want to get back to my real job. When we used to use Waterfall, I would hear these words ALL the time from developers. After spending days and days in requirements and design meetings, I couldn't really blame them. However, I knew the value that they brought to those meetings in addition to the knowledge they gained to do a better job of coding. Agile helps developers get to coding more often, which is what they love. While at the same time, they appreciate the time to understand what needs to be accomplished. It's now the time it takes to do that is in matter of minutes or hours instead of days and weeks (and sometimes months!).

Myth 5: Developers want everything figured out at once. As we first implemented Agile, I got some push back from Developers that doing things in pieces would lead to poor architectures, design, missing requirements, etc. With Waterfall, they believed that they were able to get all of those right at the beginning of each project. However, I reminded them that pretty much every project we found changing requirements which lead to changing design which lead to changing architecture. So really how valuable was it to figure everything out in the front?? With that question posed, they gave it a try. And found that they could more easily adjust architecture, design, code by working on things in increments.

Myth 6: Developers cannot self-organize or self-improve. As a manager, this is where I really learned that this was a myth. I honestly thought that project managers and myself as a functional manager needed to be there to identify where improvements needed to be made and make the team implement those improvements (whether or not they truly understand the improvements). As a result, the team became dependent on management to make things better. Those dependencies became bottlenecks, because we would try to implement big changes instead of a lot of little changes. I also thought it was management's job to help assign tasks to people and make sure those people stayed on task. Again, this caused more bottlenecks. Here, the role of management was to REMOVE bottlenecks, yet we were BECOMING the bottleneck. All because of this notion that developers needed management to do these things for them. What developers really needed was practices that provided them the focus, direction, feedback and autonomy to do their work well. They are in the best place to know how to do their jobs well, so shouldn't they be the ones who figure out how best to accomplish that? When management backs off and the teams realize that it is their responsibility to improve things, they will do so. Plus, in the process they will feel better about themselves, their role in the company and their feelings about management.

Making things more difficult than they need to be

2007-03-06T16:41:00.000-08:00

One of my IT guys told me an interesting story at lunch today.

He got an email from a person in our organization, it went something like this:

Person: "I have a problem. I have email that needs to go to another group of people after I have reviewed it. However, when I get the email it has the wrong people I want to send it to, so I need it to remove some groups or individuals and leave other groups. Please fix this for me."

IT: "What would you like me to do?"

Person: "I need a script or something that will change these email addresses from one group or individuals to another anytime these kinds of emails come up. Do you have time to do that for me?"

IT: "Is there a different way you can do it?"

Person: "If you don't have time, I understand but if you can just give me some guidance I will write the script myself."

IT: "Before we go down that road, could you give me more detail of what you want?"

Person: "Well, I receive these emails. Then I need to sometimes send these email to others groups. So, I select "Reply to All". Then I type in the new group of who I want to give it to. But, it also gives me the email addresses of everybody that saw the email beforehand."

IT: "Have you tried selecting "Forward" instead of "Reply to All"? Then you only have to specify the groups you need it to go to?"

This is a TRUE story, I'm really not making this stuff up. However, it shows how quickly we think we need to create another solution when the obvious solution is right in front of us. In this case, the person know how to forward email but had made up his mind that he was doing it the right way but the email reader wasn't working the way he expected. To fix the problem (it surely couldn't have been him, it had to be something else!), unnecessary time could have been spent creating a script.

I wonder if this happens the same way with products. We quickly expect that the products don't work and try to make them do something different as a result. We don't realize that perhaps we just don't understand how the product was designed. Once we do understand that, we may realize that the product is actually working as it should!

Anyways, I couldn't resist passing along this story!

Pay Attention to the Message behind the Manifesto

2007-03-05T11:33:00.000-08:00

I have already talked about the Agile Manifesto, but what many people don't realize is that also on the Manifesto web site there are principles behind it that were also defined by the group. I have a tendency to review these occasionally to remind me why the "founders" of Agile had in mind of what they wanted the long term goals to become. Take a look at what they wrote, I put some key phrases in bold to bring highlight to some of the language:

Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.

Welcome changing requirements, even late in
development. Agile processes harness change for
the customer's competitive advantage.

Deliver working software frequently, from a
couple of weeks to a couple of months, with a
preference to the shorter timescale.

Business people and developers must work
together daily throughout the project.

Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.

The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation.

Working software is the primary measure of progress.

Agile processes promote sustainable development.
The sponsors, developers, and users should be able
to maintain a constant pace indefinitely.

Continuous attention to technical excellence
and good design enhances agility.

Simplicity--the art of maximizing the amount
of work not done--is essential.

The best architectures, requirements, and designs
emerge from self-organizing teams.

At regular intervals, the team reflects on how
to become more effective, then tunes and adjusts
its behavior accordingly.

Interview with the Poppendiecks

2007-03-02T08:39:00.000-08:00

InfoQ just recently conducted a video interview with the Poppendiecks. If you don't know who they are, here is an introduction:

Mary and Tom Poppendieck www.poppendieck.com teach and consult worldwide on Lean principles for software. Their practical, customer-focused approach to software development identifies real business value and enables product teams to realize that value. Mary has managed solutions for both operations and new product development. Tom is an enterprise analyst, architect, and agile process mentor.

Lean software gurus Mary and Tom Poppendieck share their years of practical experience, as they speak on the history of Lean thinking, the value of fast delivery and deferred committment, their use of Value Stream Mapping to identify and reduce waste, the importance of identifying and dealing well with cross-organizational and inter-organizational boundaries, and how Lean relates to RUP and Scrum.

They cover a wide variety of topics in this Question and Answer session. Everybody should get something new from it. You can find the video here.

What Value should be

2007-03-01T09:34:00.000-08:00

A common theme that you will see throughout Agile and Lean is this concept of customer value. The order that you do the work should reflect the highest value to the customer. This really isn't a new idea, I have known over the last 20 years that I need to produce something that the customer wants. However, the problem is that even though I knew that was the right thing to do, I didn't always think about my work in the shoes of the customer. As a developer, I remember thinking that if I put this cool new function in using some cool new technology that the customer would love it. I mean why wouldn't they, I thought. But many times I found that by not checking with the customer I wasted time implementing something they didn't really want or ask for. This resulted in either rework or leaving in functionality that was never or rarely used. Neither of those scenarios are good, because I could be spending my time on things that the customer really wants. If only I had asked them (or somebody that represented them)...

So, with Agile and Lean, this is one of the "prime directives" - to make sure that what you are doing has value to the customer. This is a good thing, a REALLY good thing. You need to have a customer representative always available to the team. Not just when they determine requirements, but through the whole process - design, testing, and especially demos of the working software as it is being developed iteratively. It's all about feedback and checkpoints along the way that you aren't wasting your time. In the end, it translates to functionality the customer will use more often. This translates obviously to happy customers, which translates to good word of mouth, which translates to more customers...well, you see the point.

But since we have adopted Agile (and looking at Lean), there has been a little problem with this. You see, we have over 2500 customers that are using our products. While it is good to know what they want in the product, EVERY customer may have different ideas of what is important to them. So, isn't this customer value? Shouldn't we be doing the things that matter to them? Of course, but there are too many things coming through the pipeline and it's difficult to know which customer request is more important than others. So, what have we done? We prioritize by the size of our customers. After all, they represent a larger percentage of our customer base right? While that's true, it doesn't give the little guy much of a chance to offer good ideas. And, those ideas might be the best thing for all other customers.

So, while sitting in the Lean class this week I realized that we are focusing on only a small part of value - Customers. What we need to focus on is Business Value. What is Business Value? Well, customer value is definitely part of the equation but it also looks at what is good for the overall business. Does this request support the majority of our customers? Does it support our long-term goals? Do we have the capacity (people, skills) to fulfill the request? Does it fit our business model? Does it align with where we are taking our products? Is it something that our company should do? I believe that if we ask those questions we will have a much better idea of how to prioritize requests as well as know which requests we should take on. How do we measure that? Return on Investment (ROI)!

Innovation and Corporate Culture - Part 2

2007-02-28T12:33:00.000-08:00

In Part 1, I promised I would get back to you on the results of a meeting I attended with the Oregon Innovator's Forum. As mentioned, the topic was "Innovation, Corporate Culture, & Getting Out of Our Own Way". Here is just some of the discussion that took place (thanks to Jon Marshall for providing the notes):

"Silos are a huge factor in this conflict of culture vs. innovation. I’m sure that silos must serve some function or they would have collapsed a long time ago. Is change happening in the appropriate place in the organization, and if not, how do we enlighten the organization or help the organization understand the need for the change to happen across the organizations?"

"Our company went from a startup to having silos. My perspective is that at some point in time, you can’t play that role of doing everything – there’s just too much going on. There are functions in the business and silos serve to organize and encapsulate these functions."

"I think there is specialization that you need to have. But what gets lost is that people lose the perspective that there is a cross functional aspect that you can’t have silos for . So we have to have a function that makes sure the right people continue to get together so that we can make the right decisions. I don’t know how you do it in a larger organization: we’re struggling with 100 employees. We have to remember to go back to not just saying IT will handle it because they’re the technology part, but that all of the employees can be a part of INNOVATING THE COMPANY."

"Have you had experience with that cross-functional piece?"

"When I came on board, the software projects were my department – we were the product development team. Yet we needed marketing involvement, we needed sales involvement, we needed technical support, we really needed every other area of the company to be a part of the product development process. What we found is maybe we did a really good job of releasing our product, but then the other pieces weren’t ready, we had some disconnects and things really weren’t working well. What we decided to do was we transformed our projects to work across the organization. That works for us because we have a cross functional team that’s not an outrageous size. This really worked because it broke down a lot of the barriers."

"So is there some generalization from that experience that you would offer to us?"

"I think innovation has to ultimately become an organizational cultural attribute. It’s not just IT or marketing, but the whole company has to be involved in creating innovation. The companies that have discovered that are successful at doing it."

"I’m interested in physics and in cosmology, the theory is the universe is expanding increasingly till it starts to rip into parts.The big rip. I think this is a great metaphor for what happens in companies as they grow. When you’re small, the average distance between any two employees is relatively small. As the company grows, if no one is paying attention, eventually you will form silos as essentially different pieces rip off of the whole. Like the model of the BIG RIP of the universe where communication ceases between the parts, the silos rip out of the whole and effective communication stops in many key areas. No one has to do anything to cause this, it is just a manifestation of growth where it becomes too much trouble to try to communicate across an increasing distance. When communication stops, the other parties essentially cease to exist for any given silo. One of the things I’ve been involved with is to repair these rips in the fabric of companies. Eg, the quality assurance department is not talking to the development department any more. They only see each other at the weekly status meeting. You always hear excuses on why that is. Too hard, too many meetings, too much work, too few people on our team.
But in reality, the communication in lining up the departments is almost THE most important thing. It doesn’t matter if your team comes in early or late if no one else knows what’s going on. Everyone else needs to be flexible to your improvements or your failures. Without that your going to fail as a team because someone in the fabric is going to fail. But if your organization remains in good communication and when one area starts to fail another group can pick up the slack, then you can survive the inevitable local failures."

"About the project team BEING the cross functional facilitating entity. Apparently companies are using project teams to be the thing that carries information across company silos and serves as an integrating force. But as you both were talking, it was occurring to me to suspect that we don’t have the right teams in a lot of cases. In other words, we’ve taken the minimum number of people we think needs to be on that team to get the project out the door, not realizing that secondary function of going across the silos; it may be that we need to significantly broaden out the teams.
One of the things we’re struggling with now is sales says “we’d like to help, but I need to get back to my job”. What we’re finding is that maybe we have to redefine what the sales position is."

These were some of the things said in the meeting around the challenges of innovating in the corporate culture.

By the end of the session, we realized that you cannot force culture to change. However, that doesn't mean that cultures can NEVER change. It just takes time. Where do you start? Changing the language. Telling stories. Getting people on board through small wins. It may take some time but before you know it the culture will change and people may not even realize it in the process. Get people to talk about innovation. Educate people on it.

By the way, if you are in the Portland area and are interested in attending the next session, please send me an email. The next session is scheduled for April 18th. I will be the moderator for this session (lucky me!). The topic will be on "Tools and Techniques that Drive Innovation". For more information, go here.

Going to a Lean Development Course - Update!

2007-02-23T09:25:00.000-08:00

Next week, I will be attending a 2 day course in Seattle on Lean Software Development. The course is being conducted by Net Objectives and led by Alan Shalloway. I have enjoyed reading Alan's contribution to the Lean Development Yahoo Group and can't wait to learn more. Here is a synopsis of the class taken from their website (you can learn more by going here):

The software industry is looking for ways create quality code in an effective, efficient manner that is rapid, repeatable and reliable. Attempts such as CMMI and Agile methods have found some success but also seem to have some inherent drawbacks.

Of the many methods that have arisen to improve software development, Lean Software Development is emerging as one that is grounded in decades of work understanding how to make processes better. Lean thinking focuses on giving customers what they want, when and where they want it. It provides a way to maximize value while minimizing waste.

The varied background and history of Lean makes it very valuable to practitioners. However, the varied perspectives available from which to look at Lean sometimes makes it a challenge to learn how to apply it. This course takes four prominent perspectives of Lean and integrates them, creating a consistent, comprehensive view of how Lean can be applied to software development. These perspectives are:

Lean Manufacturing from Toyota (from Taiichi Ohno of Toyota)
Lean Thinking (from Womack and Jones’ work)
Lean Software Development (from Mary and Tom Poppendieck)
Lean Product Development System (from Toyota, as described by Michael Kennedy)

The course centers on how to create a fast, flexible flow of customer value-add. Principles of Lean that facilitate this are:

Eliminate Waste
Create knowledge
Respect people
Build quality in
Defer commitment
Deliver fast
Optimize the whole

The course teaches how to manifest these principles within the context of software development as product development. This is one of the distinctions of this course over mere agile training. Agile project management focuses on managing a single project and perhaps coordinating several projects together. However, true software development should be focused on the products these projects relate to. Selecting products, scheduling projects for the products, balancing product loads, are business perspectives that Lean addresses while agile methods do not. The focus of this course is on the mindset of Lean and the Lean-Agile Software Development process. The issues of architecture and how to evolve designs is only dealt with at a superficial level.

P.S. I found out about Net Objectives through subscribing to their Lean-Agile Straight Talk podcast. If you want to learn more about how Lean and Agile work together, visit their site. But just promise to come back here to learn more!

Update: If you are trying to implement Agile or Lean in your organization, you NEED this course! Though there are plenty of books on the topics, Alan did a great job of blending various experts and ideas out there and be able to understand the big picture. This isn't your typical training course however. We were able to have a lot of discussion among the group about our own experiences so far. Alan, being an Agile/Lean coach as well as trainer, provided a lot of good ideas for us to take back. It was like getting free consulting/coaching along with the cost of the course (which, by the way, at $1000 is a fantastic bargain). Check their web site for more details on the next courses as well as other courses in the area of Agile.

Values, Principles and Practices

2007-02-21T17:23:00.000-08:00

When I was creating this blog, I wanted to make sure I had a tag line that accurately reflected what the focus and goals of this blog were to be. Here's what I came up with:

A manager's quest to improve his organization through Lean and Agile development values, principles and practices.

Technoetic had a post awhile back that I found to be a great representation of the overall goals - values, principles and practices. While some use these terms somewhat interchangably, I truly believed that each of them represented a particular piece of the whole. The author of this post apparently agreed with me and came up with perfect definitions of each of the parts. Here they are:

Values - A description of preference between alternatives. Often the alternatives represent candidate courses of action and the value guides the selection among the alternatives. Sometimes values are basically axioms. They can be irrational in the sense that we might not be able to explain them intellectually or they might be primarily based on emotions.

Principles - Statements describing a model. A scientific principle is a statement describing an aspect of physical reality. Principles might also describe a model of social activity, for example. Principles can be fundamental or logically derived from fundamental principles. It’s commonly believed that principles are derived from values, but sometimes people have certain values because of specific principles. For example, the “value” of communication can be derived from the principle that communication improves cooperation and coordination effectiveness. This circularity between values and principles can be confusing at times.

Practices - A pattern of activity that can be repeated to achieve goals within a context. A person defining a practice is applying principles within a framework of their values (preferences). A related collection of practices might be called a process or methodology.

To make it even simpler, I could summarize the difference between these as:

Values are the feelings of what the right things are for us to do.

Principles are the guidelines of how we are to demonstrate those values.

Practices are the actions of what we do to support those principles.

As I write future posts, I will refer to these terms frequently to describe the parts of both Lean and Agile. Before I went there, I thought it would be good for us to have the same context going forward.

Innovation and Corporate Culture - Part 1

2007-02-20T12:10:00.000-08:00

This afternoon, I have been asked to join a group of 15 other innovators in Oregon to meet as part of a monthly Oregon Innovator's Forum. While I am honored to play a part, I am the first to admit that I'm not one of the top innovators in Oregon. That doesn't mean I don't have a lot to talk about in the area of innovation and my experiences around it.

I call this Part 1 because I hope to gain a lot of knowledge from the meeting to share back to this group (which of course will be Part 2). I believe that innovation can play a significant part in making Agile and especially Lean successful.

The topic for this session is called "Innovation, Corporate Culture, & Getting Out of Our Own Way" Here is a description of what will be talked about:

Have you noticed roadblocks to innovation that defy remediation?

Are you working around counterproductive business practices or systems?

Do you find that you can’t even talk about these roadblocks and awkward practices because the topics are taboo?

Then you’ve probably stumbled upon some immutable corporate cultural icons!

Please join other Oregon innovators for an informal discussion on how we can evolve the seemingly inflexible aspects of managing businesses so they better support the actual results the business hopes to achieve. The process of defining these stubbornly unyielding business practices can be difficult because your culture, like the water around a fish, is invisible. As participants in corporate culture, we do something a certain way because we think it is the only way, the right way, or at least the familiar way. These practices and systems then resist change when the business climate demands innovation.

Cultural inertia resists the changes that innovators attempt to invoke.

It's true that cultural inertia can provide certain advantages: No one wants to try to survive in a culture that is constantly changing—it would be too stressful and confusing. But here's the problem: when the “pain” associated with the current status quo becomes great enough, cultures look to their innovators for ideas on how to improve the situation. However, this hope for positive change often comes with an unconscious expectation that the “improvements” will only require minimum alterations to what we value in the cultural status quo: A “you can’t change that!” mentality.

Corporate culture thrives on the myth that issues which affect the entire company can be “thrown over a wall” to be fielded by people who will somehow handle the details by “fixing” things only on their side of the wall, with small impact anywhere else in the company. Unfortunately, that rarely works. One example where it doesn’t work is in the area of incentives for project teams. There is a growing body of evidence which indicates that providing incentives to teams produces more manageable and successful projects than providing incentives either to individuals or managers. But is your corporate culture is ready to hear that??

How do we turn this cultural Titanic around?

What are some corporate cultural icons you’ve bumped up against in the course of your travels and how did they need to change? How have you dealt with them, or how do you wish you could have dealt with them? How do we get out of our own way?

Bring your victories and war stories and let’s see if together we can shine a light on some of those change-resistant areas of corporate culture which might be blocking our paths to innovation.

I would like to hear from my readers on this topic. Please share via comments either your victories or war stories and I'll compare it to our discussion later today.

An Agile Approach to adopting Agile

2007-02-19T09:57:00.000-08:00

Several years ago, we started our quest towards a better way to develop software. In the process I discovered Extreme Programming or better known as XP. Though I liked many of the concepts that I was learning about, I definitely had my reservations on parts of it. However, the largest reservation I had was that it seemed that the founders and proponents of XP were requiring that this methodology was an "all or nothing" type of approach. If you didn't incorporate all of XP, you really would not benefit from the results that others have had with XP. In fact, some would say that you might actually experience worse results than Waterfall with a partial implementation.

This worried me greatly because I knew that past attempts of trying to implement an entire methodology had failed. Why? Too much to take on too fast. What I found interesting is at least for XP they were taking a Waterfall approach to adopting Agile. Instead, why not take an incremental and evolutionary approach to the adoption? Start with the highest priority items, incorporate them, get feedback on their adoption, and made improvements as needed. Continue this process at a pace that your organization can handle. Ultimately deciding how much agile is enough for your organization at any point of time.

At the same time this was going on, I had an interesting discussion with a colleague. He had told me he hated the idea of methodologies. Why? Because it makes the assumption that you can go buy a tool belt full of tools and assumes that you will not only use all of the tools but will know which tool should be used for which purpose. His approach? Find the tools you need that resolves the issues you need when you need them. To better find the tools you do need to spend the time understanding what tools are available and how to use them. However, it is important to note that knowing and using are two different things. Just carry the minimal tools that you need to get the job done. His approach was much more Agile in my opinion and one based on adopting a set of Best Practices than adopting entire Methodologies.

I eventually found the answer to our problems. One of the "founders" of Agile, Alistair Cockburn, had a "methodology" called Crystal Clear. Unlike others, it really wasn't a methodology but a discussion on what practices are available from XP, Scrum, FDD, and others. He truly advocated the "cut-and-paste" approach, find something that is close to what you need and put it into your own processes. Then, make it your own! He also advocated the incremental approach, at least start with SOMETHING than it will take a life of its own. He focused more on educating the reasons behind each of the practices, and being able to sell it to others through your own adoption.

This approach allowed us to explore all other methodologies as Best Practices to find the right set of tools for our tool belt at a pace where it can be learned and truly understood. If you are struggling with Agile, chances are you are going too fast and taking on too much. Slow it down and take the Agile approach. Do only the minimum of what it takes to address the highest priority issues you are dealing with. Then tackle the next one, and the next one, and so forth.

What's funny is that the founders of each of these methodologies took the same approach. They started out with something, then realized that something else was needed until they came out with a set of "somethings" that worked for their particular situation. They found a pattern as they continue to implement in different companies, and thus they developed their particular methodology. They took the incremental and evolutionary approach. They customized it through constant feedback and changes to make it work for them - for their cultures, their people, and the technologies and products they were developing. You have to do the same thing in your journey towards adoption!

Though this approach, we have grown to appreciate what each of the methodologies bring to the table. However, we had to learn what we were missing to truly appreciate what we could get with each of the Best Practices. In other words, we had to realize that we needed a particular tool to accomplish the job much better than the tools that we currently had in our possession. Bottom line, most implementation of processes fail because of lack of understanding. Either understanding of why they are valuable or understanding of how they can be used for your particular needs. If you don't know how to use a hammer, and you don't know why you would use a hammer, you probably aren't going to find the hammer that useful. Think about that for a moment!

The Anatomy of a Manifesto

2007-02-16T09:52:00.000-08:00

Though different forms of Agile processes have been around for many years, they lacked a common voice. There was even a debate on what to call these various methodologies - iterative? lightweight? evolutionary? agile? Many of the founders of these methodologies realized this several years ago and set out to settle differences and go forward in the same direction. Thus, the Agile Manifesto was born. According to the website, here is some of the history of that event:

On February 11-13, 2001, at The Lodge at Snowbird ski resort in the Wasatch mountains of Utah, seventeen people met to talk, ski, relax, and try to find common ground and of course, to eat. What emerged was the Agile Manifesto. Representatives from Extreme Programming, SCRUM, DSDM, Adaptive Software Development, Crystal, Feature-Driven Development, Pragmatic Programming, and others sympathetic to the need for an alternative to documentation driven, heavyweight software development processes convened.

While the Manifesto provides some specific ideas, there is a deeper theme that drives many, but not all, to be sure, members of the alliance. At the close of the two-day meeting, Bob Martin joked that he was about to make a "mushy" statement. But while tinged with humor, few disagreed with Bob's sentiments that we all felt privileged to work with a group of people who held a set of compatible values, a set of values based on trust and respect for each other and promoting organizational models based on people, collaboration, and building the types of organizational communities in which we would want to work. At the core, I believe Agile Methodologists are really about "mushy" stuff; about delivering good products to customers by operating in an environment that does more than talk about "people as our most important asset" but actually "acts" as if people were the most important, and lose the word "asset". So in the final analysis, the meteoric rise of interest in and sometimes tremendous criticism of Agile Methodologies is about the mushy stuff of values and culture.

So, let's look at each part of the Manifesto in more detail. (You can read the entire Agile Manifesto and other related information at agilemanifesto.org).

We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:

Just like the act of creating software is a discovery process, so is figuring out how best to develop that software in the first place. Agile recognizes that there is a constant need for improvement - not just in the software itself, but especially in how you get there. Through constant feedback from the team, and acting on that feedback, the team can own the process and make it better over time. This really goes against conventional thinking that management should control those processes to ensure that others follow them. The team owns how they do the work, the customer owns what functionality they want produced, and management owns how are we going to get there.

Individuals and interactions over processes and tools

There was a discovery that processes were becoming too heavy and that people were hiding behind those processes and the tools to manage those processes. Contrary to what some have thought about Agile, processes and tools in some form are used by all Agile teams. It's important to note that those should be in place only if they help the team produce better software more efficiently. In other words, they help you do your job better and with higher quality than without it. However, NO tool or processes should keep people from talking and working together. Project managers can't hide behind Gantt charts, they need to work with the team and talk about the progress of the project. Developers can't hide behind their tools as well, such as email, when their other team members are within distance to talk about things. The fastest and most effective form of communication is face-to-face, yet people tend to forget about it and use less effective forms.

Working software over comprehensive documentation

The goal for any software development should be to get the product out to customers as quickly as possible assuming that the level of quality meets standards. Just like tools and processes, if you are spending too much time on documentation when you could have the end product "speak for itself" than you are not meeting that goal. The code should be the documentation as much as possible. Conventional wisdom would say that customers should review and sign off of designs before product is developed, so that time is not wasted on being product the customer does not want. However, customers don't always know what they want until they see it. Paper cannot take the place of something more tangible. Once they see it, customers want to play with it, mold it and make it their own. Therefore, release often and early and provide the ability to respond to the customer using the working software product as the tool. We aren't talking throwaway prototypes here or "smoke and mirrors", we are talking software that works for the functionality that has been provided at the time.

Customer collaboration over contract negotiation

Think about past projects that you may have been involved in. How was the relationship with the customer? How much were they really involved throughout the project? Too much? Too little? Wish they were there? Wish they weren't? Most people would say that the relationships have usually been strained at best. Traditionally, most customers are involved primarily at the start and end of the project. At the start, they are asked to provide specs whether or not they understand software and how something should or could work. They are asked to provide EVERYTHING they can think of because the scope will be frozen for the remainder of the project. They are asked to SIGN IN BLOOD before the team will start development of the product. They are asked to refrain from any changes because they will cost the team especially late in the project. However, that is when most customers finally get a chance to see the real results of the project. Because many of these projects are already late, the customer "lives" with the end result most of the time (even though they know it isn't meeting their needs). Ultimately, either functionality is never or seldom used or there are future projects to redo a major portion of the developed product. And the vicious cycle continues...

Would it be better if the customer felt part of the team instead of on the other side of the table? That they felt they were being heard and were allowed to react to the product while it is being developed? Wouldn't it be the best interest of the customer to know what is going on since they are investing a lot of money into the effort? Wouldn't the team want to make sure that they are producing product that the customer will REALLY use and meets their needs? Of course!

Responding to change over following a plan

If we are to let the customer "mold" the product, we need to allow changes naturally to happen. If the customer wants something, why shouldn't we provide it to them? After all, they are buying our product! If we aren't providing something of value to them, why would we even work on those things in the first place. Traditional thinking says it is important to stay on time and within budget. Agile thinking manages the scope of the project and takes time and budget out the equation. If you are building the highest priority things for the customer, let them decide when the product is ready to be used. Why should we decide it for them?

That is, while there is value in the items on
the right, we value the items on the left more.

Critics seem to miss this last statement, and it's a VERY important one! Agile isn't saying that documentation, processes, tools, and planning is bad. If fact, you will find a little of all of these things on a typical Agile project team. However, it is recommending that a little goes a long way and not to lose site of the things that are most important to be successful (which are the items on the left). Remember to do the things on the left first, then make sure the things on the right are necessary to further support the left.

Video introduction to Lean

2007-02-15T08:52:00.000-08:00

The following video was presented by Mary Poppendieck on December 2006 at Google. It is called "Competing on the Basis of Speed". She presents many of the Lean concepts as it applies to software. I will dive into how I got into Lean thinking later, but thought this would be a good starting place for those unfamiliar with the concepts.

Are you ready for Agile?

2007-02-14T12:01:00.000-08:00

Paul Klipp, a guest author at Agile Advice (one of my favorite blogs on Agile), poses a question to those thinking about Agile with his post entitled, "To be or not be Agile". Here's his introduction:

At it's core, an agile process is designed to address a long-standing problem with traditional development methods: scope-creep. Most traditional processes begin with a thorough description of the desired product and then code until it's done. The weakness of these approaches is that in the event of a change in the business need or a reevaluation of the plan, much work can be lost and deadlines can easily slip out of control, and costs with them. The traditional way to address this problem is with change documents. A change document basically is a way of telling the customer what it will cost to make a change to the plan after development is underway. Agile processes are designed to do away with the cost of change so that the client is free to evolve the system under construction toward the ideal end goal, even if it is a moving target.

He further gets down the heart and soul of agile with this statement:

The beauty of iterative development combined with continuous integration is that if we approach a piece of software with the intention of working until all features are done (traditional approach), then at no point in the project is there usable code except at the end. Whereas with iterative development, the client can pull the plug at any time and have a working product, even if not fully-featured. For example, halfway through a large, waterfall or RAD (Rapid Application Development) style project we might have had a working administrative interface but not working user interface, or no user authentication system; in other words, a fundamentally flawed and unusable product. Halfway through an agile Project (at the end of any iteration) we have a product which, though lacking some intended features, actually had all the essential components of a software tool finished, tested, and ready to deploy if desired. The customer is not married to the project and the developers can't hold the project hostage until the bitter end; If it concludes early, the investment is not a sunk cost.

So, are you ready to be agile or not? Read more of his post to decide.

When Agile got its legs

2007-02-13T14:26:00.000-08:00

If you work within the software industry, you are probably familiar with Agile and many of you may be developing this way. If you aren't, you may see this as a lean approach that could be translated into other industries. Before I get into what this type of software development looks like, let's discuss what has been traditionally been the way to develop software - Waterfall.

Waterfall software development was named as such because work is performed in phases, with each phase needing to be completed before the next phase started. People will call these phases different things, but essentially they are the following:

Research
------> Plan
----------> Design
----------------> Implement
-------------------------> Test
------------------------------> Release
-------------------------------------> Review

Each of these phases would concentrate on the entire scope of the project, and people could not move to the next phase until the previous phase was completed. Because of this, an emphasis on formal documentation was needed due to the amount of scope going through each phase and the fact that team members may not have participated in the previous phase and need information to start their part.

End users and other primary stakeholder are usually involved in the first couple of phases, then would not see the end result until it was ready for release. The review is usually some kind of project post-mortem with lessons learned. Depending on the overall timeframe of the project, people that participate in this review may have been on the project months ago, and many people cannot remember what the start of the project was like.

Several years ago, people started to get together and determined that there must be a better way. They also started their own "lightweight" or "agile" process that suited their environment. One of the better known methodologies out there is XP or Extreme Programming.

When I started seriously thinking about a different way for my organization to develop, my research quickly sent me to the Agile Alliance. What is the Agile Alliance? Here is what the site says:

In the late 1990's several methodologies began to get increasing public attention. Each had a different combination of old ideas, new ideas, and transmuted old ideas. But they all emphasized close collaboration between the programmer team and business experts; face-to-face communication (as more efficient than written documentation); frequent delivery of new deployable business value; tight, self-organizing teams; and ways to craft the code and the team such that the inevitable requirements churn was not a crisis.

Early 2001 saw a workshop in Snowbird, Utah, USA, where various originators and practitioners of these methodologies met to figure out just what it was they had in common. They picked the word "agile" for an umbrella term and crafted the Manifesto for Agile Software Development, whose most important part was a statement of shared development values.

The Manifesto struck a chord, and it led to many new Agile projects being started. As with any human endeavor, some succeeded and some failed. But what was striking about the successes was how much both the business people and the technical people loved their project. This was the way they wanted software development done. Successful projects spawned enthusiasts.

The Agile Alliance exists to help more Agile projects succeed and to help the enthusiasts start more Agile projects. This particular page is to help people learn more about Agile software development. In keeping with the Agile emphasis on face-to-face communication, we urge you to visit a users group and talk to your peers about their experience.

With this in hand, I was ready to learn more about the details of what it meant to be Agile...

Re-thinking what makes a project successful

2007-02-12T09:49:00.000-08:00

In 1994, The Standish Group published a study called the CHAOS Report. This study was conducted to focus on what causes projects to not be "successful". Success being defined as on-time and on-budget. I have seen this report used by many Agile practitioners as a basis on why traditional waterfall develop projects do not work. Now that I have been using Agile practices for some time, I thought I would go back and review the report first-hand to get some insights.

I found the Introduction interesting, here is what it said:

In 1986, Alfred Spector, president of Transarc Corporation, co-authored a paper comparing bridge building to software development. The premise: Bridges are normally built on-time, on-budget, and do not fall down. On the other hand, software never comes in on-time or on-budget. In addition, it always breaks down. (Nevertheless, bridge building did not always have such a stellar record. Many bridge building projects overshot their estimates, time frames, and some even fell down.)

One of the biggest reasons bridges come in on-time, on-budget and do not fall down is because of the extreme detail of design. The design is frozen and the contractor has little flexibility in changing the specifications. However, in today's fast moving business environment, a frozen design does not accommodate changes in the business practices. Therefore a more flexible model must be used. This could be and has been used as a rationale for development failure.

But there is another difference between software failures and bridge failures, beside 3,000 years of experience. When a bridge falls down, it is investigated and a report is written on the cause of the failure. This is not so in the computer industry where failures are covered up, ignored, and/or rationalized. As a result, we keep making the same mistakes over and over again.

I find it interesting that in my many years of doing software projects, that there are always project stakeholders who view projects this way and compare these kinds of projects to either building bridges or houses. However, for the reasons mentioned above, this assumes that once completed the software will last forever (or at least a very long time). This kind of thinking doesn't handle change well. It assumes that you have a control over the future and can predict change. It assumes by restricting change you will have success. So, was that true? Here's what they found:

In the United States, we spend more than $250 billion each year on IT application development of approximately 175,000 projects. The average cost of a development project for a large company is $2,322,000; for a medium company, it is $1,331,000; and for a small company, it is $434,000. A great many of these projects will fail. Software development projects are in chaos, and we can no longer imitate the three monkeys -- hear no failures, see no failures, speak no failures.

The Standish Group research shows a staggering 31.1% of projects will be canceled before they ever get completed. Further results indicate 52.7% of projects will cost 189% of their original estimates. The cost of these failures and overruns are just the tip of the proverbial iceberg. The lost opportunity costs are not measurable, but could easily be in the trillions of dollars. One just has to look to the City of Denver to realize the extent of this problem. The failure to produce reliable software to handle baggage at the new Denver airport is costing the city $1.1 million per day.

Based on this research, The Standish Group estimates that in 1995 American companies and government agencies will spend $81 billion for canceled software projects. These same organizations will pay an additional $59 billion for software projects that will be completed, but will exceed their original time estimates. Risk is always a factor when pushing the technology envelope, but many of these projects were as mundane as a drivers license database, a new accounting package, or an order entry system.

On the success side, the average is only 16.2% for software projects that are completed on-time and on-budget. In the larger companies, the news is even worse: only 9% of their projects come in on-time and on-budget. And, even when these projects are completed, many are no more than a mere shadow of their original specification requirements. Projects completed by the largest American companies have only approximately 42% of the originally-proposed features and functions. Smaller companies do much better. A total of 78.4% of their software projects will get deployed with at least 74.2% of their original features and functions.

This data may seem disheartening, and in fact, 48% of the IT executives in our research sample feel that there are more failures currently than just five years ago. The good news is that over 50% feel there are fewer or the same number of failures today than there were five and ten years ago.

Would you call that success? Here we are 12 years later, yet we hear or have experienced projects that took longer, cost more, and didn't meet customers' needs. Those doing Agile have come up with a different definition of project success - "Continually producing increments of working software with the customer that is prioritized based on highest value for the cost (ROI) all while improving the software and how you produce it over time through constant customer and team feedback." Success if defined by meeting or exceeding customer expectations early and often, and not necessarily on the costs it will take to get there.

Read more of the report for yourself here.

Early observations around projects

2007-02-09T08:33:00.000-08:00

As my previous post may have indicated, we became focused on heavy processes around software development and project management. We tried to manage these processes around tools. We did have moderate success around projects. However, we also had some fantastic failures. Here were some early observations that I had with those projects. At the time, I didn't know how to resolve them but I was able to find some interesting patterns. Here is the list of my top 10:

1) Traditional Waterfall assumes that you can figure out as much as possible early in the process. What's ironic is that we also understood the Cone of Uncertainty, which states that the reliability of estimates will become greater as you move through the process. We learned that we couldn't plan or predict the unexpected. We could prepare somewhat through risk management but not necessarily prevent things from happening.

2) Shorter range projects (less than 6 months) were more successful in meeting deadlines than larger range projects. It seemed the larger the project the harder it was to stay on track.

3) Projects that involved internal resources outside of the Development group were more successful than projects that didn't. These resources included Sales, Marketing, Technical Support, Customer Training, etc.

4) Projects that involved some interaction with outside customers were more successful than solely internal representatives.

5) Changes were harder and more resistant to make later in the process because it meant reinvesting a lot of time going upstream in the process. The process assumed that you wouldn't have to do a lot of that. Because of this, we would deliver functionality that the customer really wanted something different with the promise that we would look at it in the future (but rarely did).

6) Because of that, customers and/or customer representatives would ask for anything they could think of whether or not they really needed it because there was no other opportunity to do so.

7) Most of the projects involved managers who felt they had to have control over the project - control over what each resources worked on, control over what the customers could have, control over what stakeholders should know. However, on those rare projects where the manager was more transparent, more accommodating, and empowered the team to determine how they do the work and what work they should do; we found that everybody was happier, more motivated and produced better results.

8) Tools to manage projects became harder to use because it assumed that the work breakdowns, dependencies, resources allocated would hardly change and that the only thing that would change is time. That was rarely the case, and project managers found themselves managing the tool instead of the project itself.

9) It was assumed that estimates were reliable from those that provided it. Therefore, all milestones were expected to be hit by the stakeholders even though how we had figured out the milestones was very uncertain. Projects that were similar to those in the past fared better than those that were different. Most projects fell into the latter category and involved either domain or technical experience that was new to at least one member of the team at any time.

10) Though we had documentation for things such as requirement specification, technical design, etc, we found that as the process moved along those didn't reflect reality. It became too time consuming for people to go back to the documents. Therefore, the working solution and code behind it were the only thing that people could count on. We had thought that documentation were be used as a reference for future projects, but instead weren't used. Also, the documentation wasn't always developed by the team but many times passed around. Therefore, there were many disconnects because of misinterpretation of the documentation. These disconnects caused many of the bugs and incorrect solutions that caused much fix and test cycles as well as go back and fix the incorrect solutions sometime after the solution has been provided to the customer through a production release.

Even though we saw these patterns, and knew that something was wrong with our processes and approach to both software development and project management, we assumed that we were using the processes or tools wrong. It took us some time to finally figure out that perhaps the processes and tools were the culprit.

We started to look around to see if others were experiencing the same thing. And, that's how we discovered Agile...

When processes were my passion

2007-02-08T07:27:00.000-08:00

It's interesting what others will say when you start something new or describe who you are and what you do. I have really appreciated the support so far and thank those that have mentioned this blog. One of them (who shall remain nameless) referred to this blog being more about processes than people. Actually, as you will discover as we go along, while processes are talked about the emphasis will be more about what is around those processes. In other words, processes will not be the core of the discussion.

However, there was a time when I was a process junkie. As I have found with many managers in my industry, I came up the ranks. I started as a programmer, and my manager saw some leadership qualities in me and was looking for a manager to handle daily operations while he spent time focusing on future initiatives. One thing led to another, and I began my journey from the programming world into management.

I didn't know how to be a manager at first, so like I did with programming I looked to others for advice. I didn't go to school and learn about management. So, I looked at what others were using as "tools" in the industry. As I had done with programming, I had assumed that if I just find a framework it will get me much faster to becoming a good manager. So, I went looking for a framework.

After some searching, I found some frameworks out there that seemed to be prominent at the time (this was in the late 90s). In particular, I found SEI's Capability Maturity Model (now CMMI) for software development and PMI's Project Management Book of Knowledge (PMBOK). These were THICK books full of processes. I had thought I had died and went to heaven! I couldn't wait to take this framework and apply it right away. And I did just that.

In later posts, I will get into more details of how I think about these frameworks now. For now, let's just say it didn't go well. I quickly learned that you can't just push process on people, especially if you don't really understand the concepts, values, and principles behind those processes. I was looking for process as a short-cut as well as a way to have better control of things as a young manager establishing myself. Instead, I came off as a young, cocky manager who had no idea of what he was doing and in the process making life miserable for those around him. People who used to like me as a programmer thought that the manager job had gone to my head. And you know what? I started believing it myself.

I know there are other managers who believe that if you put heavy formal processes, procedures, policies, etc. that you will be considered a great manager and that things are so structured and disciplined that anybody can do that job. These are the managers who talk in terms of subordinates. I quickly learned that I didn't want to become that kind of manager. It really wasn't my style, and I needed to find "tools" that supported what my strengths were as a manager. While at the same time, allowing those that I manage to play to their particular strengths. Also realizing that whatever was to be in place for processes needed some flexibility for the unpredictability that the future brings and would provide an environment that would not stifle creativity and innovation.

So, am I the "process guy"? Not anymore, I'm proud to say. If you could put any label on me, I guess you could call me the "enabler guy". I figure out how teams and individuals can become better. If they are better, my job is easier. If they are better, the company is more successful. Now why wouldn't I want those things as a manager?

A better way for conferences

2007-02-07T11:18:00.000-08:00

Ever heard of Open Space? If you haven't heard about it and experienced it, you will never think of a conference the same way again. I had the pleasure to attend a 2-day conference here in Portland for Agile Open Northwest.

Here's how it works, according to Wikipedia:

In Open Space, a facilitator explains the process and then participants are invited to co-create the agenda and host their own discussion groups. Discussions are held in designated areas or separate rooms known as 'breakout spaces' and participants are free to move amongst the discussion groups. Each group records the conversations in a form which can be used to distribute or broadcast the proceedings of the meeting (in hard copy, blog, podcast, video, etc). Online networking can occur both before and following the actual face-to-face meetings so discussions can continue seamlessly. In a multi-day Open Space, participants have the opportunity to announce new discussion topics / late-breaking sessions each new morning. At the end of the day (or 2 days or 2.5 days) the full group reconvenes for comments and reflection. This helps participants to re-engage in the full group over the duration of the meeting.

While the mechanics of Open Space provide a simple means to self-organize, it is the underlying principles that make it effective both for meetings and as a guidepost for individual and collective effectiveness. The Law of Two Feet -- a foot of passion and a foot of responsibility -- expresses the core idea of taking responsibility for what you love. In practical terms, the law says that if you're neither contributing nor getting value where you are, use your two feet (or available form of mobility) and go somewhere where you can. It is also a reminder to stand up for your passion. From the law, flow four principles:
* Whoever comes is the right people
* Whatever happens is the only thing that could have
* Whenever it starts is the right time
* When it's over, it's over

The organizing theme of an Open Space meeting is that people who care about the subject will come together. The initial meeting notice takes the form of an invitation, thus the people who have attended have chosen to be there and are willing to contribute. The objectives for the meeting and the time available affect design decisions such as whether action planning is included in the Open Space or not.

What did I like about it?

1) The content was tailored to the audience. Nobody selling their services or products that people may not care about. Topics were relevant to the desires of the people attending. Otherwise, nobody attended!

2) Many different perspectives were represented. I had expected that it may be more developer-centric, but there were several sessions for customers, product owners, testers and other roles. People attending also came from all of those perspectives.

3) Great respect for everybody in each session. I truly expected hidden agenda, egos, holy wars, and other things to dominant discussions. Never happened! Differences of opinions were just that - differences! No question was stupid, no perspective was invalid.

4) You could decide how much you wanted to participate. There were many that didn't say a word (or not much). Others felt very comfortable from the beginning. Some got more comfortable (like me) after some time had passed. Sometimes the experts talked. Sometimes they listened. It was very comfortable.

5) Tremendous sense of energy and community. I really can't describe it, but I never felt there was a slow time. Lots of things happening. You could feel the energy. I felt at the end a sort of sadness that the two days were ending. In most other conferences, I couldn't wait for it to end!

What could have been improved?

1) Some of the meeting rooms had more that one session in it and one of the them was also the gathering place for people between sessions. This made it very difficult to focus at times. Felt like we were on a party line competing with each other to talk. It would have been better to have one session per room and the general communal place to be located away from sessions.

2) Some of the sessions I felt that we talked problems but little solutions. Perhaps it was to be that way so that we could all go back to the real world and come up with solutions. Not sure I expected all of the problems solved but was a little frustrated because of that.

3) I didn't take any notes because all notes were to be gathered by each of the session "hosts" and placed on a common wiki for everybody to access. Unfortunately, some of the sessions never made it to the wiki or didn't have complete information. I would have filled in the blanks, but didn't take notes to focus on the conversation. This might have been just this conference as I have seen notes from other ones that seemed more complete.

If you have a chance to go to a conference or seminar that is set up this way, I highly recommend it. Don't come with too many expectations. Be open to anything happening. And hopefully my experience will encourage you to go. You will find those conferences to be the best you have attended. You will also wonder why you ever did conferences a different way. Lastly, you will probably never go to a typical conference again!

Before there were projects

2007-02-06T10:38:00.000-08:00

Though project management has been around for longer than I have been born, in the mid 80's I wasn't really exposed to the idea of software projects managed by project managers. As a programmer working in both an IT shop as well as consulting, life was much different.

I was assigned tasks by a manager (functional or account managers). Sometimes these tasks were very specific, such as "add this information to an existing report". Other times these tasks were more generic, such as "customer would like a new report that shows this kind of information". The requirements were always clear, but there was always flexibilty in the design. This applied to everything from traditional maintenance tasks (bug fixing, smaller enhancements) to major new product or system development from scratch. As a junior developer, I appreciated the extra support I had as I was learning the tools and systems necessary to excel at my job. Once I got there, I appreciated the flexibility that my manager gave me to figure out the solution with the customer more on my own. In other words, the tasks moved from specific to generic as my experience increased. It became my job to get to the details.

The manager was the initial "go-to" person when there were any questions. He or she needed to have a certain level of knowledge about what the customer wanted. However, they weren't the customer so their responsibility was to get clarity from the customer when needed. Sometimes, that could be done "off-line" so that I could focus on other things. Other times, it was determined that I needed to be there if the discussion became more technical in nature.

Another key difference is that there was little or no specifications. Essentially, most requests if written were a simple one page form providing some high-level information enough to get the conversation going. Other requests were verbal in nature. There were no project plans to review, drafting and review of a large functional specifications document, etc. So where did you capture this information, you ask?

Most of the conversations involved at that point were verbal, with the occassion taking of notes to go back to my desk. However, and here is the important part, the code WAS the specification. It represented the requirements and design. It was expected that you would go through several cycles of the changes. These weren't prototypes, but early looks at the finished product. At first, some things weren't finished or even developed yet. However, the point was to get feedback not just from the manager but many times from the end customer. "Are we on the right track?" "Is this what you were expecting?" In the process, we learned technical limitations such as "we had to redesign the report because the new data would not fit" and couldn't have expected some of those things until we got into the code.

As I progressed in my career, I look back to these early years and wonder if things are better now. Sure, technologies have changed but so have other things that perhaps have gotten in the way. All I know is that things seemed much faster in those days even though the technology was slower. But I have to believe that with improvements of technologies that things should be screaming now. However, they aren't. But that's a discussion for another day...

An Introduction

2007-02-05T09:59:00.000-08:00

Before we start this journey together, I feel an introduction is in order. Let me tell you a little about myself.

I have been in the IT or software industry for over 20 years. I started in programming and eventually moved into management, having been a CTO for the last six years in my current position. I have had experience with various technologies over the years - mainframe, client/server, networking, hardware, wireless, web, open source. I also have worked in several different capacities - in-house IT, in-house development, staff augmentation and outsourcing. I have worked in many different industries - retail, health care, financial, publishing, and automotive aftermarket. That's my credentials in a nutshell.

I am also not new to blogging. I started another blog, Random Thoughts from a CTO, in February 2005. That blog focused mostly on management and leadership topics and was moderately successful. Last October, I decided that I had said what I wanted on those topics and took a break from blogging. I wasn't sure that I was going to get back to it. But here I am!

This blog will focus more on the work itself than how to manage the work and people. Over the last few years, I have gained a lot of knowledge with Agile and experimented/implemented it in my current organization. In the last year, I have also gained exposure to Lean Product Development starting with the Toyota Production System trying to determine how the principles and practices could be applied to software development. I have learned a lot. I want to share my journey with you to this point. I want to share my quest towards an agile and lean organization. I believe that I will expand my knowledge through this blog and your feedback and support. I will not call myself an expert, but will share what I know with those not familiar with either Agile or Lean. I am not here to sell products or services, but to talk about the realities of life and to work in it. I am not here to talk theory, but will try to provide application to those theories.

Though the focus will be on how to apply to software development or IT, since that is my experience, I believe that much of what is discussed can be applied to other industries. I also believe that there is application to your personal life in addition to your professional life. "How you are" in addition to "what you do".

I look forward to the experience with each of you!
Skip