Pipelining is not Lean

LeadingAnswers has a thought-provoking post today called "The Pipelining Anti-Pattern". Here's how Mike describes it:

If you have analysts working ahead of development, or have testers working significantly behind development, then you may have “Pipelining” problems. Pipelining is the term used to describe the situation when business analysts are working ahead on the requirements for a future iteration; the development team is working on the current iteration, and the test team is engaged on a previous iteration. In some circumstances analysis may be several iterations ahead and testing several iterations behind. To some people this may seem an efficient use of resources with each group running at their optimal speed, unfettered by the co-ordination constraints of different groups. However from an agile and lean perspective this is problem, a bad-smell that needs fixing.

Here are the problems with pipelining:

Three teams not one – in a project where pipelining is occurring we do not have one cohesive team we have three teams (or more). It is hard enough co-ordinating the members of one team towards a common goal aligned to business benefits. When there are three teams it is just too easy for people to claim that they did their bit and problems lie with other groups. Yet, the fact remains that if the software does not meet business satisfaction then it is everyone’s problem.

Increased Work In Progress (WIP) – Requirements whether they are in the form of user stories, use cases, or formal specifications all represent work invested that has not delivered value to the business. The same goes for code, until this functionality has been tested to the satisfaction of the business it is not valuable. As the time increases between capturing the requirement and finishing the last test two problems occur. The first is classic accounting, money have been invested for no return yet and there is a risk associated with future returns. The second is that requirements decay; the longer we keep requirements around for, the higher the likelihood that they will no longer be required or will have to change.

Increased time from defect injection to defect remediation – the cost of change increases the longer a defect goes undetected. In a pipelining project, defects introduced by faulty analysis could take months to be detected in testing or user review. Fixing the problem after this period of time will entail refactoring far more code (for the work happening in the interim) than if it was detected earlier, and will increase technical debt...

Time fragmentation costs when groups collaborate – on pipelining projects the test group could be working on iteration 2, the developers on iteration 4, and the analysts on iteration 5 or 6. If a developer has a question for an analyst or a tester for a developer then task switching costs will be incurred. Task switching is the time required to respond to an off-topic interruption. We must mentally “park” our current work, try to think back to whatever it is this person who has interrupted us is asking about, recall the details, answer their questions and then resume where we left off. The problem with this is that studies have shown people, while great at many things, are generally very poor at task switching. Our recall of past details is poor, defense mechanisms in our brain designed to dull unpleasant experiences erode memories of nasty problems, and we are terrible at remembering where we parked our current thoughts. The net result is that just a few requests to recall past events really mess up our performance and yield low quality answers to the enquirer.

Throughput optimization – three or more teams running through work at their own speeds with fewer dependencies on others may seem like it is working, but from a systems perspective it is a problem. Processes running at different speeds require work buffers to avoid running out of work and they build piles of unfinished work. This inventory is waste in the system and leads to scrap and more rework as processes change and partially completed work needs to be updated.

I really think Mike is on to something. I have seen this struggle in our own product teams. Read more of the post to understand this problem as well as see how Mike addresses it.

A Simple Look of Lean and Agile

Somebody asked me if I were to talk about the relationship between Lean, Scrum and XP what would that be in the "simplest way possible" (At our organization, we are implementing all three). Here's how I answered:

Lean introduces the principles, or the "why's" of producing software. Focus is on maximizing customer value while minimizing waste to provide the most optimal ROI.

Scrum takes those principles and introduces the management practices, or the "when's" of producing software. Focus is on incremental delivery of working solutions that improve over time.

XP takes Lean principles and Scrum practices and introduces the technical skills, or the "how's" of producing software. Focus is on constant improvement of skills (and the underlying infrastructure) through analysis, design, and implementation of producing software.

It is my strong feeling that you really need all three to maximize your effectiveness in the development of software. Individually, each part can be implemented on it's own but you will discover something missing. Together, these really cover all aspects of developing a successful software solution.

Metrics in the Lean and Agile World

I ran across an interesting list of metrics from this site. The author of the post had gathered these in a brainstorming sessions with his organization on defining potential metrics on how they were doing with Agile. The author also poses a question to his readers on what metrics could be used by the team to possible deceive management. It's an interesting read. Here's the list of metrics that they came up with:

Short Iterations - Planning Game, Variable Scope:
* Relative velocity (measures accuracy of estimation and amount of developer time spent on other activities)
* Number of stories planned for the delivery vs number delivered
* Overflow of stories (measures accuracy of estimation)
* Degree of story change (measures activity in the business area and stability of business process)
* Number of problems fixed from previous iteration vs number of new stories implemented

Continous Integration:
* Time to build
* Number of successful/failed builds

Available Customer:
* Hours spent with developer/customer per week
* Speed of feedback vs resolution
* Number of raised features vs number of implemented features

Refactoring:
* Lines refactored
* Time spent refactoring vs new code
* Most frequently changed files
* Number of files changed per feature

PM Tools:
* Effort/time spent per feature
* Actual effort vs expected effort

Frequent delivery into Production:
* Amount of defects not caught by test/review

Regular demos for feedback:
* Degree of incorporation of feedback into iterations (measures accuracy of story gathering and accuracy of implementation)

Lessons learned for each iteration:
* Traceability of lesson as applied to future iterations/practices

Test-Driven Development:
* Degree of test case coverage
* Test case failure rates

Many managers would look at this list and think it's pretty good. In fact, they probably would think of some other measures to add to the list. However, what is the return for that investment (ROI)? Some of these metrics might be easy to gather, others could require more tools or manual effort to get. Even if you do get the information, how are you going to use the information? If you don't know how to interpret the data, you may find yourself managing the wrong things.

For these reasons, I have always struggled with finding useful metrics that are easy to gather and are promoting positive behavior. Metrics seem to be used more to punish bad behavior than to encourage better behavior. Lean software development advocates are pushing to measure up. What does that mean? Metrics are typically managed at too low a level in the organization. Therefore, companies come up with dozens of metrics (similar to a list like above) and ultimately get consumed with too many things to keep track of. Therefore, why not find metrics that can give you just enough information that measures things in terms of customers and deliveries? What if these metrics would encourage position behavior while helping you know where to look in more detail for inefficiences?

Following are the measures that I am considering in our adoption of Lean and Agile. Note, I haven't tried these out yet as the focus right now has been more on getting the practices and principles down before looking at efficiencies. However, based on my reading and understanding of both Agile and Lean here is my list:

Story Cycle Time (Measuring One Piece Flow): Average time from Customer Request to Customer Implementation for each Story. Encourages smaller pieces (Stories) that bring value. Encourages finding ways throughout organization to deliver those pieces faster.

Iteration Efficiency (Measuring Value vs. Waste): Percentage of Time spend on Value (Customer Features) vs. Waste (Defects, Refactoring, anything else the customer doesn't believe provide value). Encourages those activities that the customer sees value. Discourages those activities that the customer sees as waste.

Iteration Return on Investment (ROI) (Measuring Value vs. Cost): Ratio of Total Value (Customer and Business Value) against what it took to implement stories. Encourages the prioritization of things based on not just value but against the cost of implementation. Confirms to the team that they are working on the most important things.

Story Acceptance Measure (Measuring end user acceptance): Average acceptance (user rating) of each Story after it has been implemented. Encourages making sure the customer needs were really met. Encourages quality of solution from a customer standpoint. This may be the most difficult to gather because it requires polling at least a good sample of customers.

What do you think? Did I miss any measures at the organizational level?

Team Power

In both Agile and Lean circles, there is constant emphasis on the team rather than individual efforts in resolving issues, making decisions and suggestions for improvement. "Two heads are better than one" rings true! Over at LeadingAnswers blog, the latest post called "Team Solving: The Under Utilized Power" demonstrates this fact. Here is a synopsis of the findings:

Power #1: By asking the team for a solution we inherit consensus for the proposal. The fact that the solution came from the team and not me, meant I did not have to sell it, it already had support. It is easier to steward a sub-optimal solution with good support to a successful outcome than build support for an optimal solution and ensure its successful execution. If challenges arise and people subconsciously ask “who’s bright idea was this?” the answer comes “Oh yeah, it was ours” and they are more likely to continue.

Power #2: Accessing a broader knowledge of the facts. Team members are closer to the details and bring additional insights other than your own. I did not know that the business folks bowled, this was collective knowledge (Chris knew three users bowled and Julia knew two others did also) together we found a new piece of useful information. Asking the group taps the collective knowledge of the team.

Power #3: Solutions are practical. Anyone who has worked hard to craft a solution only to be told “That will not work here because...” will know how frustrating and disheartening these words are. Team sourced solutions have been vetted for practicality and, because created internally, solutions found for implementation issues.

Power #4: When consulted people work hard to generate good ideas. The simple act of asking for suggestions engages team members beyond the role of “Coder” or “Tester”. People appreciate having their inputs valued and generally work hard to create innovative and effective solutions. Treating workers as interchangeable resources is a poor model inherited from the command-and-control methods of the industrial revolution. Leading companies such as Toyota and 3M recognize that their best ideas come from inside their companies and we need to make use of this intellect. It is partly due to these methods that these companies innovate better, have higher quality products, and better labor relations.

Power #5: Asking for help shows confidence not weakness. Asking for ideas and solutions to problems is not a sign of incompetence or an inability to manage. Just because we ask for input it does not follow we are dumb. Instead it demonstrates valuing the opinion of others, and being thoughtful. In essence it demonstrates how all problems should be tackled, which is the next power.

Power #6: Seeking ideas models desired behavior. Project managers have a leadership role of “modeling the desired behavior” i.e. behave as we wish others to behave. If we stay silent, make decisions with incomplete awareness of the facts, and do not ask for help when we need it, what message is this sending to the team? Well, it is an obvious message that we expect team members to behave the same way and work in isolation. Lots of time and money is wasted on team building activities that are then eroded my management-in-a-vacuum.

Here are also some cautions where the team can actually lose power:

Caution #1: Real problems - We should use this on real problems only, not on which brand of printer toner to buy. Remember we are modeling desired behavior and if people take it too far and consult their team mates when deciding what color dialog box to create; no work will get done. It is a tool for when you are stuck and the problem is important.

Caution #2: Poor team cohesion – If the team is fragmented and has opposing groups then resentment for “fixing their problems” will undermine the process. We need to get the team aligned for team solving to be most effective.

Caution #3: Team and Project Changes – Over a long period if a significant portion of the team changes, we need to re-canvas the team to ensure they are still on-board with the approach. Exercising the bright ideas of others is nearly as bad as not being consulted; we need to ensure people still agree this is a good policy. Likewise if the project changes significantly, we need to checkpoint in light of these new facts and get the team to review the approach.

Caution #4: Follow-through – once you ask for solutions make sure you follow through on them with execution. It is pretty demoralizing to be asked to work on a solution and then see it wither. It is fine to go back with implementation problems that need to be solved, but don’t waste peoples time by asking for input and then ignoring it.

Agile or agile, Lean or lean?

Recently, I have seen a trend where other bloggers and authors are having difficulty or are trying to define the difference in using Agile (capital-A) vs. agile and Lean vs. lean. Some would say that by putting a capital in front of it that it means you are focusing on a particular methodology or defined Agile process. So if you are referring to something like XP or Scrum, you should use Agile. Otherwise, if you are talking about principles and particular practices, you should use agile. Others don't want to put a label on "Lean" or "Agile" because they are afraid I guess of watering down or maybe commercializing those terms? Other people would believe that you should use capitals in some situations and not in others.

Frankly, I just don't get it. All this brings is more confusion to the community. To me, putting a capital on Lean and Agile allows my readers to know that I am talking about the values, principles, practices, tools, measures, etc of a way to develop software differently. By not putting the capitals, I don't want to confuse people with the generic terms of lean and agile which are:

lean

1. To hang outwards.
2. To press against.
3. (of a person) slim not fleshy or having little fat
4. (of meat) having little fat
4. Having little extra or little to spare; as a lean budget
5. (of a fuel-air mixture) having more air than is necessary to burn all of the fuel; more air- or oxygen- rich than necessary for a stoichiometric reaction

agile

1. Having the faculty of quick motion in the limbs; apt or ready to move; nimble; active; as, an agile boy; an agile tongue.

Agile and Lean are states of mind, are a way to develop software, have common values and principles but many practices and tools to implement. agile and lean? They don't tell me much. I'm going to use capitals!

Addressing Technical Debt

What is Technical Debt and how to you manage it? These were two questions that is addressed over at the Lean Software Engineering blog.

In the first post, called "7 sources of technical debt" Bernie creates his list of where technical debt comes from:

1. Undiscovered bugs, failing tests, and open bugs in your database
2. Missing test automation (unit, feature, scenario, or system)
3. Missing build and deployment automation
4. Scenarios with incomplete user experiences
5. Code that’s too difficult to understand
6. Code that’s too difficult to extend
7. Code that’s isolated in branches

In the follow up post, called "7 strategies for measuring and tackling technical debt" Bernie provides his list of potential remedies to reduce or eliminate this debt:

1. Bugs. Measure and shrink your cycle time from when code is first written until it is tested (TDD is perfect for this, of course). Treat open bugs as work in progress (WIP) — and too much work in progress is evil. Constantly measure and manage WIP down with strategies like bug caps (no new features until existing bugs get below a defined bar).

2. Missing tests. Measure code coverage. Have a clear definition of “done” at each scale of your product: unit, feature, scenario, system. Treat units of code as incomplete until code, tests, and documentation are delivered as a set.

3. Missing automation. Make build and deployment first-class features of your project. Deliver them early, and assume (like other features) that they will constantly evolve with sub-features over the course of the project. Measure time-to-build, time-to-deploy, and time-to-test — including human time expended each cycle — and then drive those numbers down by adding automation work to your backlog.

4. Incomplete scenarios. Assign scenario ownership roles among the team. Have a clear definition of “done” for each scenario. Pre-release working scenarios to select customers as soon as possible, making special note of these bugs. And of course, minimize the number of scenarios each team focuses on at once (one at a time, if possible).

5. Not understandable. Like most open source projects, generate docs directly from the code (literate programming), including why the code does what it does. Systematically conduct code reviews. If the team can’t afford to review every line, always review a sample set of functions from each developer and component. Ask reviewers to rate code for understandability, track this data in code comments or a spreadsheet, and add workitems to the backlog to clean up those most needing improvement.

6. Not extensible. Encourage design pattern thinking. Buy your team a library of design patterns books and posters to help form that common vocabulary. Look at how open source projects tend to build big things out of many small, independent pieces — treat every component as a library with its own API and a clear (and minimized) set of dependencies.

7. Not integrated. Minimize the number of branched codelines, whether they’re formally in source code control, done on the side for customers, or sitting on developer’s disk drives. Unintegrated code is debt — it has work and surprises waiting to happen. Strive for continuous integration. This isn’t to say don’t use branching — even the smallest project should think about their mainline model — but always keep everything in source control, then track and minimize the number of active branches.

This is by no means an extensive list as I'm sure there are other contributors of debt and possible solutions. However, I agree with Bernie that these are the most common things to be on the lookout for. Just think how better the quality of software, and the ability of a team to deliver new features can be maximized if teams would see things in terms of debt!

Read more in both of these posts.

Changing the way you do long term planning

We are in the process of implementing a pull scheduling system for each of our product development teams. No longer do we think in terms of projects, but in terms of releases and iterations. No longer do we figure out the entire scope in detail of what will be contained in the product, but we determine the details when we are ready in a particular iteration in a release. No longer do we have rigid control of scope for the entire project, but we try to contain what we do in a 2-week iteration once it has been planned and committed by the team. No longer do we try to figure out exactly what we will do in the next 1-2 years, but we "pull" enough work to determine the next release when we are ready to think about it. No longer do we wait for customer feedback only at the end of the project through alpha and beta cycles, but we get internal and external feedback through demos at the end of each iteration. No longer do we conduct vague and long weekly project status meetings, but we conduct quick and concise daily standup meetings. Our world has definitely changed, but in the minds of our executive stakeholders they have struggled a little with it. In particular, if you are only planning within short iterations and the current release cycle, how do you do long-term product planning for customers that want to have some idea of a delivery date for something "down the road"?

I have no problem in planning long-term if the planning is focused more on priorities than dates. Though the team is pulling just enough work for the next release, the more work that is prioritized and thought out (at least to a high level) will make the release and iteration planning go much smoother. And, actually I don't even have a problem committing to a particular date for the next release or any iteration in the release because those are fixed. However, committing to particular scope outside of the current iteration is something that I find challenging. Why?

Because we know only look at the details of the release within the current iteration, we haven't given enough thought about the remaining stories in the release. Sure, we have done some high-level estimates but we have learned that what you think you are going to do after thinking about it for a few minutes is much different when you think about it for a few hours in iteration planning. Until you really start looking at particular tasks and tests that the team will do in fulfilling each story, you don't really have an accurate estimate. And even with that said, you may find surprises within the iteration after planning.

The other challenge is that you are dealing with change in a whole different way. When you find a defect, you don't "log" it in a repository automatically because you don't have the time to fix it during the project. Instead, you deal with defects as they happen. When you show functionality in a demo, you don't log future enhancements automatically to be considered in future projects but you make appropriate changes if they are valuable and while they are fresh in the team's minds. When new issues come up, you create new stories to deal with them if they are deemed the right time. All of these things change scope, which changes what is done in a particular release. This changes the expected delivery dates many times. Which makes it very difficult to predict the scope of a release, and makes it almost impossible to predict when new functionality will be available in 12-18 months.

Now, this may sound bad initially to your stakeholders, because they want to make promises to customers as soon as possible. So, this may sound like Agile and Lean is the wrong way to go right? While I can't predict what will be done in 18 months, I can show you progress every two weeks. This progress will have higher quality because it has been tested and could be released to customers if desired. What we work on has been determined the most important to our customers, so this is a good thing. The best thing of all? The fact that we make sure that every deliverable, minimal useful functionality, really is what the customer wants because they have been more involved more frequently. So does it really matter if we try to fix scope, resources and schedule but having such as strict change control process if in the end we don't deliver what is really wanted?

So, what I have agreed on with the stakeholders is that we will go out beyond a release but strictly from a planning and goals perspective. At the point where it is determined that the goals are not obtainable in the process, I will make sure that the stakeholders are notified and can adjust the scope or define new goals based on this information. What's interesting is that I will never have to have that kind of conversation because the stakeholders will be involved in every part of the process. They will see progress being made, they will set priorities of work, they will attend demos and make changes, and they will see the impact of those changes. In the end, they will learn that they can draw the "line in the sand" when they feel most appropriate. Based on the confidence that they are gaining with the teams, they will change the long-term discussion with the customer to what is being developed than what is being planned. When that isn't enough, they can give the customer an idea of the priority of work being considered. If the customer feels that the priority isn't meeting their needs, they can voice their opinion. Then the stakeholders can determine what to do with that feedback and make priority changes as necessary. As long as customers see steady progress with frequent releases that are meeting needs, they should be satisfied with this approach.

Another View of Lean Leadership

Joe Little from Agile and Business has an excellent post today called "Leaders, Managers, Bosses and Administrators". Actually, he focuses on what leadership means in the Lean and Agile world. He refers to the Poppendieck book, "Implementing Lean Software Development: From Concept to Cash" (If I had to pick one book to understand Lean as applied to software, this would be the one!). Here's his take on leadership after reading the book:

They [Poppendiecks] raise several excellent points.

1. A team needs leadership. Which is to say, vision. Someone to inspire and someone to help them put their hearts in the game. And keep it there.

2. The project needs decisiveness. If the team has too many leaders, and the leaders squabble about decisions, then the team wastes time. The team needs to know how it will make decisions. There is a trade-off between making the right decisions, and making decisions quickly.

3. Generally, the team needs to learn to make decisions only at the last responsible moment. So, much of the decision-making is about when to make a decision. At what point have you learned enough to make a good/better decision?

4. The project needs business decisions and technical decisions. This is very true. So the team needs business people and needs technical people who are ready and able to make those decisions. And, preferrably, business people who understand technology and technology people who understand business (and the customers).

5. And there are many other types of decisions to be made too. People decisions. Decisions about whose insights to go with on specific areas. Process decisions. Decisions about who is working effectively and who is not. Decisions about how to get the team to communicate better and learn faster.

6. In Scrum, we have ScrumMasters and Product Owners. These roles are endowed with certain leadership aspects. This is different than the leadership of the Chief Engineer, which is a role Toyota uses.

7. Project managers have also provided leadership. (And we have the whole PMP, PMI thing, too.) PMs have also provided managership, bossiness, administration and other things.

8. We know that no bosses are wanted. We want all the best from every person, and a boss will only inhibit that. A boss wants to command others, and thinks he knows all. These are not helpful traits in a learning situation. (So, semantically, we are using "boss" here to represent all the bad things that a boss can be. Of course, few managers or leaders are as bad as a boss, but we all can be that way sometimes.)

Read more in Joe's post.

Growing Your Leaders

Alan Shalloway from the Net Objectives Thoughts blog challenges how Agile has viewed management and leadership up to this point in his post "The Need for Leadership in Scrum". Here are some highlights:

There underscores the notion that self-managing teams and direction from the top are somehow opposed to each other. In practice, this may be. But that is because in practice, leadership is missing. Direction in the form of command and control is bad. Direction in the form of vision and matching people to their abilities is not. This points to one deficiency I have seen in the general Agile community, and one that, I believe, is a serious impediment to having the Agile community have success at the enterprise level. I am referring to the fact that most Agilists do not trust management to do what is right – they feel the team must be protected from management. I have heard this opinion voiced in several books and from several leading agile consultants/practitioners – and I disagree with it.

Unfortunately, this attitude has been pervasive from the beginning of the Agile community. Given the heavy focus on over-bearing process that has gripped this industry for years, this is not surprising. However, it is time to include leadership and management in the transition to agile methods. We must see how we can marry respect for people (and hence the team) along with process. This requires leadership and proper management.

Buckingham, in his excellent book, The One Thing You Should Know describes leadership as the “ability to create a solid vision of a better future for those people he/she is leading. ” A leader must have a compelling desire to move towards that vision. Buckingham defines management as the “ability to match people’s tasks with their skills. ” A good manager is liked a skilled Chess Master. As opposed to checkers, where all the pieces (until one gets kinged) are the same, in chess, different pieces have different strengths. A good chess player knows how to use these strengths. A good manager knows how to maximize the strengths of his staff. Hence, leadership and management are not opposed to each other – they just have different functions.

Keeping Buckingham’s views in mind, the role of leadership and management then is to create the vision for the Scrum teams to be effective and to staff the teams with the appropriate resources. But just creating the vision is not enough. Leadership and management must help create the structure within which the team works to assist the team. This is something Scrum teams cannot do on their own. Ignoring this issue is what often has Scrum teams feel they must protect themselves from the organization when what they need to do is help the organization see how to better support the team.

If Scrum can’t help us here, what can? This is where Lean Thinking comes in. In a nutshell, Lean Thinking tells us to set things up so we can have fast-flexible-flow. That is, we can get customer requests in and get business value out – quickly and repeatedly. At Net Objectives we base our entire software development philosophy on this (which is why we call it SPeeD: Sustainable Product Development). The issue is – “how can I develop software quickly now (to deliver business value quickly) while retaining the ability to add business value quickly in the future.”

Read more in Alan's post.

In the book The Toyota Way by Jeffery Liker, one of the 14 principles outlined says "Grow Leaders Who Throughly Understand the Work, Live the Philosophy, and Teach It to Others". At the beginning of the chapter, there's an excellent quote that parallels Alan's frustration mentioned above:

Until senior management gets their egos out of the way and goes to the whole team and leads them all together...senior management will continue to miss out on the brain power and extraordinary capabilities of all their employees. At Toyota, we simple place the highest value on our team members and do the best we can to listen to them and incorporate their ideas into our planning process. -- Alex Warren, former Senior VP, Toyota Motor Manufacturing, Kentucky.

In summarizing this principle, Liker says:

If we look at all of the great leaders in Toyota's history we see they share several common traits:
1) Focused on a long-term purpose for Toyota as a value-added contributor to society.
2) Never deviated from the precepts of the Toyota Way DNA and lived and modeled themselves around this for all to see.
3) Worked their way up doing the detailed work and continued to go to the gemba - the actual place where the real added-value work is done.
4) Saw problems as opportunities to train and coach their people.

This doesn't sound like the ol' command and control type of manager works in this paradigm does it? However, managers are needed to provide Agile teams this kind of guidance. To do so, managers must respect team members as well as the other way around. Each have their place and need to understand their interdependencies in order to be a successful organization.

One Story Flow

One of the practices in Lean is this idea of One Piece Flow. What does this mean exactly? It focuses on getting more components completed more quickly by focusing on the completion of each one instead of having too much work in process. The thinking is that if your team can produce one component quickly with quality, you will have peak efficiency for future components.

So, where does this fit into Agile? In iteration planning, the team is working with a set of Stories. Here's where you can go in two different directions and has been a constant challenge in our organization. Where you go from here can potentially have two very different results.

First, you could ask the team who wants to work on each Story. Each individual responsible for the Story can then go off and determine the Tasks and Tests necessary to complete the Story. So, if you have a team of five developers, you will have five Stories being worked on (at a minimum, they may take on more Stories if they believe they have time in the Iteration and the Stories are small enough).

Or, you could have the entire team in Iteration Planning determine both the Tests and Tasks necessary for the first two or three Stories (more if you have a smooth process ). Then, have the team implement the first Story together (or as many team members as possible). If there isn't enough work for everyone, some go and start implementing the second Story. If the team members responsible for the first Story need help, others working on the second Story join the effort. The goal here is for the team to get each Story implemented as quickly as possible before they go on to the next Story. This is what I will coin "One Story Flow".

As I mention, we are still really struggling with this because it has been the nature of developers to have each person work on their individual piece and to bring the pieces together during some time of integration. However, this way of thinking starts to produce mini-Waterfalls within each iteration. Also, I have seen more unfinished Stories as a result. I would rather have more Stories that have been carried over to the next iteration at 0% than at 50%. I would also want more Stories that have been completed by the team at 100%.

If you are struggling with the team's performance and completion of Stories, try the "One Story Flow" approach. You will have more flexibility as a team and the team will be more in sync on how each Story has been implemented and how it contributes to the big picture of the product. More discussions up front will resolve conflicts and turn assumptions into facts. Finding ways to implement one Story after another will address issues such as specialization of skills, knowledge, and bottlenecks in the development process. These are painful but good things to discover. Because if the team can find ways to implement one Story better, they will be able to implement all Stories better!

The Ultimate Measure of Agile?

I attended a local lunch meeting where a panel of people talked about Agile development and how they addressed challenges in each of their organizations. Two of the panelists were managers in software product environments. As they talked, it was obvious that each person was at a different stage of maturity in their adoption of Agile. The first speaker seemed to think they were doing Agile development, but his description of their lifecycle seemed to indicate that they were just doing Iterative development with mini-waterfalls. The second speaker seemed to be much further along as they were committed to short, timeboxed iterations of 2 weeks with co-located teams serving both testing and development efforts.

When it came to Q&A, there was something that was bugging me so I asked them this question, "How has the adoption of Agile impacted the Release Cycle time?" The answer from both was what I expected after hearing the earlier discussion, "Not much at all". Because of marketing and other budgetary reasons, they felt they could only do a release or perhaps two a year.

And yet, especially that they are managers, I think they are missing perhaps the ultimate measure of Agile - How quickly are we giving our customers working software? Perhaps this is what I have learned from Lean and the emphasis on Waste and Value. Within Lean, the emphasis is to always improve the cycle time to the customer by providing the most valuable functions and minimizing waste. Part of waste is partially completed functionality.

I think as people implement Agile, they will realize that they need to mature to a point where they are incorporating Lean. After all, shouldn't software be about maximizing ROI from both the customer and shareholder point-of-view? You cannot improve ROI if you aren't improving the cycle time. You cannot improve cycle time without improving quality and removing waste.

While each of these individuals have gained some benefits with Agile such as better adapting to change, getting frequent feedback, and better ways to track progress, they still haven't realized the greatest reward of all - getting working software out to customers as quickly as possible. This goes beyond budgets and marketing plans, and turns the organization 180 degrees to fully accommodate customers. They don't want to have to wait a whole year for another product, they want changes as quickly as possible. If you can't find a way to make it happen, it's possible a competitor will.

Maximizing Customer and Shareholder ROI by provide working product that meets both business and customer value as quickly as possible. This is the ultimate measure of being an Agile organization!

YAGNI is very lean

In Extreme Programming (XP), there is an accepted principle with the acronym of YAGNI. What is it? Here's a good summary from Wikipedia:

The YAGNI principle of software development says that when there's a good chance that "you ain't gonna need it", it's better to postpone adding a program feature.

Even when it turns out that the functionality really is needed, the habit of writing code that is not necessary at the moment has some overlooked disadvantages:
* The time spent is taken from necessary functionality.
* The new features must be debugged, documented, and supported.
* Any new feature imposes constraints on what can be done in the future, so an unnecessary feature now may prevent implementing a necessary feature later.
* Until the feature is actually needed it is not possible to define what it should do, or to test it. This often results in such features not working right even if they eventually are needed.
* It leads to code bloat; the software becomes larger and more complicated while providing no more functionality.
* Unless there are specifications and some kind of revision control, the feature will never be known to programmers who could make use of it.
* Adding the new feature will inevitably suggest other new features. The result is a snowball effect which can consume unlimited time and resources for no benefit.

This principle very much supports both the concepts of Eliminating Waste and Defer Commitment. If you are not sure that code or functionality is really needed, you need to do some more homework. This homework could be additional research, talking with customers, or just waiting until you deliver higher priority functionality to the customer and really see if they need something else. In Lean, if there isn't direct Customer Value it is considered Waste. That is a bad thing. As mentioned above, it also leads to more complexity which can impact the long-term maintainability and quality of your code. Therefore, why not keep things as simple as possible until you absolutely need to put more into it?

I have seen in many design sessions where things get into a frenzy because the developers have figured out something cool with the technology they are using and are very eager to implement their ideas. This is fine, as long as the technology and their ideas translate into something the customer really wants or needs. Too many times, customers are usually forced into having something new that the development team thought they needed only to find they hardly use it. What happens next? Well, usually both the customer and the development team live with this unused, most likely complicated functionality forever. In the end, nobody is really happy about it.

However, when you find yourself in such a meeting (and you will at some point), remind the team about YAGNI. This will be a good reminder to them about the important of value and to wait until it is the right time to implement such functionality.

Go to the Source

In Lean, a key practice is one called "Go to the Gemba". Here's a brief description according to Wikipedia:

Gemba is a Japanese term meaning "the place where the truth can be found." Others may call it "the value proposition."

A Gemba visit is often simply called a customer visit. The hallmarks that make it uniquely useful are:
1) The purpose is firstly to observe, occasionally to question, rarely to guide or direct.
2) The visit occurs in the context where the product or service is used, which allows direct observation of problems that arise, workarounds that are applied, and capabilities or services that are never used.
3) Sometimes the customer (or client or user) is asked to describe what he is doing while he is doing it; this provides insight into the thought processes, which often reveal differences between the customer's mental model and the model of the developers or providers of the product or service.
4) The customer will often express wishes or needs while working in context that would be forgotten or suppressed in a different context such as a structured interview or sales meeting.

Common cases for a customer visit include:
1) Enhancing the features or usability of products (especially software) or devices (especially ones aimed at very broad or very niche consumers).
2) Improving processes or tools.

I think this describes the primary use of this practice, but I can see other uses as well. Here are some other sources that should be considered as a manager:

Individual Worker - Look at each person. Do they have a significant role? Do they understand their role? Have they developed the right skills for the role? As a manager, it is your job to ensure that each person understands their purpose in the organization. It is also your job to encourage that person to continually improve how they do their work as well as increase their value to the organization. Management should make sure that the culture is such that each individual once they understand their role, have autonomy to improve it as they see fit.

Development Team - How is the team functioning? Are the processes and tools supporting the team or getting in the way? Are they communicating well? Working well together? It is crucial that the team are working in a way that work flows smoothly, there is little or no waste and everything the team does goes towards providing value to the customer.

Management - Does your management style support Lean? Do you understand what Lean is all about? Are you establishing a culture that compliments Lean? Are you a champion of Lean? Are you helping others understand it? As a manager, you need to be the experts of Lean and do all you can to understand what it takes for an organization to become Lean. If things are falling apart because people don't understand what they are doing or have gone back to their old ways, it is up to managers to educate and instill the self-discipline each person needs to have to make it successful.

This is an area that we are still struggling with but realize that it is critical to our success moving towards a Lean organization. We must "Go to the Gemba" and find the problems (and solutions) where the root source is to properly address. Otherwise, we are making a lot of assumptions that may not prove accurate. As the old saying goes, "When you assume, you make an ass out of you and me." It's important to base decisions on real facts than assumptions. Going to the source will get you to the right place to identify the facts. It's then up to you to figure our fact from fiction.

What does it mean to be Lean?

Note: I wrote this post originally on my first blog, Random Thoughts from a CTO, but thought it was good enough to share with the readers here.

Lean can mean different things to different people.

For some, lean can refer to how you maintain your physical shape through dieting and exercise. For those that have been successful in staying lean, they will say that it takes a paradigm shift of how you take care of your body through what you eat and how you exercise. When you start the process of becoming lean, you begin to cut out those things that are unhealthy and make you fat. You begin to eat better. You begin to feel better. You start seeing results in the mirror. You also are constantly monitoring your weight, heartbeat, blood pressure, calories, etc through the process. For some of us, myself included, we get lazy once we reach this point and start falling back to bad habits -- at first, a mistake here and there on what we eat, or missing a workout or two. Then, you stop getting on the scale and recording your results. Then, before you know it you are not lean again!

For others, lean can refer to how you maintain your financial status through budgeting and tracking your expenses. For those that have been successful in saving money and making more through investing, they will say that it takes a paradigm shift of what you do with your money and where it goes. When you start the process of saving money, you begin to determine where your expenses are going and which of those expenses you can eliminate that takes you from your goals. You also look at ways to increase your income with those savings through investments. You also look to remove your debt. You begin to save money. You begin to feel better. You start seeing results in your portfolio. You also are constantly monitoring your income flow, expenses and net worth through the process. For some of us, in this case myself not included, we get lazy once we reach this point and start falling back to bad habits of spending excess money, getting into debt, and eventually not saving money. Then, before you know it you are not financially set again!

In the software engineering industry, lean is now referring to your cycle time - how quickly you get to a working solution for the end customer without losing quality. For those that have been successful with this approach, they will say that it takes a paradigm shift of how you development and manage your software solutions. When you start the process of reducing cycle times, you begin to determine where communication and collaboration need to happen, what activities or processes take away from reducing cycle times, how we do our jobs and the way we interact with other areas. You will begin to see results. You will see the customer is more satisfied. You will find better ways to get your work done. You also are constantly monitoring your estimates, work completed, costs, quality much better. For some of us, myself a little bit, it is very easy to fall back into old habits of traditional and formal processes because you are so familiar with them and may get discouraged early on in not seeing immediate results you are hoping. You then begin to see things slip, take longer and less of a feeling of accomplishment either by the team or the end customer. Then, before you know it you are feeling that things aren't getting done as they should!

There are many other scenarios that this lean approach can take - time management, stress management, strategy, project management, the list goes on and on. What is getting in the way of your goals? How do you improve on those goals? What do you do to maintain those goals? How do you measure the process of those goals?

Focus on the goals and maintain discipline to see it through! This is what lean should mean!

The Predictability Paradox

We had just several months earlier started our implementation of Scrum to all of our project teams. As we were learning what to do, more importantly we were learning things not to do. One of the things I found challenging in our adoption of Agile was that many felt that the traditional (Waterfall) way we were delivering software before was more predictable and that Scrum would be totally unpredictable given its emphasis on focusing one iteration to the next instead of heavy upfront requirements, design and planning. I knew in my heart that Waterfall really couldn't predict any better but really couldn't put my finger on why. It was then that I read this paper called "Lean Development and the Predictability Paradox" that I got my answer. It was this paper that changed my thinking and introduced the Lean concepts to apply to what I had already learned with Scrum. Here is the introduction (read more including more details on each aspect of Lean by downloading the paper here):

Wall Street has little sympathy for companies that can’t meet their forecasts every quarter. In turn, senior management expects department managers to make and meet forecasts. By the time these expectations arrive at an IT department, meeting forecasts often becomes a significant challenge. Unfortunately, it seems that a large fraction of software projects fail to deliver on their promises for one reason or another.

Why is it that software development projects seems to have so much difficulty delivering predictable outcomes? It seems that all too often projects are late or over budget or they deliver the wrong system or all of the above. How can the predictability of software development outcomes be increased? This executive report discusses a dilemma: in our zeal to improve the reliability of software development, we have institutionalized practices that decrease, rather than increase, the predictability of outcomes.

The best way to achieve predictable software development outcomes is to start early, learn constantly, commit late, and deliver fast. This may seem to cut against the grain of conventional project management practice, which is supposed to give more managed, predictable results. But predictability is a funny thing; you cannot build with confidence on a shifting foundation. The problem with conventional approaches is that they assume the foundation is firm; they have little tolerance for change.

The paradox is that trying too hard to create predictability creates opposite effect. Conventional practices are fragile in the face of change, and even in the face of learning. And yet, the more complex the system, the more necessary learning becomes. What is needed is an approach that encourages learning, and does not commit until learning is complete.

It should be obvious that decreasing the amount of speculation involved in making a decision increases predictability of the outcome. If you can make decisions based on facts rather than forecasts, you get results that are more predictable. Lean development is the art and discipline of basing commitments on facts rather than forecasts. It starts earlier, encourages change, freezes decisions later, and delivers faster than traditional practices, but nevertheless lean development produces outcomes that are more predictable.

The paradox of lean development is that you have give up some of the trappings of predictability in order to get true predictability. You have to abandon some conventional wisdom to gain the benefits of making decisions with more certainty. Fundamentally, you have to develop the capability to respond to events as they unfold, rather than hold dear the capability to orchestrate events in advance.

The Agile Elevator Speech

I was going to also call this post, "How to talk about Agile to the non-Agile" but thought the title above was more appropriate. Joe Little from Agile and Business blog has a post was trying to figure out how to tell friends and family about what Agile and Lean are without having to spend a lot of time getting them up to speed. In his post, he ends up summarizing it as:

* Focus on delivering small slices of concrete business value frequently (1-4 weeks)
* Work with a small team of people, to enable them to be as productive as possible
* Use concrete, understandable results to allow all parties to adapt and move forward
* Arrange things so that change and learning are benefits
* Tell the truth, and make the important things as visible as possible
* Minimize waste and extra effort; maximize customer value, don't impress with hard work

I think this is an excellent introduction to both Agile and Lean that can be accomplished in a few minutes and give the listener an idea without having to know much about software development, product development or manufacturing. Read more in his post on how he led up to this list.

Interview with the Poppendiecks

InfoQ just recently conducted a video interview with the Poppendiecks. If you don't know who they are, here is an introduction:

Mary and Tom Poppendieck www.poppendieck.com teach and consult worldwide on Lean principles for software. Their practical, customer-focused approach to software development identifies real business value and enables product teams to realize that value. Mary has managed solutions for both operations and new product development. Tom is an enterprise analyst, architect, and agile process mentor.

Lean software gurus Mary and Tom Poppendieck share their years of practical experience, as they speak on the history of Lean thinking, the value of fast delivery and deferred committment, their use of Value Stream Mapping to identify and reduce waste, the importance of identifying and dealing well with cross-organizational and inter-organizational boundaries, and how Lean relates to RUP and Scrum.

They cover a wide variety of topics in this Question and Answer session. Everybody should get something new from it. You can find the video here.

Going to a Lean Development Course - Update!

Next week, I will be attending a 2 day course in Seattle on Lean Software Development. The course is being conducted by Net Objectives and led by Alan Shalloway. I have enjoyed reading Alan's contribution to the Lean Development Yahoo Group and can't wait to learn more. Here is a synopsis of the class taken from their website (you can learn more by going here):

The software industry is looking for ways create quality code in an effective, efficient manner that is rapid, repeatable and reliable. Attempts such as CMMI and Agile methods have found some success but also seem to have some inherent drawbacks.

Of the many methods that have arisen to improve software development, Lean Software Development is emerging as one that is grounded in decades of work understanding how to make processes better. Lean thinking focuses on giving customers what they want, when and where they want it. It provides a way to maximize value while minimizing waste.

The varied background and history of Lean makes it very valuable to practitioners. However, the varied perspectives available from which to look at Lean sometimes makes it a challenge to learn how to apply it. This course takes four prominent perspectives of Lean and integrates them, creating a consistent, comprehensive view of how Lean can be applied to software development. These perspectives are:

Lean Manufacturing from Toyota (from Taiichi Ohno of Toyota)
Lean Thinking (from Womack and Jones’ work)
Lean Software Development (from Mary and Tom Poppendieck)
Lean Product Development System (from Toyota, as described by Michael Kennedy)

The course centers on how to create a fast, flexible flow of customer value-add. Principles of Lean that facilitate this are:

Eliminate Waste
Create knowledge
Respect people
Build quality in
Defer commitment
Deliver fast
Optimize the whole

The course teaches how to manifest these principles within the context of software development as product development. This is one of the distinctions of this course over mere agile training. Agile project management focuses on managing a single project and perhaps coordinating several projects together. However, true software development should be focused on the products these projects relate to. Selecting products, scheduling projects for the products, balancing product loads, are business perspectives that Lean addresses while agile methods do not. The focus of this course is on the mindset of Lean and the Lean-Agile Software Development process. The issues of architecture and how to evolve designs is only dealt with at a superficial level.

P.S. I found out about Net Objectives through subscribing to their Lean-Agile Straight Talk podcast. If you want to learn more about how Lean and Agile work together, visit their site. But just promise to come back here to learn more!

Update: If you are trying to implement Agile or Lean in your organization, you NEED this course! Though there are plenty of books on the topics, Alan did a great job of blending various experts and ideas out there and be able to understand the big picture. This isn't your typical training course however. We were able to have a lot of discussion among the group about our own experiences so far. Alan, being an Agile/Lean coach as well as trainer, provided a lot of good ideas for us to take back. It was like getting free consulting/coaching along with the cost of the course (which, by the way, at $1000 is a fantastic bargain). Check their web site for more details on the next courses as well as other courses in the area of Agile.

Video introduction to Lean

The following video was presented by Mary Poppendieck on December 2006 at Google. It is called "Competing on the Basis of Speed". She presents many of the Lean concepts as it applies to software. I will dive into how I got into Lean thinking later, but thought this would be a good starting place for those unfamiliar with the concepts.