Moving On

To my readers,

When starting this blog back in February, I had no idea where life would take me. In late February, I attended a 2-day class on Lean Software Development. While I had read the Poppendieck books and had made the connection between Lean and Agile, I was eager to get more perspective.

I attended the course at Net Objectives headquarters in Bellevue, Washington (a short 3 hour drive away). Alan Shalloway, the CEO, conducted the course. I had found out about Net Objectives through their podcast called Lean Agile Straight Talk. I already knew the podcasts were very good, but the class exceeded all expectations. Not only was Alan very inspiring and knowledgeable on the topic, but I felt right at home with the material and soaking in every bit of knowledge from Alan. While there, Alan and the rest of the staff that I got to meet left a big impression with me.

Several weeks later, I got an email from Net Objectives wanting to do a blogging project with me. They had been impressed with Leaning towards Agility and wanted to collaborate together on something. It was at that time that I expressed my interest in seeing if there was a longer term engagement available. While there wasn't anything available at the beginning, we kept the conversation going while building a relationship.

I am happy to announce that I will be joining Net Objectives in mid August, as one of their Senior Consultants. Initially, I will be an Agile Mentor/Coach for their clients and then hope to expand into other areas of the organization. As part of that, I am retiring this blog and you will see me soon on the Net Objectives Thoughts blog.

I know it was a short time, but I appreciate the support I received by you. I am not going away by any means and you can expect me to be even a greater force in the Lean and Agile community in the months to come. Thanks!

List of Lean or Agile Community Groups

Here is a list of both Google and Yahoo groups that have some focus on Lean and/or Agile Development. You will need to sign up with Google and Yahoo to join these groups. Most of these groups are very active and you will find it's a good resource for both newbies as well as seasoned agilists. Enjoy!

Google Groups:
Agile Tangents (http://groups.google.com/group/agile-tangents) - Good overall discussions around all things Agile

Yahoo Groups:
Agile Project Management (http://finance.groups.yahoo.com/group/agileprojectmanagement/) - Focus on the management side of Agile projects

Test Driven Development (http://tech.groups.yahoo.com/group/testdrivendevelopment/) - Everything you want to understand about this practice also known as TDD

Scrum Development (http://groups.yahoo.com/group/scrumdevelopment/) - Emphasis on just Scrum brought to you by the Scrum Alliance

Lean Development (http://tech.groups.yahoo.com/group/leandevelopment/) - Focus on the content of the Lean Development books by the Poppendieck's

LeanAgileScrum (http://tech.groups.yahoo.com/group/leanagilescrum/) - A newer group started by Alan Shalloway from Net Objectives. Brings together Lean, Agile and Scrum discussions in one place and talks about how they work together.

If there are other community groups, please let me know by responding via comments. They can be other Google or Yahoo groups or other types of communities that focus on Agile or Lean.

Good document on Scrum

Graeme Matthew has provided an in-depth look at Scrum. You can get the document here.

Overcoming Fears with Collaboration

Since Agile is so team-centric, collaboration between team members and customers is essential for success. However, for some people collaboration isn't an easy thing. There may be some fears that need to be overcome to fully see the benefits of collaboration.

Fears fall into these categories:

1) External Fears - The organization may have fears of being too transparent to their customers, competitors or other outside interests. Sharing too much information could be considered giving away company secrets to your competitor. Having your customers talk with other may uncover issues they have with your organization. Your customer may have information you might not want to hear (but should).

2) Organizational Fears - Executive management is supposed to drive the long term vision and strategy of the company. Operational teams are supposed to implement this strategy. But what if the two groups should clash? What if the operational teams come up with ideas that will cause a need to change the direction of the company? What if strategy is on a "need to know" basis and executive management doesn't give enough information to operational teams? What if management doesn't trust operational teams? What if operational teams don't trust management?

3) Individual Fears - Perhaps the individual hasn't had the education and skills developed that encourage collaboration. Most people grew up going to school where the teacher talks and everybody listens. The teacher gives you the homework and you follow the instruction. No coloring outside the lines. No challenging ideas. Speak when spoken to. Then, that person starts their career where the manager talks and you listen. The manager gives you tasks and you complete them. Same structure, but highly different than a collaborative environment.

So, how do you combat these fears?

1) External Fears - Managers worry about transparency because they think that outsiders don't know what is going on in the company, but in this Information Age the opposite is true. Customers will talk with other customers. Competitors will find out what you are doing. The competitive advantage is no longer how you hold on to information, but how quickly and better you can implement those ideas. Once you understand this, you will be more open to knowing what is out there. The more transparent the organization can be in an collaborative environment, the more accurate you will meet the needs of those "outsiders". The competitor who is most open and both listens and responds to what is being said, the better they will be.

2) Organizational Fears - Collaborative environments have one thing in common - an environment of high trust and respect for each individual. Strategies need to be shared. Everybody may have knowledge on any topic so every topic should be shared. People need to understand the vision. Most of all, everybody in the organization should help influence where the company needs to go. It's not just Executive management's role, but everybody who has a stake in the company.

3) Individual Fears - New people have the most opportunity for collaboration because they come with a fresh and naive perspective. No questions are "dumb" ones. Organizations need to encourage new employees to contribute in any way possible as soon as possible. There needs to be retraining and rethinking about how individuals should contribute. Some people are more comfortable in a verbal discussions. Others need time to digest information then respond. Yet others feel more comfortable in "tweaking" existing ideas instead of coming up with new ideas. Others like to respond in writing. You need to provide an environment that plays to each person's strengths and allows them to participate.

Pipelining is not Lean

LeadingAnswers has a thought-provoking post today called "The Pipelining Anti-Pattern". Here's how Mike describes it:

If you have analysts working ahead of development, or have testers working significantly behind development, then you may have “Pipelining” problems. Pipelining is the term used to describe the situation when business analysts are working ahead on the requirements for a future iteration; the development team is working on the current iteration, and the test team is engaged on a previous iteration. In some circumstances analysis may be several iterations ahead and testing several iterations behind. To some people this may seem an efficient use of resources with each group running at their optimal speed, unfettered by the co-ordination constraints of different groups. However from an agile and lean perspective this is problem, a bad-smell that needs fixing.

Here are the problems with pipelining:

Three teams not one – in a project where pipelining is occurring we do not have one cohesive team we have three teams (or more). It is hard enough co-ordinating the members of one team towards a common goal aligned to business benefits. When there are three teams it is just too easy for people to claim that they did their bit and problems lie with other groups. Yet, the fact remains that if the software does not meet business satisfaction then it is everyone’s problem.

Increased Work In Progress (WIP) – Requirements whether they are in the form of user stories, use cases, or formal specifications all represent work invested that has not delivered value to the business. The same goes for code, until this functionality has been tested to the satisfaction of the business it is not valuable. As the time increases between capturing the requirement and finishing the last test two problems occur. The first is classic accounting, money have been invested for no return yet and there is a risk associated with future returns. The second is that requirements decay; the longer we keep requirements around for, the higher the likelihood that they will no longer be required or will have to change.

Increased time from defect injection to defect remediation – the cost of change increases the longer a defect goes undetected. In a pipelining project, defects introduced by faulty analysis could take months to be detected in testing or user review. Fixing the problem after this period of time will entail refactoring far more code (for the work happening in the interim) than if it was detected earlier, and will increase technical debt...

Time fragmentation costs when groups collaborate – on pipelining projects the test group could be working on iteration 2, the developers on iteration 4, and the analysts on iteration 5 or 6. If a developer has a question for an analyst or a tester for a developer then task switching costs will be incurred. Task switching is the time required to respond to an off-topic interruption. We must mentally “park” our current work, try to think back to whatever it is this person who has interrupted us is asking about, recall the details, answer their questions and then resume where we left off. The problem with this is that studies have shown people, while great at many things, are generally very poor at task switching. Our recall of past details is poor, defense mechanisms in our brain designed to dull unpleasant experiences erode memories of nasty problems, and we are terrible at remembering where we parked our current thoughts. The net result is that just a few requests to recall past events really mess up our performance and yield low quality answers to the enquirer.

Throughput optimization – three or more teams running through work at their own speeds with fewer dependencies on others may seem like it is working, but from a systems perspective it is a problem. Processes running at different speeds require work buffers to avoid running out of work and they build piles of unfinished work. This inventory is waste in the system and leads to scrap and more rework as processes change and partially completed work needs to be updated.

I really think Mike is on to something. I have seen this struggle in our own product teams. Read more of the post to understand this problem as well as see how Mike addresses it.

A Simple Look of Lean and Agile

Somebody asked me if I were to talk about the relationship between Lean, Scrum and XP what would that be in the "simplest way possible" (At our organization, we are implementing all three). Here's how I answered:

Lean introduces the principles, or the "why's" of producing software. Focus is on maximizing customer value while minimizing waste to provide the most optimal ROI.

Scrum takes those principles and introduces the management practices, or the "when's" of producing software. Focus is on incremental delivery of working solutions that improve over time.

XP takes Lean principles and Scrum practices and introduces the technical skills, or the "how's" of producing software. Focus is on constant improvement of skills (and the underlying infrastructure) through analysis, design, and implementation of producing software.

It is my strong feeling that you really need all three to maximize your effectiveness in the development of software. Individually, each part can be implemented on it's own but you will discover something missing. Together, these really cover all aspects of developing a successful software solution.

The Exposure of Lean and Agile

A few years ago, I attended a local conference on Software Quality. We were just starting to roll out our own Agile implementation, so I was eager to hear some discussion from others at the conference. In the keynote, there was an introduction to Agile. The speaker polled the audience to see what they knew about it. I would guess that probably 75% hadn't even heard of Agle, and out of the other 25% only a few were actively using Agile practices. There was absolutely no discussion around Lean development as well. I was sorely disappointed that there wasn't more of a buzz, as I could see great potential in adopting Agile. This was 2004.

Last week, I attended a one-day seminar on Collaboration. This was by no means a technical conference, in fact there were many people who weren't even in the software industry. For those that did develop software, they were more of the project manager or program manager types. I surely didn't expect much to be talked about Lean or Agile. I was VERY surprised. It seemed that most people were actively using some Agile practices, and those that were not were eagerly listening to others. It seemed most talked had something around Agile or Lean, or the discussions following the talks would discuss either one. There were a few who had already made the connection between Lean and Agile. Most surprising of all, there was much discussion with the people outside of the software industry about how Lean and Agile could be adapted to work outside of software development and be used for other types of projects! Now, in 2007, people are seeing that potential for their own.

Talk about a transformation in such little time! It's an exciting time to be involved with software development.

Is something missing in the Agile Manifesto?

Brian Marick believes so. His post, "Six years later: What the Agile Manifesto left out" explores some missing values that aren't readily apparent by reading the Manifesto but are essential to have to be successful in Agile. Here's a summary of it (I've made bold the values that he believes is missing):

Now Agile is more respectable, a safer choice. The challenge isn’t so much getting a chance at Agile as it is executing once you’ve gotten the chance. There, the values of the Manifesto are not so helpful. Except for “individuals and interactions”, they face outward from the team. They don’t strongly apply to the relationships between team members, between the teams and their environments, and between the teams and their code. That’s a problem, because it seems to me that many new Agile teams are not executing. They’re floundering.

What should teams do with the time they’re not spending going too fast? They should invest in one of the four values I want to talk about: skill. Two skills that apply to pretty much any software project are refactoring and programmer testing (test-driven design). Those are skills that require a great deal of discipline. It’s awfully tempting not to write a test when it’s harder than writing the code, or to refactor some icky code you stumble on when that would mean blowing your estimate. Some of XP’s practices help with discipline. Pair programming turns what could be a solitary vice into a social act: you and your pair have to look at each other and acknowledge that you’re about to cheat. Peer pressure comes into play, as it does because of collective code ownership. Someone will notice the missing tests someday, and they might know it was your fault. Maybe the key driver of discipline is the practice of creating working software—running, tested features—at frequent intervals. If you’re doing that, you just don’t have time for sloppiness. You have to execute well.

I’ve observed that a characteristic of a good code base is that it’s habitable, that changes are comfortable, that the process of programming is one of ease. Agile teams suffer because they don’t think that wanting to make their lives easy is a relevant value, but it should be. There’s absolutely nothing wrong with finishing up a story by spending a half hour or so tinkering with code you’ve touched, just improving it—doing the software equivalent of hanging power within easy reach because the power strip on the desk is always annoying you.

I think Agile is suffering today because these fundamental values didn’t get written down and are too easily forgotten. As Agile moves into bigger companies and into less adventurous ones, those undocumented values are getting watered down. If that continues, I’m afraid Agile will be this decade’s fad that changes nothing. That would be sad.

Read more in his post.

Can CMMI and Agile play well together?

Brad Appleton on his ACME Blog has a very extensive list of links that say "YES". As a follower of CMMI in the past, this will be interesting reading to me to see what others are thinking.

Metrics in the Lean and Agile World

I ran across an interesting list of metrics from this site. The author of the post had gathered these in a brainstorming sessions with his organization on defining potential metrics on how they were doing with Agile. The author also poses a question to his readers on what metrics could be used by the team to possible deceive management. It's an interesting read. Here's the list of metrics that they came up with:

Short Iterations - Planning Game, Variable Scope:
* Relative velocity (measures accuracy of estimation and amount of developer time spent on other activities)
* Number of stories planned for the delivery vs number delivered
* Overflow of stories (measures accuracy of estimation)
* Degree of story change (measures activity in the business area and stability of business process)
* Number of problems fixed from previous iteration vs number of new stories implemented

Continous Integration:
* Time to build
* Number of successful/failed builds

Available Customer:
* Hours spent with developer/customer per week
* Speed of feedback vs resolution
* Number of raised features vs number of implemented features

Refactoring:
* Lines refactored
* Time spent refactoring vs new code
* Most frequently changed files
* Number of files changed per feature

PM Tools:
* Effort/time spent per feature
* Actual effort vs expected effort

Frequent delivery into Production:
* Amount of defects not caught by test/review

Regular demos for feedback:
* Degree of incorporation of feedback into iterations (measures accuracy of story gathering and accuracy of implementation)

Lessons learned for each iteration:
* Traceability of lesson as applied to future iterations/practices

Test-Driven Development:
* Degree of test case coverage
* Test case failure rates

Many managers would look at this list and think it's pretty good. In fact, they probably would think of some other measures to add to the list. However, what is the return for that investment (ROI)? Some of these metrics might be easy to gather, others could require more tools or manual effort to get. Even if you do get the information, how are you going to use the information? If you don't know how to interpret the data, you may find yourself managing the wrong things.

For these reasons, I have always struggled with finding useful metrics that are easy to gather and are promoting positive behavior. Metrics seem to be used more to punish bad behavior than to encourage better behavior. Lean software development advocates are pushing to measure up. What does that mean? Metrics are typically managed at too low a level in the organization. Therefore, companies come up with dozens of metrics (similar to a list like above) and ultimately get consumed with too many things to keep track of. Therefore, why not find metrics that can give you just enough information that measures things in terms of customers and deliveries? What if these metrics would encourage position behavior while helping you know where to look in more detail for inefficiences?

Following are the measures that I am considering in our adoption of Lean and Agile. Note, I haven't tried these out yet as the focus right now has been more on getting the practices and principles down before looking at efficiencies. However, based on my reading and understanding of both Agile and Lean here is my list:

Story Cycle Time (Measuring One Piece Flow): Average time from Customer Request to Customer Implementation for each Story. Encourages smaller pieces (Stories) that bring value. Encourages finding ways throughout organization to deliver those pieces faster.

Iteration Efficiency (Measuring Value vs. Waste): Percentage of Time spend on Value (Customer Features) vs. Waste (Defects, Refactoring, anything else the customer doesn't believe provide value). Encourages those activities that the customer sees value. Discourages those activities that the customer sees as waste.

Iteration Return on Investment (ROI) (Measuring Value vs. Cost): Ratio of Total Value (Customer and Business Value) against what it took to implement stories. Encourages the prioritization of things based on not just value but against the cost of implementation. Confirms to the team that they are working on the most important things.

Story Acceptance Measure (Measuring end user acceptance): Average acceptance (user rating) of each Story after it has been implemented. Encourages making sure the customer needs were really met. Encourages quality of solution from a customer standpoint. This may be the most difficult to gather because it requires polling at least a good sample of customers.

What do you think? Did I miss any measures at the organizational level?

Team Power

In both Agile and Lean circles, there is constant emphasis on the team rather than individual efforts in resolving issues, making decisions and suggestions for improvement. "Two heads are better than one" rings true! Over at LeadingAnswers blog, the latest post called "Team Solving: The Under Utilized Power" demonstrates this fact. Here is a synopsis of the findings:

Power #1: By asking the team for a solution we inherit consensus for the proposal. The fact that the solution came from the team and not me, meant I did not have to sell it, it already had support. It is easier to steward a sub-optimal solution with good support to a successful outcome than build support for an optimal solution and ensure its successful execution. If challenges arise and people subconsciously ask “who’s bright idea was this?” the answer comes “Oh yeah, it was ours” and they are more likely to continue.

Power #2: Accessing a broader knowledge of the facts. Team members are closer to the details and bring additional insights other than your own. I did not know that the business folks bowled, this was collective knowledge (Chris knew three users bowled and Julia knew two others did also) together we found a new piece of useful information. Asking the group taps the collective knowledge of the team.

Power #3: Solutions are practical. Anyone who has worked hard to craft a solution only to be told “That will not work here because...” will know how frustrating and disheartening these words are. Team sourced solutions have been vetted for practicality and, because created internally, solutions found for implementation issues.

Power #4: When consulted people work hard to generate good ideas. The simple act of asking for suggestions engages team members beyond the role of “Coder” or “Tester”. People appreciate having their inputs valued and generally work hard to create innovative and effective solutions. Treating workers as interchangeable resources is a poor model inherited from the command-and-control methods of the industrial revolution. Leading companies such as Toyota and 3M recognize that their best ideas come from inside their companies and we need to make use of this intellect. It is partly due to these methods that these companies innovate better, have higher quality products, and better labor relations.

Power #5: Asking for help shows confidence not weakness. Asking for ideas and solutions to problems is not a sign of incompetence or an inability to manage. Just because we ask for input it does not follow we are dumb. Instead it demonstrates valuing the opinion of others, and being thoughtful. In essence it demonstrates how all problems should be tackled, which is the next power.

Power #6: Seeking ideas models desired behavior. Project managers have a leadership role of “modeling the desired behavior” i.e. behave as we wish others to behave. If we stay silent, make decisions with incomplete awareness of the facts, and do not ask for help when we need it, what message is this sending to the team? Well, it is an obvious message that we expect team members to behave the same way and work in isolation. Lots of time and money is wasted on team building activities that are then eroded my management-in-a-vacuum.

Here are also some cautions where the team can actually lose power:

Caution #1: Real problems - We should use this on real problems only, not on which brand of printer toner to buy. Remember we are modeling desired behavior and if people take it too far and consult their team mates when deciding what color dialog box to create; no work will get done. It is a tool for when you are stuck and the problem is important.

Caution #2: Poor team cohesion – If the team is fragmented and has opposing groups then resentment for “fixing their problems” will undermine the process. We need to get the team aligned for team solving to be most effective.

Caution #3: Team and Project Changes – Over a long period if a significant portion of the team changes, we need to re-canvas the team to ensure they are still on-board with the approach. Exercising the bright ideas of others is nearly as bad as not being consulted; we need to ensure people still agree this is a good policy. Likewise if the project changes significantly, we need to checkpoint in light of these new facts and get the team to review the approach.

Caution #4: Follow-through – once you ask for solutions make sure you follow through on them with execution. It is pretty demoralizing to be asked to work on a solution and then see it wither. It is fine to go back with implementation problems that need to be solved, but don’t waste peoples time by asking for input and then ignoring it.

Exploring what makes Scrum successful

Alan Shalloway from the Net Objectives Thoughts blog is digging a little deeper into the "soul" of Scrum to truly understand why Scrum works in his post "Challenging why (not if) Scrum works". Please read his post in more detail, but here is some reasons he came up with:

I have repeatedly heard that “Scrum succeeds largely because the people doing the work define how to do the work” (see From The Top by Ken Schwaber for his full text). However, I do not think that is even a third of why Scrum works - and be clear, I am certainly not questioning that Scrum works - I just want to get at the why it works.

OK, so why does Scrum work? Remember, I am not challenging that it does, just asking why. I would say Scrum works for the following reasons:

* the iterative nature of Scrum development (including the re-assessment of priorities between iterations)
* the workcell nature of the Scrum team (everyone who needs to work together is together)
* the fact that many Scrum teams are given the chance to co-locate
* that most Scrum teams work on one project at a time
* that many old, cumbersome checkpoints/forms/status reports that used to be required are abandoned because the team is doing something new
* a focus on defining acceptance tests concurrent with requirements
* the availability of a product champion (product owner to Scrum enthusiasts)

I think this is a really good list, I would also add a couple of items to it:

* frequent communication of status and removing of impediments through the Daily Standups
* continue emphasis on feedback and improvement through sprint reviews
* the team nature of the Scrum team (everybody on the team should understand and commit to the sprint through sprint planning)

For those that are using Scrum, you are well familiar with these benefits. For those that have not moved to Agile or they are having problems with their practices, consider looking at Scrum to gain these benefits. Scrum by no means is a "silver bullet", but it has already proven to be very successful where it has been implemented.

Balancing Perfection against Innovation

Jeffrey Phillips from Thinking Faster is struggling with finding the balance between the desire to have perfection with products against the desire (sometimes by the same people) to provide innovative products. His post, "Trading off perfection and innovation" shows us a comparison of the two. Here's just part of his thinking on this (read more of his post to get everything):

One one hand, we have the constant demand for perfection. Six Sigma, Lean, eliminating mistakes and errors to create an incredible, consistent, useful product or service. On the other hand, the need to innovate, to create something new and take new risks to dramatically change a product or market. Both approaches are correct, and relatively mutually exclusive. What does a manager do?

When choosing perfection, we make a series of assumptions about the product or service, its competitors and our ability to keep up with the market. Once a product or service is perfected - it it every really is - even a small change to the process or product can cause major disruptions for a customer, so we prolong the life of any product or process and minimize changes to achieve perfection. If you can dominate your market, or if perfection is important to your customer, then this is a great strategy. However, it assumes that there will be little or no disruption in the market, and that the customer demands perfection over innovation and new capabilities.

When you choose innovation, on the other hand, you will not be able to provide a perfect product or service. If your innovations are truly new, you can't determine in advance exactly what "perfection" is - you can only try to provide what you think customers will value and iterate from that point once you discover and uncover real wants and needs. This approach assumes that customers don't demand perfection, but do demand interesting, innovative products and services that solve unspoken or unmet needs. After all, if you can solve a problem that I have even fairly well that has never been solved before, I won't ask for perfection yet.

The real crux of the matter is that if your position is perfection, then you can't afford to make mistakes, and if you do, you need to quickly and completely correct them. On the other hand (aren't you glad we only have two hands) if your position is innovative, you can't afford not to make mistakes, and you want your customers to participate in and help you learn from your mistakes - you can't learn if you don't make mistakes.

In the end, you have to give up some level of perfection if you want something innovative. Because the very definition of innovative is doing something that hasn't been done before (or at least not that often). You also have to be able to take on risks and make mistakes as part of the experimentation that comes with innovation. I have to agree with him. Too many times, I have seen people take the "tried and true" approach to a solution just to ensure they don't cause inefficiencies in hitting dates. Even though they would agree there is a possible better solution, they aren't willing to pay the price to get there. I like Jeffrey's thinking here, your customers need to understand the tradeoffs. The only way for them to do that is to be more part of the process. For companies that can figure out how to do this, those will be the ones who provide innovative products that their customers believe is more than "good enough".

Commitment, Delivery and Customer Expectations

Since we have been on the road of adopting Agile and Lean in our organization, I have had increasing pressure from stakeholders to give them an idea of when we will deliver functionality beyond the current release. Our releases are currently around 4-6 months at this point, so they are asking for things that will be delivered towards the end of the year and into next year. One stakeholder wanted to go out even further than that (up to 24 months in advance)!

At first, I thought that they just wanted to get an idea internally by developing a high-level product road-map to help us with our priorities. However, my thoughts were that this road-map wasn't going to focus on dates. Why? Because that information would not have any sense of reality. This information is at best high-level and with that not enough knowledge and information to make a better prediction of what could be delivered. That better prediction won't happen until we do release planning for the release that we are considering the future functionality. Even with that said, we really won't be able to commit to dates until we get really into the details of the work during iteration planning.

So, there are really three levels of precision: "fine grain" in the current iteration (90% or greater chance of stories in the iteration getting done), "coarse grain" in the current release (60% or greater chance of stories in the release getting done - this number could go down if our releases were even shorter), then "swags" in future releases (40% or less of a chance of stories getting done in the expected time period - this number going down the further out you go). These aren't exact numbers but you get the idea.

What really worried me is that they wanted these long-term dates so they could make commitments to key customers on when future functionality would be delivered. When I discussed my concerns with the stakeholders and talked about the various levels of precision, they believed we could actually get better at our precision if we just planned a little more. However, that would become waste in Lean terms because we would have a lot of work (inventory) "in-progress", plus what we know now about something that will be started six or more months in the future WILL change once we get there. Even with all of that said, my greater concern was making commitments to customers before we can make commitments with the development teams that they can deliver. It should be the other way around in my opinion. Let's figure out what we can commit to and deliver, then start setting expectations to customers.

One of our vendors lives by this philosophy. If we ask for an enhancement, if it isn't something they are committing to for the next release, they will give the answer "we'll put this in our backlog of priorities and consider this for a future release". Then, they will focus my attention of things of value that are coming in the next release and get my feedback on them. Even if my particular request isn't there, I find great value in what is about to be delivered. Sure, there may be some features I don't plan on using or care about but there are usually features that I care very much about. As a customer, this vendor is setting my expectation based on what they know they can deliver, then based on what they hope they can deliver. Hopes don't cut it! If they had promised me a future date, and then miss that date I would have been much more frustrated as a customer. I would rather not know until they were really confident they could hit that date. I appreciate that this vendor has delivered what they promised and has built a reputation with me that they are making constant improvements to their product. Even if those improvements aren't everything that I have asked for!

This is just one of the bad habits we need to break by moving to Traditional development over to Agile. It wasn't that we were better at providing estimates and hitting dates using Traditional development, because we weren't (especially for features out several releases). Sure, we could try to hit those arbitrary dates by cutting scope, freezing scope changes, adding people, working overtime, cutting corners on quality, and other "tricks of the trade". However, would we really provide what the customer really wants? Isn't the goal of Agile to provide what the customer finds value, even if the cost of that value is sometimes difficult to determine? If the quality isn't there, they don't want it early. If the functionality doesn't really do what they expected, they don't want it early. Why this importance of hitting dates because of commitment to customers, when in the end the customers will be disappointed because what they got in return missed their expectations? Let's commit to dates first as a development team and "internal" customers, then set expectations to our "external" customers as we deliver what is important to them.

I believe we got into this dilemma because we developed a reputation of over-promising and under-delivering (because of the problems with Traditional development). Plus, what we would deliver (usually 8-12 months) didn't really cut it with the customer and left them disappointed and frustrated with how long things take. Therefore, their lack of trust in hitting dates causes them to even push harder for us to commit to completing functionality that is long overdue. We need to build trust again with our customers by delivering high-value and high-quality features more frequently in order to gain back a reputation of under-promising and over-delivering. This will take time, but in the interim we need to change the way we have set future expectations with our customers. This may cause some short-term frustration, but will quickly change as customers will change their focus as well on what is being delivered then what is promised to be delivered. After all, isn't that what really counts?

The Case for Story Point Estimates

I have been a fan of story points ever since I attended a seminar several years ago where Mike Cohn presented the concept. I never really trusted other estimating practices such as function points and time-based estimates. Why? Software development projects are rarely similar from project to project, yet these practices focused entirely on past experience. Therefore, to get a "reliable" estimate of time for every new project you needed to gain a lot of experience. In other words, you have to figure out up-front how you will do the work. Not only does this take a lot of investment up front, it also does not account that the work you do later could change based on the work you do now. The estimates assume that nothing will change in the effort of doing the work, which is definitely not true in the Agile world. What I like about Story points is the focus is on the relative size of "things", then how they will be accomplished. As Mike would say, "Estimate size now, derive duration later".

Even though I am sold on story points, I have had much difficulty selling the concept and value to others. I'm having an easier time with the development team, but really struggling with stakeholders/product owners. These groups always want to know when things will get done as soon as possible. Story points in itself don't give you the answer, however they help determine velocity of the team which eventually will help you better predict what could be part of a release. For various reasons, it is too much of a paradigm shift for some to accept an estimate that isn't time based, yet they have much comfort of the team throws out some arbitrary number early on a release without much knowledge of how events are going to unfold.

It looks like I'm not the only one dealing with this struggle. Tobias Meyer, on his blog Agile Thoughts, addresses some of the arguments others have had about using Story Points as estimates. Following are the arguments, read more on his post "Estimation: Time or Size" on how he address each arguments. Good stuff!

Argument 1: Estimating in hours allows a developer to measure his estimate against his actual, and use that data to improve his estimates in future.

Argument 2: Our customers/product owners don’t understand story points; they need to know how many hours developers are working so they know how much the work will cost.

Argument 3: Story Points will map to time anyway, very soon we’ll see that (e.g.) one story point is worth 2.5 hours, so it is better to skip the intermediate step and just measure in hours.

Argument 4: Story Points don’t allow you to improve your estimation techniques.

Agile or agile, Lean or lean?

Recently, I have seen a trend where other bloggers and authors are having difficulty or are trying to define the difference in using Agile (capital-A) vs. agile and Lean vs. lean. Some would say that by putting a capital in front of it that it means you are focusing on a particular methodology or defined Agile process. So if you are referring to something like XP or Scrum, you should use Agile. Otherwise, if you are talking about principles and particular practices, you should use agile. Others don't want to put a label on "Lean" or "Agile" because they are afraid I guess of watering down or maybe commercializing those terms? Other people would believe that you should use capitals in some situations and not in others.

Frankly, I just don't get it. All this brings is more confusion to the community. To me, putting a capital on Lean and Agile allows my readers to know that I am talking about the values, principles, practices, tools, measures, etc of a way to develop software differently. By not putting the capitals, I don't want to confuse people with the generic terms of lean and agile which are:

lean

1. To hang outwards.
2. To press against.
3. (of a person) slim not fleshy or having little fat
4. (of meat) having little fat
4. Having little extra or little to spare; as a lean budget
5. (of a fuel-air mixture) having more air than is necessary to burn all of the fuel; more air- or oxygen- rich than necessary for a stoichiometric reaction

agile

1. Having the faculty of quick motion in the limbs; apt or ready to move; nimble; active; as, an agile boy; an agile tongue.

Agile and Lean are states of mind, are a way to develop software, have common values and principles but many practices and tools to implement. agile and lean? They don't tell me much. I'm going to use capitals!

Addressing Technical Debt

What is Technical Debt and how to you manage it? These were two questions that is addressed over at the Lean Software Engineering blog.

In the first post, called "7 sources of technical debt" Bernie creates his list of where technical debt comes from:

1. Undiscovered bugs, failing tests, and open bugs in your database
2. Missing test automation (unit, feature, scenario, or system)
3. Missing build and deployment automation
4. Scenarios with incomplete user experiences
5. Code that’s too difficult to understand
6. Code that’s too difficult to extend
7. Code that’s isolated in branches

In the follow up post, called "7 strategies for measuring and tackling technical debt" Bernie provides his list of potential remedies to reduce or eliminate this debt:

1. Bugs. Measure and shrink your cycle time from when code is first written until it is tested (TDD is perfect for this, of course). Treat open bugs as work in progress (WIP) — and too much work in progress is evil. Constantly measure and manage WIP down with strategies like bug caps (no new features until existing bugs get below a defined bar).

2. Missing tests. Measure code coverage. Have a clear definition of “done” at each scale of your product: unit, feature, scenario, system. Treat units of code as incomplete until code, tests, and documentation are delivered as a set.

3. Missing automation. Make build and deployment first-class features of your project. Deliver them early, and assume (like other features) that they will constantly evolve with sub-features over the course of the project. Measure time-to-build, time-to-deploy, and time-to-test — including human time expended each cycle — and then drive those numbers down by adding automation work to your backlog.

4. Incomplete scenarios. Assign scenario ownership roles among the team. Have a clear definition of “done” for each scenario. Pre-release working scenarios to select customers as soon as possible, making special note of these bugs. And of course, minimize the number of scenarios each team focuses on at once (one at a time, if possible).

5. Not understandable. Like most open source projects, generate docs directly from the code (literate programming), including why the code does what it does. Systematically conduct code reviews. If the team can’t afford to review every line, always review a sample set of functions from each developer and component. Ask reviewers to rate code for understandability, track this data in code comments or a spreadsheet, and add workitems to the backlog to clean up those most needing improvement.

6. Not extensible. Encourage design pattern thinking. Buy your team a library of design patterns books and posters to help form that common vocabulary. Look at how open source projects tend to build big things out of many small, independent pieces — treat every component as a library with its own API and a clear (and minimized) set of dependencies.

7. Not integrated. Minimize the number of branched codelines, whether they’re formally in source code control, done on the side for customers, or sitting on developer’s disk drives. Unintegrated code is debt — it has work and surprises waiting to happen. Strive for continuous integration. This isn’t to say don’t use branching — even the smallest project should think about their mainline model — but always keep everything in source control, then track and minimize the number of active branches.

This is by no means an extensive list as I'm sure there are other contributors of debt and possible solutions. However, I agree with Bernie that these are the most common things to be on the lookout for. Just think how better the quality of software, and the ability of a team to deliver new features can be maximized if teams would see things in terms of debt!

Read more in both of these posts.

Participate in the making of a new book on Agile

James Shore, a local consultant here in Portland along with Shane Warden, are writing a book called The Art of Agile Development to be published later this year. If you want to get a glimpse of what the book will be about, they have a working draft on Jim's website.

Here's a portion of the preface to give you an idea of what the book is about:

We want to help you master the art of agile development.

Agile development, like any approach to team-based software development, is a fundamentally human art, one subject to the vagaries of individuals and their interactions. To master agile development, you must learn to evaluate myriad possibilities, moment to moment, and intuitively pick the best of course of action.

How can you possibly learn such a difficult skill? Practice!

Most of this book is an étude. An étude is a piece of music that's also a teaching tool. Études help the artist learn difficult technical skills. The best études are also musically beautiful.

Our agile étude serves two purposes. First and foremost, it's a detailed description of one way to practice agile development. It's a practical guide that, if followed mindfully, will allow you to successful bring agile development to your team—or help you decide that it isn't a good choice in your situation.

Our second purpose is to help you master the art of agile development. Team software development is too nuanced for any book to tell you how to master it. Mastery comes from within: from experience and an intuitive understanding of ripples caused by the pebble of a choice.

We can't teach you how your choices will ripple throughout your organization. We don't try. You must provide the nuance and understanding. This is the only way to master the art. Follow the practices. Watch what happens. Think about why they worked... or didn't work. Then do them again. Watch what happens. What was the same? What was different? Why? Then do it again. And again. Watch what happens each time.

At first, you may struggle to understand how to do each practice. They will look easy on paper, but putting each practice into action will sometimes be difficult. Keep practicing until it's easy.

As it gets easier, you will discover that some of our rules don't work for you. In the beginning, you won't be able to tell if the problem is in our rules or in the way you're following them. Keep practicing until you're certain. Then break the rules.

Parts I and II of this book contain our agile étude. Part I helps you get started; Part II provides detailed guidance for each practice. Parts I and II should keep you occupied for many months.

When you're ready to break the rules, turn to Part III. A word of warning: there is nothing in Part III that will help you practice agile development. Instead, it's full of ideas that will help you break rules.

One day you'll discover that rules no longer hold any interest for you. After all, agile development isn't about following rules. "It's about simplicity, and feedback... communication and trust," you'll think. "It's about delivering value—and having the courage to do the right thing at the right time." You'll intuitively evaluate myriad possibilities, moment to moment, and pick the best course of action.

When you do, pass this book on to someone else, dogeared and ragged though it may be, so that they too may master the art of agile development.

But just don't take a look at this draft of the book, help the authors out! You can participate in the review process by joining the art-of-agile mailing list! I'm sure that they will appreciate the help and you can feel good in contributing back to the Agile community!

Video: Agile Testing

Here is another great video. This time, the topic is Agile Testing. I met the speaker, Elisabeth Hendrickson, a few months ago at Agile Open Northwest here in Portland. She led a couple of sessions, and they were some of the best attended and interesting ones of the bunch. This lady has ENERGY and a PASSION (BIG CAPS HERE) about testing. Before meeting her, I didn't know somebody could care so much about testing software. This video is from December 2006, but is still very relevant. If you are transitioning to Agile and have been used to having a separate testing group, this video is for you. If you are still struggling with how testing should be done on an Agile project, you will get something of value. Elisabeth does a great job helping understand the shift that needs to take place to do testing differently within the context of Agile. Here is the video:

Changing the way you do long term planning

We are in the process of implementing a pull scheduling system for each of our product development teams. No longer do we think in terms of projects, but in terms of releases and iterations. No longer do we figure out the entire scope in detail of what will be contained in the product, but we determine the details when we are ready in a particular iteration in a release. No longer do we have rigid control of scope for the entire project, but we try to contain what we do in a 2-week iteration once it has been planned and committed by the team. No longer do we try to figure out exactly what we will do in the next 1-2 years, but we "pull" enough work to determine the next release when we are ready to think about it. No longer do we wait for customer feedback only at the end of the project through alpha and beta cycles, but we get internal and external feedback through demos at the end of each iteration. No longer do we conduct vague and long weekly project status meetings, but we conduct quick and concise daily standup meetings. Our world has definitely changed, but in the minds of our executive stakeholders they have struggled a little with it. In particular, if you are only planning within short iterations and the current release cycle, how do you do long-term product planning for customers that want to have some idea of a delivery date for something "down the road"?

I have no problem in planning long-term if the planning is focused more on priorities than dates. Though the team is pulling just enough work for the next release, the more work that is prioritized and thought out (at least to a high level) will make the release and iteration planning go much smoother. And, actually I don't even have a problem committing to a particular date for the next release or any iteration in the release because those are fixed. However, committing to particular scope outside of the current iteration is something that I find challenging. Why?

Because we know only look at the details of the release within the current iteration, we haven't given enough thought about the remaining stories in the release. Sure, we have done some high-level estimates but we have learned that what you think you are going to do after thinking about it for a few minutes is much different when you think about it for a few hours in iteration planning. Until you really start looking at particular tasks and tests that the team will do in fulfilling each story, you don't really have an accurate estimate. And even with that said, you may find surprises within the iteration after planning.

The other challenge is that you are dealing with change in a whole different way. When you find a defect, you don't "log" it in a repository automatically because you don't have the time to fix it during the project. Instead, you deal with defects as they happen. When you show functionality in a demo, you don't log future enhancements automatically to be considered in future projects but you make appropriate changes if they are valuable and while they are fresh in the team's minds. When new issues come up, you create new stories to deal with them if they are deemed the right time. All of these things change scope, which changes what is done in a particular release. This changes the expected delivery dates many times. Which makes it very difficult to predict the scope of a release, and makes it almost impossible to predict when new functionality will be available in 12-18 months.

Now, this may sound bad initially to your stakeholders, because they want to make promises to customers as soon as possible. So, this may sound like Agile and Lean is the wrong way to go right? While I can't predict what will be done in 18 months, I can show you progress every two weeks. This progress will have higher quality because it has been tested and could be released to customers if desired. What we work on has been determined the most important to our customers, so this is a good thing. The best thing of all? The fact that we make sure that every deliverable, minimal useful functionality, really is what the customer wants because they have been more involved more frequently. So does it really matter if we try to fix scope, resources and schedule but having such as strict change control process if in the end we don't deliver what is really wanted?

So, what I have agreed on with the stakeholders is that we will go out beyond a release but strictly from a planning and goals perspective. At the point where it is determined that the goals are not obtainable in the process, I will make sure that the stakeholders are notified and can adjust the scope or define new goals based on this information. What's interesting is that I will never have to have that kind of conversation because the stakeholders will be involved in every part of the process. They will see progress being made, they will set priorities of work, they will attend demos and make changes, and they will see the impact of those changes. In the end, they will learn that they can draw the "line in the sand" when they feel most appropriate. Based on the confidence that they are gaining with the teams, they will change the long-term discussion with the customer to what is being developed than what is being planned. When that isn't enough, they can give the customer an idea of the priority of work being considered. If the customer feels that the priority isn't meeting their needs, they can voice their opinion. Then the stakeholders can determine what to do with that feedback and make priority changes as necessary. As long as customers see steady progress with frequent releases that are meeting needs, they should be satisfied with this approach.

Video: Agile Estimating and Planning

I guess this week is the week of video postings, as I have discovered some more great ones to share with you. Today's is from Mike Cohn. I went to a seminar a couple of years ago on the same topic and presenter as this video. There is great content here, especially if your team is struggling with estimating and don't know about story points and planning poker concepts. Mike's presentation style is also engaging and entertaining. This presentation is is two parts, click the videos below to watch each part.

Part One



Part Two

Video: Scrum at Google

Jeff Sutherland provides an excellent video discussing how the Google AdWords team adopted and adapted Scrum to fit their team and the Google environment. As my organization has been going through adoption of Agile, I saw similar "lessons learned" as described in the video. I think there is a lot of valuable knowledge to be gained listening to Jeff. Enjoy!

Another View of Lean Leadership

Joe Little from Agile and Business has an excellent post today called "Leaders, Managers, Bosses and Administrators". Actually, he focuses on what leadership means in the Lean and Agile world. He refers to the Poppendieck book, "Implementing Lean Software Development: From Concept to Cash" (If I had to pick one book to understand Lean as applied to software, this would be the one!). Here's his take on leadership after reading the book:

They [Poppendiecks] raise several excellent points.

1. A team needs leadership. Which is to say, vision. Someone to inspire and someone to help them put their hearts in the game. And keep it there.

2. The project needs decisiveness. If the team has too many leaders, and the leaders squabble about decisions, then the team wastes time. The team needs to know how it will make decisions. There is a trade-off between making the right decisions, and making decisions quickly.

3. Generally, the team needs to learn to make decisions only at the last responsible moment. So, much of the decision-making is about when to make a decision. At what point have you learned enough to make a good/better decision?

4. The project needs business decisions and technical decisions. This is very true. So the team needs business people and needs technical people who are ready and able to make those decisions. And, preferrably, business people who understand technology and technology people who understand business (and the customers).

5. And there are many other types of decisions to be made too. People decisions. Decisions about whose insights to go with on specific areas. Process decisions. Decisions about who is working effectively and who is not. Decisions about how to get the team to communicate better and learn faster.

6. In Scrum, we have ScrumMasters and Product Owners. These roles are endowed with certain leadership aspects. This is different than the leadership of the Chief Engineer, which is a role Toyota uses.

7. Project managers have also provided leadership. (And we have the whole PMP, PMI thing, too.) PMs have also provided managership, bossiness, administration and other things.

8. We know that no bosses are wanted. We want all the best from every person, and a boss will only inhibit that. A boss wants to command others, and thinks he knows all. These are not helpful traits in a learning situation. (So, semantically, we are using "boss" here to represent all the bad things that a boss can be. Of course, few managers or leaders are as bad as a boss, but we all can be that way sometimes.)

Read more in Joe's post.

The Agile Team's Dashboard

Last summer, I shopped for a new car. I decided with rising gas prices to purchase a Hybrid. My goal in buying this car was to improve my miles per gallon (MPG) so I didn't have to spend as much as often at the pump. With this car, I have two gauges that I didn't have before. The first is next to my speedometer and shows me my MPG consumption (or electric if no gas is being used). The other gauge will show me the Tank average for MPG.

At first, I wasn't used to these measures and didn't pay much attention to them. Therefore, my driving habits were similar to what I had done before with my other car. I drove the car as I had driven the previous one. As I got familiar with my car, I started to explore both of these gauges. I started experimenting with how to increase my MPG usage. I discovered that if I accelerated a little slower after being stopped it made a difference. I discovered that I could coast more often when going downhill instead of hitting the gas and still maintain the same speed (but save on MPG usage). As I result, I started to see the MPG increase.

Then, I decided to not pay attention to either gauge for awhile. Just try to drive the car using what I have learned and see if I got the same results. Though I did better than when I first started using the car, I still was getting lower MPG by not paying attention to these gauges. These gauges were a good visual tool to help me accomplish my goal of higher MPG, and were a good reminder when my bad habits would come back and cause my MPG to drop.

So, why all of this talk about cars? Well, think about Agile teams. Their goal is develop as much working (deliverable, quality, meets customer needs) software as possible in set increments of time. How do they measure this? There are two gauges that most Agile teams use to ensure they focus on the goal of delivering working software.

First, the team calculates their velocity from Iteration to Iteration. Velocity measures the total amount of work that was accomplished to deliver working, completed stories by the end of the iteration. If a story is partially completed, the team does not get credit until the iteration in which the story has been completed. This encourages teams to focus on completing stories than completed tasks leaving incomplete stories. If the team is focusing on the completion of stories, as is a dedicated team of individuals, you should be able to sustain a certain range of velocity from iteration to iteration. Where Velocity gets lower, you should see why things are getting done. Where Velocity is higher, you need to see why the team was able to accomplish more. If the team pays attention to Velocity, they will begin to see better development practices (much like my driving habits improved after paying attention to MPG).

Second, the team uses a chart to track progress at both a Release and Iteration Level. The burndown chart shows how work in being completed on a "trip" by "trip" basis. At a Release Level, the chart shows how each Iteration is progressing and the team can get a good idea of what amount of scope is possible based on this information. At an Iteration Level, the chart takes the daily remaining hours for the team to be used in the Daily Standup to see progress. If the chart isn't going down as quickly as expected, this is an indication that the team has impediments that must be addressed. These impediments could be thrashing (team isn't getting to a resolution), dependencies to other stories/tasks (team needs to remove dependencies where they can), additional scope (additional stories and/or tasks have been determined necessary to get complete). In any of these cases, the team can quickly see the impediments impact to the iteration and make adjustments.

Both Velocity and Burndown charts are good gauges to ensure that you are doing what you can to deliver working software. These measures should expose you to some problems (waste in Lean thinking) that need to be resolved. Most of the time this requires a change in how you do things in order to accomplish the goal better. If Agile teams aren't using these gauges, much like my driving, they can get back into bad habits and find themselves not having great results after each Iteration.

Thinking inside the Agile Box

Pretend you are taking a week long trip with a friend. You start packing and end up with a couple of large suitcases. Your friend arrives with one medium suitcase. You start thinking, "What are they thinking? They don't have enough clothes to last the week." You arrive to your destination. When you start unpacking, your friend's clothes are all nicely folded and no wrinkles. Your clothes look like a mess and will need to be ironed. Your friend is able to put things into drawers quickly as well as the bathroom items. You have to go through your bag and it takes you longer to find things and put them in their place. However, your friend has less clothes than you. As the week progresses, you begin to see that your friend has found ways to reuse what they wear - different shirts with the same pants - yet still appear like they aren't worn. You on the other hand, find that at the end of the trip you had clothes that you didn't wear, but had to carry around. Now you start thinking, "My friend has a better way. I've got to learn from them for the next trip!"

Was the friend always this organized? Probably not, they probably learned by somebody else. Experienced travelers have tricks on how to properly fold clothes and how to maximize every spaces in a suitcase. They have also learned what's "good enough" and not to take more than they need. Even if they end up needing fresh clothes, they will end up using a local cleaners or wash their own clothes instead of taking too much. They have learned how to best put clothes and other accessories together to maximize their use.

As my development staff is asked to reduce cycle times through iterative development using Agile best practices, their initial reaction is "That sounds all good, but we have found things to take weeks or months and that won't work with this approach." They are correct, it won't. But, just because you have done things a certain way doesn't mean that it is the ONLY way to do things. We have learned how to do things as individuals. We will now need to do those things together as a team. We have learned that each person plays a particular role and has their own specialization. We will now need to remove those specializations and have people wear multiple hats. We have learned to think about everything up front is much detail. We will now need to learn to do just enough for now and think about details when it is appropriate later. We have learned to break the work and requirements down just enough to make sense for a 6-8 month project, we will now need to refine that work to make sense for 2-4 week iterations. We have learned a way to deliver software after 8 months, we need a way to deliver that same software in a portion of that time and possibly in weeks instead of months. We need systems that get tested and built at least every day if not several times a day, the systems we have used cannot handle that load.

What's great about Agile is that it isn't just "theory" anymore, people have found it to work and improve the flexibility and responsiveness of the team to provide working solutions to customers. We can tap into these "experienced travelers" and figure out how they have done it a different and better way. It just seems strange to us because we have learned how to do things differently, not necessarily better. However, if you think that Agile is just about doing things faster you will miss the point. Agile is about doing things differently in order to deliver working software more effectively. That means we need to relearn how to build, maintain and support software under this new way. Relearn skills, tools, knowledge, processes and essentially how you do your jobs differently.

You might have thought you were a great traveler and packed accordingly, until you realized that there is a better way. However, you can't just copy the better way like it was out of a cookbook. You must understand why this experienced traveler didn't to make the change. Then, you must understand how he made the change work for them. Then, you must make the changes work for yourself. Same with Agile, you don't just adopt a methodology and away you go. You must understand the principles behind it, then determine what set of best practices fit your organization. When people ask us if we are using XP or Scrum, I respond with "Yes". "Which one?", they ask. "Both" I respond. There are practices within each that we have adopted to support Agile principles. There are other practices that we have come up with that perhaps nobody else is using. That's ok, as long as we are staying true to the goals of Agile development. I will say that it has required EVERY person to think about their jobs differently. In some cases, much differently. In the end, it required each person to "think inside a different box". An Agile Box!

Where does QA fit within Agile?

Like many development shops that have been around doing Traditional Waterfall development over the years, we have a separate department called Quality Assurance. This team consisted of non-developer types that focused primarily with ensuring quality of the product through testing activities. In our case, because we have legacy products that have been developed in technologies that don't lend itself well toward test automation, these testing activities were largely done manually. Unlike the unit-level testing that was done by the developers, the QA team primarily focused on overall system testing. This included test verification of current requirements but also a fair amount of regression testing (again manually done) to ensure that
past functionality continued to work.

As with Waterfall development, QA was largely involved late in the process. Once developers felt their code was completed, they would "throw in over the wall" to QA for testing. QA would do their initial testing, and submit bugs for things not working. Until the overall product was stable enough to give to our customers (through several release cycles), it would go through several fix/build cycles. How long this would go was highly unpredictable, and therefore would cause much frustration if our initial test estimates were too far off.

A couple of years ago, I attended the local Northwest Software Quality Conference here in Portland, Oregon. This is an excellent conference and while anybody in the software industry can get something useful out of it, the primary audience are people involved in QA. As Agile started to heat up in the software industry, presentations started to show up in every conference. Given this audience however, very few had even heard much of Agile and for those that did, weren't using Agile practices. Why? Because Agile assumed (at least at that time) that the testing role and other QA activities were just another hat that developers wore. So for those that had heard of Agile, they were resistant because they feared their jobs in the long run.

I was seeing that same reaction from my own QA group, and was dealing with how to best transition that group into a valuable role of an Agile team. After many sessions with both developers and QA, I started to see the light of how QA could fill some holes that the teams were having. So, here's how QA will look in our organization:

1) QA integrated into every team - Each member of the QA team is now co-located with the Development instead of being in a separate department.

2) QA plays Testing Manager role - QA is responsible for helping the team identify how we know when a story or task is "done". They define done by developing tests with the team that ANYBODY can run including the QA person. They also determine how best to implement that test (manual or automated, which tools, etc.)

3) QA plays Process Improvement Manager role - QA is now going to lead retrospectives at the end of iterations and releases with the team. They will ensure that there is just enough process for the team to ensure quality but not too much that the team doesn't see value in the process. They are also to ensure that all action items from the retrospective get reflected in the work in future iterations and releases.

With these three implementations, we hope to build more quality into the entire process and ensure quality earlier and often in the process. While anybody can wear the Tester hat, now QA has a much broader role in ensuring quality beyond just testing.

The Top 5: Feb - April 2007

Here are the top 5 posts most visited from February 5th through April 24th:

#1: What does it mean to be Lean? - Lean is a paradigm shift from normal thinking. Learn what it takes to think in Lean terms.

#2: An Introduction - Just some background of why I created this site and who I am.

#3: The Predictability Paradox - A link to a wonderful white paper written by Mary Poppendieck discussing predictability in software projects.

#4: An Agile Approach to adopting Agile - Some advocate taking the big bang approach to adopting Agile. Not me! If Agile is all about incremental and evolutionary delivery, why can't your adoption be the same way?

#5: Deming Revisited - I tackled Deming awhile ago from a management perspective, now I update Deming's list of Fourteen Points from an Agile and Lean perspective.

Growing Your Leaders

Alan Shalloway from the Net Objectives Thoughts blog challenges how Agile has viewed management and leadership up to this point in his post "The Need for Leadership in Scrum". Here are some highlights:

There underscores the notion that self-managing teams and direction from the top are somehow opposed to each other. In practice, this may be. But that is because in practice, leadership is missing. Direction in the form of command and control is bad. Direction in the form of vision and matching people to their abilities is not. This points to one deficiency I have seen in the general Agile community, and one that, I believe, is a serious impediment to having the Agile community have success at the enterprise level. I am referring to the fact that most Agilists do not trust management to do what is right – they feel the team must be protected from management. I have heard this opinion voiced in several books and from several leading agile consultants/practitioners – and I disagree with it.

Unfortunately, this attitude has been pervasive from the beginning of the Agile community. Given the heavy focus on over-bearing process that has gripped this industry for years, this is not surprising. However, it is time to include leadership and management in the transition to agile methods. We must see how we can marry respect for people (and hence the team) along with process. This requires leadership and proper management.

Buckingham, in his excellent book, The One Thing You Should Know describes leadership as the “ability to create a solid vision of a better future for those people he/she is leading. ” A leader must have a compelling desire to move towards that vision. Buckingham defines management as the “ability to match people’s tasks with their skills. ” A good manager is liked a skilled Chess Master. As opposed to checkers, where all the pieces (until one gets kinged) are the same, in chess, different pieces have different strengths. A good chess player knows how to use these strengths. A good manager knows how to maximize the strengths of his staff. Hence, leadership and management are not opposed to each other – they just have different functions.

Keeping Buckingham’s views in mind, the role of leadership and management then is to create the vision for the Scrum teams to be effective and to staff the teams with the appropriate resources. But just creating the vision is not enough. Leadership and management must help create the structure within which the team works to assist the team. This is something Scrum teams cannot do on their own. Ignoring this issue is what often has Scrum teams feel they must protect themselves from the organization when what they need to do is help the organization see how to better support the team.

If Scrum can’t help us here, what can? This is where Lean Thinking comes in. In a nutshell, Lean Thinking tells us to set things up so we can have fast-flexible-flow. That is, we can get customer requests in and get business value out – quickly and repeatedly. At Net Objectives we base our entire software development philosophy on this (which is why we call it SPeeD: Sustainable Product Development). The issue is – “how can I develop software quickly now (to deliver business value quickly) while retaining the ability to add business value quickly in the future.”

Read more in Alan's post.

In the book The Toyota Way by Jeffery Liker, one of the 14 principles outlined says "Grow Leaders Who Throughly Understand the Work, Live the Philosophy, and Teach It to Others". At the beginning of the chapter, there's an excellent quote that parallels Alan's frustration mentioned above:

Until senior management gets their egos out of the way and goes to the whole team and leads them all together...senior management will continue to miss out on the brain power and extraordinary capabilities of all their employees. At Toyota, we simple place the highest value on our team members and do the best we can to listen to them and incorporate their ideas into our planning process. -- Alex Warren, former Senior VP, Toyota Motor Manufacturing, Kentucky.

In summarizing this principle, Liker says:

If we look at all of the great leaders in Toyota's history we see they share several common traits:
1) Focused on a long-term purpose for Toyota as a value-added contributor to society.
2) Never deviated from the precepts of the Toyota Way DNA and lived and modeled themselves around this for all to see.
3) Worked their way up doing the detailed work and continued to go to the gemba - the actual place where the real added-value work is done.
4) Saw problems as opportunities to train and coach their people.

This doesn't sound like the ol' command and control type of manager works in this paradigm does it? However, managers are needed to provide Agile teams this kind of guidance. To do so, managers must respect team members as well as the other way around. Each have their place and need to understand their interdependencies in order to be a successful organization.

Now, Rules for Self-Leadership

A few posts ago, I talked about a great post by Rosa Say around self-management. Now, she has followed this up with a separate list this time focused on leadership qualities. I agree with Rosa that there is a difference between management and leadership qualities and that everybody should strive to accomplish both. Here, Rosa elaborates on this as well as presents the 12 Rules of Self-Leadership. Enjoy!

Management and Leadership are not interchangeable words for me. We need both of them, for in part, management tends to be more internally focused (within a company, within an industry, within a person) whereas leadership is more externally focused on the future-forward actions you will take in the greater context of industry, community, or society. They have commonality to be sure, for instance, both are about capitalizing on human capacity, however they are defined by the differences we value in them: Management tends to be about systems and processes, whereas Leadership is more about ideas and experiments.

I believe there is both art and discipline in each, and I think of these rules as the discipline which helps reveal the great capacity of the art. Thus last time, twelve suggestions to help you self-manage, with a more disciplined you newly able to reveal your art. Now, twelve to help you self-lead, so a more disciplined you is newly able to reveal the art in others, those who choose you to lead them.

12 Rules for Self-Leadership

1. Set goals for your life; not just for your job. What we think of as “meaning of life” goals affect your lifestyle outside of work too, and you get whole-life context, not just work-life, each feeding off the other.
2. Practice discretion constantly, and lead with the example of how your own good behavior does get great results. Otherwise, why should anyone follow you when you lead?
3. Take initiative. Volunteer to be first. Be daring, bold, brave and fearless, willing to fall down, fail, and get up again for another round. Starting with vulnerability has this amazing way of making us stronger when all is done.
4. Be humble and give away the credit. Going before others is only part of leading; you have to go with them too. Therefore, they’ve got to want you around!
5. Learn to love ideas and experiments. Turn them into pilot programs that preface impulsive decisions. Everything was impossible until the first person did it.
6. Live in wonder. Wonder why, and prize “Why not?” as your favorite question. Be insatiably curious, and question everything.
7. There are some things you don’t take liberty with no matter how innovative you are when you lead. For instance, to have integrity means to tell the truth. To be ethical is to do the right thing. These are not fuzzy concepts.
8. Believe that beauty exists in everything and in everyone, and then go about finding it. You’ll be amazed how little you have to invent and much is waiting to be displayed.
9. Actively reject pessimism and be an optimist. Say you have zero tolerance for negativity and self-fulfilling prophecies of doubt, and mean it.
10. Champion change. As the saying goes, those who do what they’ve always done, will get what they’ve always gotten. The only things they do get more of are apathy, complacency, and boredom.
11. Be a lifelong learner, and be a fanatic about it. Surround yourself with mentors and people smarter than you. Seek to be continually inspired by something, learning what your triggers are.
12. Care for and about people. Compassion and empathy become you, and keep you ever-connected to your humanity. People will choose you to lead them.

Read more in her post!

One Story Flow

One of the practices in Lean is this idea of One Piece Flow. What does this mean exactly? It focuses on getting more components completed more quickly by focusing on the completion of each one instead of having too much work in process. The thinking is that if your team can produce one component quickly with quality, you will have peak efficiency for future components.

So, where does this fit into Agile? In iteration planning, the team is working with a set of Stories. Here's where you can go in two different directions and has been a constant challenge in our organization. Where you go from here can potentially have two very different results.

First, you could ask the team who wants to work on each Story. Each individual responsible for the Story can then go off and determine the Tasks and Tests necessary to complete the Story. So, if you have a team of five developers, you will have five Stories being worked on (at a minimum, they may take on more Stories if they believe they have time in the Iteration and the Stories are small enough).

Or, you could have the entire team in Iteration Planning determine both the Tests and Tasks necessary for the first two or three Stories (more if you have a smooth process ). Then, have the team implement the first Story together (or as many team members as possible). If there isn't enough work for everyone, some go and start implementing the second Story. If the team members responsible for the first Story need help, others working on the second Story join the effort. The goal here is for the team to get each Story implemented as quickly as possible before they go on to the next Story. This is what I will coin "One Story Flow".

As I mention, we are still really struggling with this because it has been the nature of developers to have each person work on their individual piece and to bring the pieces together during some time of integration. However, this way of thinking starts to produce mini-Waterfalls within each iteration. Also, I have seen more unfinished Stories as a result. I would rather have more Stories that have been carried over to the next iteration at 0% than at 50%. I would also want more Stories that have been completed by the team at 100%.

If you are struggling with the team's performance and completion of Stories, try the "One Story Flow" approach. You will have more flexibility as a team and the team will be more in sync on how each Story has been implemented and how it contributes to the big picture of the product. More discussions up front will resolve conflicts and turn assumptions into facts. Finding ways to implement one Story after another will address issues such as specialization of skills, knowledge, and bottlenecks in the development process. These are painful but good things to discover. Because if the team can find ways to implement one Story better, they will be able to implement all Stories better!

Rules for Self-Management

If you want Agile to be more successful for your organization, you need each individual to think differently about their role and expectation from management. For the teams to be self-organizing, you need each individual to be self-managed. This isn't to say that management in an Agile organization isn't necessary, as they need to be there to provide direction and remove roadblocks for the team if the team cannot do it for themselves.

However, the more the team can manage for themselves, the greater the productivity and efficiency. After all, part of Agile is to be able to deliver working software with better quality and response time.

Rosa Say has always been one of my favorite bloggers on the top of management. She has a recent post on her Talking Story blog that as I read it realized how well it would work on an Agile team if the team adopted her 12 Rules of Self-Management. Here they are, read it for yourself and imagine if each individual followed these rules:

1. Live by your values, whatever they are. You confuse people when you don’t, because they can’t predict how you’ll behave.
2. Speak up! No one can “hear” what you’re thinking without you be willing to stand up for it. Mind-reading is something most people can’t do.
3. Honor your own good word, and keep the promises you make. If not, people eventually stop believing most of what you say, and your words will no longer work for you.
4. When you ask for more responsibility, expect to be held fully accountable. This is what seizing ownership of something is all about; it’s usually an all or nothing kind of thing, and so you’ve got to treat it that way.
5. Don’t expect people to trust you if you aren’t willing to be trustworthy for them first and foremost. Trust is an outcome of fulfilled expectations.
6. Be more productive by creating good habits and rejecting bad ones. Good habits corral your energies into a momentum-building rhythm for you; bad habits sap your energies and drain you.
7. Have a good work ethic, for it seems to be getting rare today. Curious, for those “old-fashioned” values like dependability, timeliness, professionalism and diligence are prized more than ever before. Be action-oriented. Seek to make things work. Be willing to do what it takes.
8. Be interesting. Read voraciously, and listen to learn, then teach and share everything you know. No one owes you their attention; you have to earn it and keep attracting it.
9. Be nice. Be courteous, polite and respectful. Be considerate. Manners still count for an awful lot in life, and thank goodness they do.
10. Be self-disciplined. That’s what adults are supposed to “grow up” to be.
11. Don’t be a victim or a martyr. You always have a choice, so don’t shy from it: Choose and choose without regret. Look forward and be enthusiastic.
12. Keep healthy and take care of yourself. Exercise your mind, body and spirit so you can be someone people count on, and so you can live expansively and with abundance.

Iteration Planning for the Unplanned

On any project, you will have both planned and unplanned tasks. Planned are those tasks that you knew ahead of time and scheduled some of your work towards those efforts. Unplanned are everything else. In many software shops including mine, the people doing the development work also help internal operations with ongoing technical support for internal and external customers.

Though Agile helps adapt to change by accommodating unplanned tasks in future iterations, there is still a challenge of what happens in the current iteration. If you are doing Scrum, it is expected that at the start of each iteration that the team commits to the scope of that iteration and that any additional work requested by customers should be considered in future iterations. Sounds good, but most technical support is needed on a much frequent basis and turnaround must be at a minimum. Therefore, you can have support issues to resolve on an hourly basis at times and you must respond within the next 12 - 24 hours. So, how do teams address these issues? Mishkin Berteig from Agile Advice has identified a few in his latest post:

1) The part-time allocation to the iteration's work is most common. There are a couple things that need to be done to make this work well in the long term. The main thing is to make the allocation of time inflexible: if you allocate 50% to the iteration and 50% to support, then you should never be flexible about that allocation. This is necessary in order for the team to make a commitment at the start of each iteration.

2) Another common method is the "fluorescent note card" method which requires stakeholder negotiation around the impact of interruptions. With this method, any time a stakeholder comes to the team with a request, the Process Facilitator writes the request on a bright colored note card. The team then does a task breakdown on the card and using their normal process (whatever that is) estimates the work. The requesting stakeholder then has to negotiate with any other stakeholders about what work to remove from the iteration in order to make room for the new work. The trick here is that the team has to be involved because they have already started on some of the work and it might be difficult to dis-entangle things enough. This process works well primarily because it makes the tradeoffs visible. It does not work so well with letting the team make their commitments.

3) A third common method is to form two separate teams: one doing new work, one doing support work. This is simple, effective, and annoying for the people on the team doing the support work! Please don't consider a rotation system since this destroys the process of team development and makes it nearly impossible for the team doing new work to learn their capacity for the purposes of commitment.

4) A less common, but fourth method is to have extremely short iterations. In this method, choose your iteration length to be so short that you can always start work on urgent interruptions before anyone gets impatient! This can be exhausting, but it is one of the best ways to get the team and the organization to understand the large toll that these interruptions take.

We have tried both #1 and #3. The problem with #3 is that developers that are put on the support work would love to do some development work. Plus, it is often difficult to have the knowledge (at least in our organization) to have a select few people who can address all of the support issues. However, if you try and combine support issues with ongoing development, you will find yourself trying #1 next. Unfortunately, it is very difficult to estimate what percentage of time should be put towards Support vs. Development as Support issues will come and go at an expected rate.

I don't think we can manage one week iterations (idea #4) effectively at this point at least until we become more mature at Agile and have the right infrastructure in place to turnaround iterations this quickly.

This leaves us with #2, which is what we are trying to do at this point. On our current Iteration Schedule board, we will have a Story called Issues that will be at the top of the list. Then, every issue will be identified as a Task and will have a color designation to help it stand out against "Planned" tasks and stories. I like this idea because it helps the team address what is reality but also balance the priorities of the Issues against Development stories. It also makes since that the Development team OWN the support issues since they have the most knowledge and skills to address issues in their own product and technical domain.