A Simple Look of Lean and Agile

Somebody asked me if I were to talk about the relationship between Lean, Scrum and XP what would that be in the "simplest way possible" (At our organization, we are implementing all three). Here's how I answered:

Lean introduces the principles, or the "why's" of producing software. Focus is on maximizing customer value while minimizing waste to provide the most optimal ROI.

Scrum takes those principles and introduces the management practices, or the "when's" of producing software. Focus is on incremental delivery of working solutions that improve over time.

XP takes Lean principles and Scrum practices and introduces the technical skills, or the "how's" of producing software. Focus is on constant improvement of skills (and the underlying infrastructure) through analysis, design, and implementation of producing software.

It is my strong feeling that you really need all three to maximize your effectiveness in the development of software. Individually, each part can be implemented on it's own but you will discover something missing. Together, these really cover all aspects of developing a successful software solution.

Exploring what makes Scrum successful

Alan Shalloway from the Net Objectives Thoughts blog is digging a little deeper into the "soul" of Scrum to truly understand why Scrum works in his post "Challenging why (not if) Scrum works". Please read his post in more detail, but here is some reasons he came up with:

I have repeatedly heard that “Scrum succeeds largely because the people doing the work define how to do the work” (see From The Top by Ken Schwaber for his full text). However, I do not think that is even a third of why Scrum works - and be clear, I am certainly not questioning that Scrum works - I just want to get at the why it works.

OK, so why does Scrum work? Remember, I am not challenging that it does, just asking why. I would say Scrum works for the following reasons:

* the iterative nature of Scrum development (including the re-assessment of priorities between iterations)
* the workcell nature of the Scrum team (everyone who needs to work together is together)
* the fact that many Scrum teams are given the chance to co-locate
* that most Scrum teams work on one project at a time
* that many old, cumbersome checkpoints/forms/status reports that used to be required are abandoned because the team is doing something new
* a focus on defining acceptance tests concurrent with requirements
* the availability of a product champion (product owner to Scrum enthusiasts)

I think this is a really good list, I would also add a couple of items to it:

* frequent communication of status and removing of impediments through the Daily Standups
* continue emphasis on feedback and improvement through sprint reviews
* the team nature of the Scrum team (everybody on the team should understand and commit to the sprint through sprint planning)

For those that are using Scrum, you are well familiar with these benefits. For those that have not moved to Agile or they are having problems with their practices, consider looking at Scrum to gain these benefits. Scrum by no means is a "silver bullet", but it has already proven to be very successful where it has been implemented.

Addressing Technical Debt

What is Technical Debt and how to you manage it? These were two questions that is addressed over at the Lean Software Engineering blog.

In the first post, called "7 sources of technical debt" Bernie creates his list of where technical debt comes from:

1. Undiscovered bugs, failing tests, and open bugs in your database
2. Missing test automation (unit, feature, scenario, or system)
3. Missing build and deployment automation
4. Scenarios with incomplete user experiences
5. Code that’s too difficult to understand
6. Code that’s too difficult to extend
7. Code that’s isolated in branches

In the follow up post, called "7 strategies for measuring and tackling technical debt" Bernie provides his list of potential remedies to reduce or eliminate this debt:

1. Bugs. Measure and shrink your cycle time from when code is first written until it is tested (TDD is perfect for this, of course). Treat open bugs as work in progress (WIP) — and too much work in progress is evil. Constantly measure and manage WIP down with strategies like bug caps (no new features until existing bugs get below a defined bar).

2. Missing tests. Measure code coverage. Have a clear definition of “done” at each scale of your product: unit, feature, scenario, system. Treat units of code as incomplete until code, tests, and documentation are delivered as a set.

3. Missing automation. Make build and deployment first-class features of your project. Deliver them early, and assume (like other features) that they will constantly evolve with sub-features over the course of the project. Measure time-to-build, time-to-deploy, and time-to-test — including human time expended each cycle — and then drive those numbers down by adding automation work to your backlog.

4. Incomplete scenarios. Assign scenario ownership roles among the team. Have a clear definition of “done” for each scenario. Pre-release working scenarios to select customers as soon as possible, making special note of these bugs. And of course, minimize the number of scenarios each team focuses on at once (one at a time, if possible).

5. Not understandable. Like most open source projects, generate docs directly from the code (literate programming), including why the code does what it does. Systematically conduct code reviews. If the team can’t afford to review every line, always review a sample set of functions from each developer and component. Ask reviewers to rate code for understandability, track this data in code comments or a spreadsheet, and add workitems to the backlog to clean up those most needing improvement.

6. Not extensible. Encourage design pattern thinking. Buy your team a library of design patterns books and posters to help form that common vocabulary. Look at how open source projects tend to build big things out of many small, independent pieces — treat every component as a library with its own API and a clear (and minimized) set of dependencies.

7. Not integrated. Minimize the number of branched codelines, whether they’re formally in source code control, done on the side for customers, or sitting on developer’s disk drives. Unintegrated code is debt — it has work and surprises waiting to happen. Strive for continuous integration. This isn’t to say don’t use branching — even the smallest project should think about their mainline model — but always keep everything in source control, then track and minimize the number of active branches.

This is by no means an extensive list as I'm sure there are other contributors of debt and possible solutions. However, I agree with Bernie that these are the most common things to be on the lookout for. Just think how better the quality of software, and the ability of a team to deliver new features can be maximized if teams would see things in terms of debt!

Read more in both of these posts.

Changing the way you do long term planning

We are in the process of implementing a pull scheduling system for each of our product development teams. No longer do we think in terms of projects, but in terms of releases and iterations. No longer do we figure out the entire scope in detail of what will be contained in the product, but we determine the details when we are ready in a particular iteration in a release. No longer do we have rigid control of scope for the entire project, but we try to contain what we do in a 2-week iteration once it has been planned and committed by the team. No longer do we try to figure out exactly what we will do in the next 1-2 years, but we "pull" enough work to determine the next release when we are ready to think about it. No longer do we wait for customer feedback only at the end of the project through alpha and beta cycles, but we get internal and external feedback through demos at the end of each iteration. No longer do we conduct vague and long weekly project status meetings, but we conduct quick and concise daily standup meetings. Our world has definitely changed, but in the minds of our executive stakeholders they have struggled a little with it. In particular, if you are only planning within short iterations and the current release cycle, how do you do long-term product planning for customers that want to have some idea of a delivery date for something "down the road"?

I have no problem in planning long-term if the planning is focused more on priorities than dates. Though the team is pulling just enough work for the next release, the more work that is prioritized and thought out (at least to a high level) will make the release and iteration planning go much smoother. And, actually I don't even have a problem committing to a particular date for the next release or any iteration in the release because those are fixed. However, committing to particular scope outside of the current iteration is something that I find challenging. Why?

Because we know only look at the details of the release within the current iteration, we haven't given enough thought about the remaining stories in the release. Sure, we have done some high-level estimates but we have learned that what you think you are going to do after thinking about it for a few minutes is much different when you think about it for a few hours in iteration planning. Until you really start looking at particular tasks and tests that the team will do in fulfilling each story, you don't really have an accurate estimate. And even with that said, you may find surprises within the iteration after planning.

The other challenge is that you are dealing with change in a whole different way. When you find a defect, you don't "log" it in a repository automatically because you don't have the time to fix it during the project. Instead, you deal with defects as they happen. When you show functionality in a demo, you don't log future enhancements automatically to be considered in future projects but you make appropriate changes if they are valuable and while they are fresh in the team's minds. When new issues come up, you create new stories to deal with them if they are deemed the right time. All of these things change scope, which changes what is done in a particular release. This changes the expected delivery dates many times. Which makes it very difficult to predict the scope of a release, and makes it almost impossible to predict when new functionality will be available in 12-18 months.

Now, this may sound bad initially to your stakeholders, because they want to make promises to customers as soon as possible. So, this may sound like Agile and Lean is the wrong way to go right? While I can't predict what will be done in 18 months, I can show you progress every two weeks. This progress will have higher quality because it has been tested and could be released to customers if desired. What we work on has been determined the most important to our customers, so this is a good thing. The best thing of all? The fact that we make sure that every deliverable, minimal useful functionality, really is what the customer wants because they have been more involved more frequently. So does it really matter if we try to fix scope, resources and schedule but having such as strict change control process if in the end we don't deliver what is really wanted?

So, what I have agreed on with the stakeholders is that we will go out beyond a release but strictly from a planning and goals perspective. At the point where it is determined that the goals are not obtainable in the process, I will make sure that the stakeholders are notified and can adjust the scope or define new goals based on this information. What's interesting is that I will never have to have that kind of conversation because the stakeholders will be involved in every part of the process. They will see progress being made, they will set priorities of work, they will attend demos and make changes, and they will see the impact of those changes. In the end, they will learn that they can draw the "line in the sand" when they feel most appropriate. Based on the confidence that they are gaining with the teams, they will change the long-term discussion with the customer to what is being developed than what is being planned. When that isn't enough, they can give the customer an idea of the priority of work being considered. If the customer feels that the priority isn't meeting their needs, they can voice their opinion. Then the stakeholders can determine what to do with that feedback and make priority changes as necessary. As long as customers see steady progress with frequent releases that are meeting needs, they should be satisfied with this approach.

Thinking inside the Agile Box

Pretend you are taking a week long trip with a friend. You start packing and end up with a couple of large suitcases. Your friend arrives with one medium suitcase. You start thinking, "What are they thinking? They don't have enough clothes to last the week." You arrive to your destination. When you start unpacking, your friend's clothes are all nicely folded and no wrinkles. Your clothes look like a mess and will need to be ironed. Your friend is able to put things into drawers quickly as well as the bathroom items. You have to go through your bag and it takes you longer to find things and put them in their place. However, your friend has less clothes than you. As the week progresses, you begin to see that your friend has found ways to reuse what they wear - different shirts with the same pants - yet still appear like they aren't worn. You on the other hand, find that at the end of the trip you had clothes that you didn't wear, but had to carry around. Now you start thinking, "My friend has a better way. I've got to learn from them for the next trip!"

Was the friend always this organized? Probably not, they probably learned by somebody else. Experienced travelers have tricks on how to properly fold clothes and how to maximize every spaces in a suitcase. They have also learned what's "good enough" and not to take more than they need. Even if they end up needing fresh clothes, they will end up using a local cleaners or wash their own clothes instead of taking too much. They have learned how to best put clothes and other accessories together to maximize their use.

As my development staff is asked to reduce cycle times through iterative development using Agile best practices, their initial reaction is "That sounds all good, but we have found things to take weeks or months and that won't work with this approach." They are correct, it won't. But, just because you have done things a certain way doesn't mean that it is the ONLY way to do things. We have learned how to do things as individuals. We will now need to do those things together as a team. We have learned that each person plays a particular role and has their own specialization. We will now need to remove those specializations and have people wear multiple hats. We have learned to think about everything up front is much detail. We will now need to learn to do just enough for now and think about details when it is appropriate later. We have learned to break the work and requirements down just enough to make sense for a 6-8 month project, we will now need to refine that work to make sense for 2-4 week iterations. We have learned a way to deliver software after 8 months, we need a way to deliver that same software in a portion of that time and possibly in weeks instead of months. We need systems that get tested and built at least every day if not several times a day, the systems we have used cannot handle that load.

What's great about Agile is that it isn't just "theory" anymore, people have found it to work and improve the flexibility and responsiveness of the team to provide working solutions to customers. We can tap into these "experienced travelers" and figure out how they have done it a different and better way. It just seems strange to us because we have learned how to do things differently, not necessarily better. However, if you think that Agile is just about doing things faster you will miss the point. Agile is about doing things differently in order to deliver working software more effectively. That means we need to relearn how to build, maintain and support software under this new way. Relearn skills, tools, knowledge, processes and essentially how you do your jobs differently.

You might have thought you were a great traveler and packed accordingly, until you realized that there is a better way. However, you can't just copy the better way like it was out of a cookbook. You must understand why this experienced traveler didn't to make the change. Then, you must understand how he made the change work for them. Then, you must make the changes work for yourself. Same with Agile, you don't just adopt a methodology and away you go. You must understand the principles behind it, then determine what set of best practices fit your organization. When people ask us if we are using XP or Scrum, I respond with "Yes". "Which one?", they ask. "Both" I respond. There are practices within each that we have adopted to support Agile principles. There are other practices that we have come up with that perhaps nobody else is using. That's ok, as long as we are staying true to the goals of Agile development. I will say that it has required EVERY person to think about their jobs differently. In some cases, much differently. In the end, it required each person to "think inside a different box". An Agile Box!

Growing Your Leaders

Alan Shalloway from the Net Objectives Thoughts blog challenges how Agile has viewed management and leadership up to this point in his post "The Need for Leadership in Scrum". Here are some highlights:

There underscores the notion that self-managing teams and direction from the top are somehow opposed to each other. In practice, this may be. But that is because in practice, leadership is missing. Direction in the form of command and control is bad. Direction in the form of vision and matching people to their abilities is not. This points to one deficiency I have seen in the general Agile community, and one that, I believe, is a serious impediment to having the Agile community have success at the enterprise level. I am referring to the fact that most Agilists do not trust management to do what is right – they feel the team must be protected from management. I have heard this opinion voiced in several books and from several leading agile consultants/practitioners – and I disagree with it.

Unfortunately, this attitude has been pervasive from the beginning of the Agile community. Given the heavy focus on over-bearing process that has gripped this industry for years, this is not surprising. However, it is time to include leadership and management in the transition to agile methods. We must see how we can marry respect for people (and hence the team) along with process. This requires leadership and proper management.

Buckingham, in his excellent book, The One Thing You Should Know describes leadership as the “ability to create a solid vision of a better future for those people he/she is leading. ” A leader must have a compelling desire to move towards that vision. Buckingham defines management as the “ability to match people’s tasks with their skills. ” A good manager is liked a skilled Chess Master. As opposed to checkers, where all the pieces (until one gets kinged) are the same, in chess, different pieces have different strengths. A good chess player knows how to use these strengths. A good manager knows how to maximize the strengths of his staff. Hence, leadership and management are not opposed to each other – they just have different functions.

Keeping Buckingham’s views in mind, the role of leadership and management then is to create the vision for the Scrum teams to be effective and to staff the teams with the appropriate resources. But just creating the vision is not enough. Leadership and management must help create the structure within which the team works to assist the team. This is something Scrum teams cannot do on their own. Ignoring this issue is what often has Scrum teams feel they must protect themselves from the organization when what they need to do is help the organization see how to better support the team.

If Scrum can’t help us here, what can? This is where Lean Thinking comes in. In a nutshell, Lean Thinking tells us to set things up so we can have fast-flexible-flow. That is, we can get customer requests in and get business value out – quickly and repeatedly. At Net Objectives we base our entire software development philosophy on this (which is why we call it SPeeD: Sustainable Product Development). The issue is – “how can I develop software quickly now (to deliver business value quickly) while retaining the ability to add business value quickly in the future.”

Read more in Alan's post.

In the book The Toyota Way by Jeffery Liker, one of the 14 principles outlined says "Grow Leaders Who Throughly Understand the Work, Live the Philosophy, and Teach It to Others". At the beginning of the chapter, there's an excellent quote that parallels Alan's frustration mentioned above:

Until senior management gets their egos out of the way and goes to the whole team and leads them all together...senior management will continue to miss out on the brain power and extraordinary capabilities of all their employees. At Toyota, we simple place the highest value on our team members and do the best we can to listen to them and incorporate their ideas into our planning process. -- Alex Warren, former Senior VP, Toyota Motor Manufacturing, Kentucky.

In summarizing this principle, Liker says:

If we look at all of the great leaders in Toyota's history we see they share several common traits:
1) Focused on a long-term purpose for Toyota as a value-added contributor to society.
2) Never deviated from the precepts of the Toyota Way DNA and lived and modeled themselves around this for all to see.
3) Worked their way up doing the detailed work and continued to go to the gemba - the actual place where the real added-value work is done.
4) Saw problems as opportunities to train and coach their people.

This doesn't sound like the ol' command and control type of manager works in this paradigm does it? However, managers are needed to provide Agile teams this kind of guidance. To do so, managers must respect team members as well as the other way around. Each have their place and need to understand their interdependencies in order to be a successful organization.

The Predictability Paradox

We had just several months earlier started our implementation of Scrum to all of our project teams. As we were learning what to do, more importantly we were learning things not to do. One of the things I found challenging in our adoption of Agile was that many felt that the traditional (Waterfall) way we were delivering software before was more predictable and that Scrum would be totally unpredictable given its emphasis on focusing one iteration to the next instead of heavy upfront requirements, design and planning. I knew in my heart that Waterfall really couldn't predict any better but really couldn't put my finger on why. It was then that I read this paper called "Lean Development and the Predictability Paradox" that I got my answer. It was this paper that changed my thinking and introduced the Lean concepts to apply to what I had already learned with Scrum. Here is the introduction (read more including more details on each aspect of Lean by downloading the paper here):

Wall Street has little sympathy for companies that can’t meet their forecasts every quarter. In turn, senior management expects department managers to make and meet forecasts. By the time these expectations arrive at an IT department, meeting forecasts often becomes a significant challenge. Unfortunately, it seems that a large fraction of software projects fail to deliver on their promises for one reason or another.

Why is it that software development projects seems to have so much difficulty delivering predictable outcomes? It seems that all too often projects are late or over budget or they deliver the wrong system or all of the above. How can the predictability of software development outcomes be increased? This executive report discusses a dilemma: in our zeal to improve the reliability of software development, we have institutionalized practices that decrease, rather than increase, the predictability of outcomes.

The best way to achieve predictable software development outcomes is to start early, learn constantly, commit late, and deliver fast. This may seem to cut against the grain of conventional project management practice, which is supposed to give more managed, predictable results. But predictability is a funny thing; you cannot build with confidence on a shifting foundation. The problem with conventional approaches is that they assume the foundation is firm; they have little tolerance for change.

The paradox is that trying too hard to create predictability creates opposite effect. Conventional practices are fragile in the face of change, and even in the face of learning. And yet, the more complex the system, the more necessary learning becomes. What is needed is an approach that encourages learning, and does not commit until learning is complete.

It should be obvious that decreasing the amount of speculation involved in making a decision increases predictability of the outcome. If you can make decisions based on facts rather than forecasts, you get results that are more predictable. Lean development is the art and discipline of basing commitments on facts rather than forecasts. It starts earlier, encourages change, freezes decisions later, and delivers faster than traditional practices, but nevertheless lean development produces outcomes that are more predictable.

The paradox of lean development is that you have give up some of the trappings of predictability in order to get true predictability. You have to abandon some conventional wisdom to gain the benefits of making decisions with more certainty. Fundamentally, you have to develop the capability to respond to events as they unfold, rather than hold dear the capability to orchestrate events in advance.

The Agile Elevator Speech

I was going to also call this post, "How to talk about Agile to the non-Agile" but thought the title above was more appropriate. Joe Little from Agile and Business blog has a post was trying to figure out how to tell friends and family about what Agile and Lean are without having to spend a lot of time getting them up to speed. In his post, he ends up summarizing it as:

* Focus on delivering small slices of concrete business value frequently (1-4 weeks)
* Work with a small team of people, to enable them to be as productive as possible
* Use concrete, understandable results to allow all parties to adapt and move forward
* Arrange things so that change and learning are benefits
* Tell the truth, and make the important things as visible as possible
* Minimize waste and extra effort; maximize customer value, don't impress with hard work

I think this is an excellent introduction to both Agile and Lean that can be accomplished in a few minutes and give the listener an idea without having to know much about software development, product development or manufacturing. Read more in his post on how he led up to this list.