From Scrum to Kanban

Posted on July 28th, 2009 in Agile, Kanban by siddharta || 29 Comments

Teams that do Scrum for a long period of time naturally tend to hit into some walls. In the process of inspecting and adapting over a period of time, they eventually end up with something like a Kanban process. In this post, I’ll explain how the evolution worked for us.

Velocity based sprint planning

To start with, we did plain vanilla Scrum. Two week sprints, sprint planning meetings, releases and so on. However, we soon ran into an issue with velocity based capacity planning.

Velocity is a probabilistic distribution, so if your average velocity is 20 points, then that doesn’t mean you will do exactly 20 points every sprint. Some sprints may be less, some more, but on average it is 20 points.

The first problem is that in Scrum, the team commits to the sprint plan. So if the average velocity is 20 points, typically you will commit to 20 points of work. Problem is, you will rarely do exactly 20 points. Most of the time you are less or more. This is quite natural, because velocity is probabilistic. If we were to use the terminology from Demming, you would say it is common cause variation.

Now, assume that you have committed for 20 points, but you are only going to make it to 15 points. What then? In Scrum, you would work harder (extra nights or weekend maybe) in order to make the commitment. Personally, I do not think this is healthy, because the variation is natural, not something extraordinary. But because of the commitment at the plan time, teams have to do this to keep the promise.

Sometimes the opposite happens, and you are doing better than average. Lets say you finish 20 points early. Scrum says you should not take up new features mid-sprint.  So teams usually do other non-value work to fill in the rest of the sprint.

Therefore in Scrum you end up with a situation where you have to work extra when you are behind, but you waste the time when you are ahead. And most of the time you are ahead or behind. Very rarely are do you go exactly as planned.

The problem is that Scrum ignores the reality of variation in velocity and attempts to force teams to a constant velocity. The root cause is the commitment oriented sprint plan.

Kanban gets around this by embracing variation. Since there is no commitment based iteration plan, teams work at the natural rhythm. This means that sometimes you deliver more and sometimes you deliver less, and there is nothing unnatural about that. Kanban differentiates between common cause variation and special cause variation, and each is handled differently.

The first adaptation we did was to get rid of a commitment based sprint plan.

We would plan out, just to give a feel of the sprint and to maintain coherence between selected stories. But if we did less in a sprint, we just moved unfinished stuff to the next sprint and release only what was completely done. And if we finished early, we’d pick up some new stories, even if we knew that it might spill over the release boundary. Suddenly, we no longer needed to fit a story within a sprint. If it spilled over, thats okay, we just continued it next sprint. This led naturally to the next change.

Swapping items mid-sprint

Once we stopped looking at the sprint plan as a commitment, it enabled us to do something else. Now if some high priority item came into the backlog, we didn’t really mind adding it mid-sprint and taking out something else that was not started. This makes sense – if you haven’t started work on a story, then what is the difference if you take it out and put something else in its place? The team suffers no penalty because you are taking out something that is not started anyway, and it enables delivery of higher priority items. This is a complete win situation. Whats the problem?

Limiting Work In Progress

The problem in traditional Scrum is this – sometimes its hard to find something that’s not started. This is because once the sprint plan is done, teams often start work on all items simultaneously. If you then want to do a swap, you have to remove something that is in progress – in that case there is a penalty for doing a swap mid-sprint. To avoid dropping a story that is already started, you could end up with a situation where the story is added without anything being removed. Because sprints are commitment based, this is a bad, bad thing.

The solution?

Limit work in progress. Don’t start a new story until another one is finished.

Implement a pull system to balance demand with capacity.

Implement a stop the line system so that blockers are taken care of.

We now start noticing something – If items can be swapped mid-way, and the sprint plan is no longer commitment oriented, then why exactly are we planning once every sprint? Which leads to the next point.

Decoupling cadence cycles

Once we started picking items mid-sprint we asked – Why spend the time creating a sprint plan if we are going to be changing it? All we need is a well prioritized backlog. The team can then just keep picking stuff of the top of the queue. Want a change in priority? Just reprioritize the queue.

Suddenly, the concept of iterations and planning cycles were no longer coupled. Planning could happen either when something major came up, or on their own cadence, or when the backlog started running low. Implementation could happen on a different cycle altogether, just pulling stuff from the backlog and releasing on their own cadence.

Similarly, teams would find that retrospectives have a lot of value at certain times (at the start of a project, or after some special cause event), but not much value at other times (in a steady state perhaps). Why tie the retrospective cadence to the delivery cadence? It is an artificial coupling. So retrospective cadence is now separate from a delivery cadence.

The whole concept of fixed sprints with coordinated events at the start and end of a sprint were gone. Note that the conecpt of cadence didn’t disappear. It just got decoupled.

Kanban Demystified

There are a lot of myths about Kanban floating around the Internet. That Kanban is a return to waterfall and handoffs. That Kanban does away with cross-functional teams. That Kanban has no cadence.

Nowhere does Kanban say anything about waterfall, handoffs or cross functional teams. Can you do Kanban with a cross functional team? Sure! Can you have everyone in the same room? Why not! Do you need to have handoffs? Nope. Does Kanban get rid of cadence? No, it just decouples it – but no one stops you from having it coupled if that works for you!

Kanban also works better in a traditional setup. What if you do have handoffs, distributed teams and all that? Scrum will fail miserably – ScrumMasters will (rightly) tell you that Scrum is exposing the organizational limitations. But its hard to make all changes in one go. This is why we have a bunch of teams doing “ScrumBut.”

However, you can still use Kanban in these traditional organizations. You will get a part of the value. And if organizations want the whole value, they can make a decision to reduce handoffs, move to colocated teams and so on in a phased manner.

There has been a lot made out of Kanban boards, lack of estimation and so on. Yes, they are all used in Kanban, but they are incidental. A Kanban board is simply an outcome of using a pull system, which in turn derives from limiting work in progress.

I make this differentiation because there is a misconception that “Kanban Process” equals the “Kanban board”. Some think that they use a Kanban board, so they are doing Kanban. This is missing the forest for the trees. The Kanban board is just a rather small part of things – an outcome of more fundamental principles.

Summary

The way I look at it, Kanban is an inevitable progression from Scrum that occurs naturally over a period of time with teams that inspect and adapt. Inspect and adapt is a core part of Scrum, which is ignored by many teams that are too keen to follow the “rules” of Scrum. Ironically, by following all the rules, except the one that matters most, it is exactly these teams that get into a major tangle.

Did you know for instance, that only 39% of teams do retrospectives? That retrospectives do not find a mention in the Nokia test? (The Nokia test is the “almost official” way to test whether you are doing Scrum) There goes inspect and adapt out of the window.

The retrospective is important enough for Alistair Cockburn to list it as one of only three “must have” properties in the Crystal methodology, but it seems to have gone missing from Scrum.

Kanban is simply a set of natural modifications to Scrum. Each modification automatically leads to another over a period of time, until you go from Scrum to Kanban.

We arrived at Kanban through these principles

  • Understand and embrace variation
  • Limit work in progress

Each of these further led to others, like pulling value, limiting batch sizes and so on.

For us, understanding variation was the key trigger that led to the chain of changes. For others it might be something else, like too much work in progress, or too many small stories. There are many ways you can end up with Kanban.

Above all, inspect and adapt.

Doing Distributed Agile?

Share and collaborate with distributed teams with our electronic agile board tools. Get all the benefits of electronic tools without sacrificing the benefits of physical boards. Supports Scrum taskboards, Kanban boards and user story maps. Check it out!

29 Responses to “From Scrum to Kanban”

  1. Paul Jackson Says:

    Hi Siddharta,

    This is a really interesting and topical article. Swapping items mid-Sprint is something we’ve also been struggling with and we followed a similar common sense approach to the one you describe. This also led us to decoupling the cadence cycles. With one client, we asked them to set some realistic (but not set in stone) release dates. Within this release window we run a series of sprints, tackling features in their priority order. At a given point in time (normally a couple of weeks before the release date), we’ll dedicate the final sprint to managing the implementation of all the “done” features to the live environment. We’re still experimenting with this but it appears to be working. The client still gets to prioritise the features and drive the release date. And if they want to initiate the release early, we just “terminate” the sprint and move into the “release” sprint to deliver all the done features. Not strictly within the Scrum rules, I know, but at the end of the day our aim is to deliver to and satisfy our client.

    I’ve been reading up on Kanban Systems for Lean Software Development and found this book by Corey Ladas to be quite interesting: http://tinyurl.com/kjgk4m.

    Regards,

    Paul

  2. siddharta Says:

    Hi Paul,

    Looking at your post, it sounds like Kanban would be a good fit, especially since the backlog can change at any moment.

    One thing to think about – is it possible to deploy to live at a regular cadence? Maybe once in two weeks or once a month.

    If you can do that, then you don’t need to terminate a sprint. You will enter a steady state where you keep working on the top of the backlog and the client will get a new version continuously.

  3. Eben Halford Says:

    Interesting article – I have some ‘common cause variation’ with your stated version of Scrum however.

    “In Scrum, you would work harder (extra nights or weekend maybe) in order to make the commitment”

    Umm not in my experience – the team and/or scrum master would simply talk to the product owner and negotiate the depth of acceptance criteria on the least valuable item you have in the sprint – basically reducing it’s point value. You’d do this by watching the burndown which will guide you in knowing if you’ve over or under shot.

    “Scrum says you should not take up new features mid-sprint”

    I don’t agree with this either – if you finish all your stories then you take the next highest priority item on the backlog after discussion between the team and the product owner.

    Cheers,
    Eben

  4. siddharta Says:

    Eben, agreed. That is pretty much what we did too. Basically stop looking at it as a commitment, but something that changes according to what happens.

    But that is not how the official scrum states things.

    There is a difference between “committing” to the sprint plan, and saying we’ll work at a natural pace and see what gets done. Scrum asks for the former, and this is the way people are trained.

    Of course, if there are major issues, you can ask for sprint termination. And sometimes stuff is left out even after all efforts. But otherwise the team is expected to meet the commitment.

    If you just do whatever you are able to do, sometimes less, sometimes more, then there is no meaning to creating a sprint plan, is it not? Wouldn’t you be better off just taking a prioritized backlog and doing stuff from the top and whatever gets done is what gets released?

    That is what we ultimately started doing. We just got rid of a sprint plan, and took stuff one by one off the top of the backlog. Whatever got done, we released. If we were going slower, we would end up taking fewer items from the backlog. If we were going faster, we took more.

  5. rainwebs Says:

    If you skip the Spring Backlog how do you manage task dependencies then? Why do you have problems to persuade the Product Owner for a period of 4 weeks to keep what was planned? For me this is too much disturbance for the team.

  6. siddharta Says:

    Hi Rainwebs,

    Good question. The key to getting this to work is to limit work in progress.

    If no one has started any work on an item, and if I take that out and put something else of equal size, then it should not make a difference to the team?

    The problem arises only when someone has already started working on the item.

    When you limit work in progress, then changing the sprint backlog is no longer a disturbance.

  7. neal Says:

    It’s interesting article, but 3 questions still,

    a. if we go without commitment sprint and sprint planning, why do we still keep sprint? The concept of sprint lost its sense after this change.

    b. if there’s no commitment sprint, team tends to work less hard and they don’t have clear target for their tasks at hand. How does Kanban handle this?

    c. in a scrum team of 8 people, usually it’s not likely that whole team work on one item all together and move to next when it’s done. More people working on one item doesn’t make it faster. The result is people start with several items at the same time and that’s just natural I think. How do we handle this in Kanban to limit the work in progress?

  8. Prasad Says:

    Hi Siddhart, Thank you for an excellent post. Currently I am doing a PoC using Kanban. I am in a midway
    challenges
    a) Kanban is less prescrptive- team need to be highly matured and experineced to adopt right engineering methods
    b)SLA’s and cycle time – WIP limit very difficult to put a number
    c) confusion with roles & responsiblities

  9. siddharta Says:

    @Neal :

    Agreed, thats why kanban gets rid of the concept of sprint.. since anyway everything is decoupled from it.

    About working less hard, well I’m not sure about that. You still want to minimize lead time, so everyone will be working towards that. You can still have standups if items are stalled. Plus, if something is going slow it blocks the pipeline and attention is quickly drawn towards getting it done.

    As for your third question, you need to set the work in progress limit appropriately. Not everyone has to work on the same item, its okay if each works on a different one. What you want to avoid is to start working on one, then stop and start another one. Also, practices like pair programming will allow you to focus two people on one task and get it done quicker with better quality.

  10. siddharta Says:

    @Prasad :

    Yes, Kanban is less prescriptive. While it does not say you should use unit testing or any engineering practices, that is possibly a good thing to do anyway. Kanban mainly talks about the end to end flow and how to optimize it.

    Again agreed that its hard to figure out what WIP limits you should use. As a starting point I’d limit it to the number of people, then identify where there are piling up of work and where its smooth and adjust it to match capacity.

    About roles, Kanban doesn’t specify any, so it can fit in with whatever you are currently doing.

  11. Mario Says:

    I would be a cautious about your statements on Scrum. When the blog says “Now, assume that you have committed for 20 points, but you are only going to make it to 15 points. What then? In Scrum, you would work harder (extra nights or weekend maybe) in order to make the commitment”, this is not the case. Most Scrum teams I work with are not focused on the “number” of points but instead ensuring that the work gets done correctly. If it happens that people are working late, it is a sign of a working culture that makes people work long hours, but certainly not advocated by Scrum. In fact, Scrum advocates a regular work work. Also, then it says, “The problem is that Scrum ignores the reality of variation in velocity and attempts to force teams to a constant velocity.” This again is not true. You cannot assume that the team is focused on delivering a specific set of story points when in reality, the team hardly cares about this and instead cares about delivering quality product. How many story points that actually get finished is a number that is used as a point of reference to understand velocity but certainly not to drive it. There are advantages to Kanban, but be cautious in unecessarily placing assumptions to Scrum that are not true.

  12. siddharta Says:

    Mario, then why bother with the Sprint plan meeting? Let the team do whatever they can in the sprint and what is done in the end gets delivered.

  13. Zorry Says:

    Hi Siddharta,

    That’s a good article and I completely agree that Kanban brings a lot of value when it comes to end-to-end process optimization, and the concepts behind also help solving a lot of issues. However, what I like about Scrum and the sprint (time-boxed) approach is that it really keeps the team focused on achieving a certain goal within a certain timeframe. In the end – come on, we are all people and it is very natural to us to be much more focused and streamlined in our work when we do have certain deadline to finish a job. Minimizing lead time alone is not enough, from my viewpoint to keep the team up-to-speed. I also agree to Mario’s comment about the working culture.
    Regarding your comment about Sprint plan meeting – I think what we are missing here is really what the focus of the meeting should be. The ProductOwner explains WHAT needs to be created, and the team should focus on HOW to create it. In essence, the planning meeting is – or should be, more of a design and architecture discussion, rather than simply story point calculation exercise. That’s at least what I have seen as a good practice in my experience with Scrum.

  14. jame speckham Says:

    My only problem with my understanding of Kanban is that it seems like there is nothing to address sizing so what is to keep a customer from asking for something that ends up being infinitely large. “I want a job board” becomes the card that moves into the kanban board and it just hangs there forever in analysis? or you realize it’s too freaking big because the programmers are bored so my guess is they would find other crap to do while they waited on analysis. I donno it just sounds like a return to the issues we had in waterfall… i would love to see it in action to see how the sizing and scope control is suppose to work. Scrum at least gives us PM’s a way to look and see that ‘yes’ there is a limited amount of scope and it will be done at a certain point in time.

  15. siddharta Says:

    Well, there is nothing to stop you from estimating (sizing) in Kanban. Many kanban teams use t-shirt sizing – categorizing features into small, medium, large, extra large for example. If you need finer grained estimates, then you could use the usual planning poker or any other estimating method. The business need will probably determine what kind of estimating you do.

  16. Fun Chiat Says:

    Ken Schwaber page on new rules on sprint
    http://www.scrum.org/scrum-guide-updates#backlog

    “Development Teams do not commit to completing the work planned during a Sprint Planning Meeting. The Development Team creates a forecast of work it believes will be done, but that forecast will change as more becomes known throughout the Sprint.”

    This is different from “Wouldn’t you be better off just taking a prioritized backlog and doing stuff from the top and whatever gets done is what gets released?”

    If work is not organized inside sprints, it’s not Scrum. What are the positive and negative for the latter and former?

  17. siddharta Says:

    @Fun Chiat

    I think more and more of scrum is looking like kanban these days.

    For example, the latest Scrum thinking (like the quote that you referenced) is that you should not do a ‘hard commit’ to the sprint backlog. Another one is to work on one backlog item at a time, and start the next one only after the first one is complete (ie limit work in progress within a sprint).

    In fact, apart from the iteration timebox there is not much of a difference between today’s Scrum and Kanban really.

    You asked “If work is not organized inside sprints, it’s not Scrum”

    My question is – if the whole team is only going to work on one or two items at a time, then what is there to organize for the whole sprint? The team can just organize on each backlog item just before they start working on it. When its done, move on to the next one.

    And when you do that, its more or less Kanban that you are doing.

  18. Fun Chiat Says:

    I once seen global directors who quote ‘Kanban’ because they handle sales request at global level. Any work package from them could come to delivery centres and then assigned according to capacity. Meetings are held every week. Monday are for sales opportunity and Thursday for project status. Representative will attend the weekly conference call. Developers continue their work meanwhile. If requirements are unclear, developers contact representative or the project manager or the director directly.

    “The team can just organize on each backlog item just before they start working on it. When its done, move on to the next one.”

    Because of my past experience and compared to change introduced by sprints, I keep thinking of this scenario by that sentence:

    1. Assigning work package to a team.
    2. Work package have criteria like service level or customer promises. Service level depend on client contract.
    3. Team estimate roughly how long they will take.
    4. Based on step 2, manager make sure team is not overwhelmed. Or if quoted hours are not acceptable then negotiate with team – scope, time, resource to an acceptable level.
    5. Weekly meetings to check for exceptions.
    5.1 Exception: More urgent work comes in
    5.2 Exception: PM check for who is available. 5.3 Put aside whatever they are doing. Urgent work come first. Or ask they work extra hours to cover all work but compensate accordingly. (HR procedure, labor law, bonuses)
    6. Work going smoothly.
    7. Finished. Team ask for the next piece of work.
    8. Rinse and Repeat. After a few more work packages, cut-over to production and demo meetings are decided as and when by managers.
    Maybe next week. Maybe two weeks. Maybe in 2 days.

    I think I missed something on what you mean. When they introduced rule of sprint, the above scenario became very little or none.

  19. siddharta Says:

    “I think I missed something on what you mean. When they introduced rule of sprint, the above scenario became very little or none.”

    Well, thats not true.

    Even in Scrum, you need to negotiate scope with both customer and team, and make sure team is not overwhelmed and prioritise the backlog and manage exceptions & urgent work etc.

    In Scrum these discussions, like negotiating scope, take place during the Sprint planning meeting, or even earlier in the pre-sprint backlog management done by the product owner.

  20. Fun Chiat Says:

    The Scrum Linkedln group now are actively discussing “Any thoughts on How to handle PO introducing new user story after the sprint began?” [8th September 2011]

    Is that what you are talking about?

    What I meant is something different. Before I used scrum, the team still prioritize and accept work. However despite work in progress, project manager revise date & scope for cut-over for production, have weekly status meetings regardless work is finished or not, meetings to clarify related and unrelated scope and inconsistent demo dates.
    We sometimes had to research on software inquiries which may or may not be bugs.

    In Scrum however, I knew a ScrumMaster who protect her team when product owner tried stunts like that. The team used sprint planning, daily standup, backlog grooming, sprint goals and retrospective for inspection and strategy. There is ad-hoc bug and organizational initiatives but time is fixed + timebox. Within a sprint the team is self managing and does not worry about service level or directives.

    In that same company, they had a big product backlog that demanded multiple teams jointly work together. Using sprints, their teams found it easier to coordinate and synchronize with each other. For example, they have a joint retrospective after major release to market.
    Another example, two out of the multiple teams work on related modules. They share some inter-related expertise. They could plan their inter-related resources ahead using sprint.

    Suppose, management ask the team:
    -When can you deliver Feature A (is feature, not story)?
    -I need your outcome of your research in one month

    Or, another may ask the team:
    -When could Andy, Sharepoint expert come into our team? We need to plan how to start our work.

    What can I advice the management if I use Kanban?

  21. siddharta Says:

    Well, Kanban is just a method to improve flow of features. No one says that you cannot protect the team in Kanban or you should not collaborate with the team and so on. You can create self organized cross functional teams and still do Kanban, just like how you do it in Scrum.

  22. Fun Chiat Says:

    Do you know where I can find resources on Kanban applied to real world projects?

  23. siddharta Says:

    Try https://groups.yahoo.com/group/kanbandev and http://www.limitedwipsociety.org/

  24. Statistics for Agile Teams: Understanding Variation » Tools For Agile Blog » Blog Archive Says:

    […] Even better, don’t use target sprint commitment. Let the team do however much they can, and deliver whatever was completed. […]

  25. Thorbjorn Sigberg Says:

    Interesting post, and many interesting comments!

    @Fun Chiat: For real world lean/kanban experience, feel free to follow my blog where i blog about what I experience while running development projects in a small software development company.

  26. MGC Says:

    Interesting but there are many untrue characterizations and straw-man arguments concerning Scrum.

    “In Scrum, you would work harder (extra nights or weekend maybe) in order to make the commitment”

    Scrum does not say anywhere that you should do this. Scrum says you should de-scope when it becomes clear that these items are not going to get done.

    “The problem is that Scrum ignores the reality of variation in velocity and
    attempts to force teams to a constant velocity.”

    Also not true. Nowhere in the Scrum literature does it say to do this. Velocity is used to measure progress and to help guide planning, not to force a hard commitment to anything.

    “Scrum says you should not take up new features mid-sprint. So teams usually do other non-value work to fill in the rest of the sprint.”

    Not true. Scrum says you can only take up new features if the team decides they will not jeopardize the sprint goal. It is absurd to interpret this as saying that the team should sit idle for a week or two just because they finished early.

    “The root cause is the commitment oriented sprint plan.”

    Not true. The idea behind commitment based planning is that only the team can decide how much to commit to and management is not allowed to dump more work on the team that it can handle. It is being misinterpreted here to mean that the team must accomplish X story points no matter what. This was never the intention.

    “The retrospective is important enough for Alistair Cockburn to list it as one of only three “must have” properties in the Crystal methodology, but it seems to have gone missing from Scrum.”

    Nonsense. Retrospectives have always been part of Scrum and they still are.

    “The whole concept of fixed sprints with coordinated events at the start and end of a sprint were gone. Note that the conecpt of cadence didn’t disappear. It just got decoupled.”

    This is not necessarily a good thing. However, I can see how this argument has appeal to software developers. In software, tight coupling is generally considered a bad thing. Trying to apply this principle to human activities is a fundamentally flawed concept. By not giving any guidance on when to do retrospectives, you are essentially making them optional. What makes you think that retrospectives would be done any more often with Kanban than in Scrum?

    Scrum is not perfect and no process will succeed if applied too literally without adjusting to context and unique situations. However, I find it disappointing that all this Scrum bashing is happening as people promote Kanban as the next silver bullet.

  27. Fun Chiat Says:

    I think the comments you infer belong to a moment before the current Scrum Guide October 2011. Scrum did evolve. Retrospective is indeed optional at one time. Basically you are right about Scrum as at April 2012.

    I really wish to understand the heart of Kanban, for professional knowledge. They keep saying each camp have misunderstandings about each other. I want to understand this totally. How can I achieve that understanding?

  28. MGC Says:

    I think that it’s valuable to study Kanban as well. Every methodology that comes along has value because it attempts to solve a problem. This includes Scrum, Kanban, and lots of other things as well. Each one is a tool that solves a particular set of problems. The more tools at our disposal, the better off we’ll be if we know how to choose the right tool for the job.

    Where I take issue is that each new methodology is promoted as the next silver bullet that’s going to solve all our problems, while last years’s silver bullet is vigorously discredited. It get’s adopted with great enthusiasm as everyone jumps on the bandwagon and then thrown under the bus a few years later when disappointment sets in and something else comes along. Then the cycle repeats itself. I’ve seen this happen with Scrum and the same thing will happen with Lean and Kanban in a few years.

  29. Valeriy Says:

    I think that first of all you guys did not have good enough scrum master. The common cause variation problem could be solved easily by good SM. Another thing is that if you can omit sprint commitment, then you do not need scrum at all. Then, I wonder why you started practicing scrum.

Leave a Reply