Scrum (and agile in general) has been responsible for a lot of the current visibility on softer, social aspects of software development – empowered teams, tight collaboration, higher communication and so on. XP’s engineering practices have also been extremely valuable. Those parts work well and we use them along with Kanban.
The problem we found with scrum is that it works well in a certain context: single team, completely dedicated to single project, completely cross functional, no external dependencies, product owner knows exactly what to build, experienced team members with good skills in engineering practices. In such situations it works well. But when you move away from the ideal context then you run into issues. External dependencies? Not fully cross functional? Boom. When I was working in Singapore, we were doing Scrum on embedded systems, and there had to be one set of manual acceptance tests that took a week to complete and couldn’t be automated. Trouble!
There is a joke that goes something like this: A physicist, engineer and psychologist go to a dairy farm to maximize milk production. The engineer derives the optimum packing of cows. The psychologist derives the optimum colour of the barn. The physicist starts “First, assume the cow is a sphere..”
Its the same thing with scrum. If the organization is already at the ideal state (the sphere) then scrum can work really well. If it is not a sphere, then before you can get started, the first step in scrum is to make it a sphere.. and then things get messy.
99% of the time, the organization itself is the “impediment”. It is interesting, because in many cases it is an impediment only from scrum’s point of view. Almost all the “ScrumButs” out there are because of this requirement that everything is an impediment until you get to the sphere… and only then can scrum start working. Every time scrum doesn’t work, invariably it is something to do with the environment not being a sphere – team is not cross functional, PO doesn’t know what to do, developers don’t understand TDD, testers don’t know coding etc.
I really do understand the need for adopting new values and new skills, but they don’t come about overnight. Until that time, you are going to have a hard time with scrum.
The other problem with scrum is that it has an underlying undercurrent of being anti-management. Scrum is essentially a solution to tell management to keep their hands off the team and leave the team alone to figure out how to meet the commitments by themselves. This negative view of management runs through the process – from dividing up people into chickens (management) and pigs (team), preventing all external interruptions and so on. The message is to stop micromanaging and leave the team alone. In many, the scrummaster is equated to a guard dog that protects the sheep (team) from the wolves (management).
Unfortunately once you train ScrumMasters to take a dim view of management, then it gets tough to get management cooperation when you need to interact outside the team.
IMHO, these two places are where Kanban really helps. With Kanban, you no longer have the requirement of making the organization a sphere before even getting started. You can start out with things exactly how they currently are. And Kanban is much better at integrating outside the team. There is still value in doing a lot of things that scrum requires – cross functional teams or engineering practices. You can now do them one by one at a pace at that suits the organization.