By far the toughest problem facing organizations that want to adopt agile methods of software development is to be able to adopt the values and principles that underlie agile methodologies. This is because agile principles often run counter to the established organizational culture in most companies. Changing this culture is hard, and it takes a lot more than just data to achieve this change.
If the data indicates that one method is better than another, common sense would tell you to adopt the better method. However, because change is so difficult, this is a very difficult thing to do.
I came across the story of Semmelweis recently. Here is how it goes:
Ignaz Philipp Semmelweis was a hungarian doctor who worked in the Vienna General Hospital in the mid-1800s. The hospital had two maternity clinics. At that time, about 15% of mothers would die after childbirth due to “childbed fever.” Semmelweis observed that one clinic had a much lower mortality rate than the other. Women who gave birth at home also had a low incidence of childbirth fever. Semmelweis set out to find the cause.
The concept of germs and infections was not yet known at that time, and the prevailing theory was that all diseases are caused by an imbalance of the four liquids (blood, phlegm, yellow bile, black bile) in the body. Thus the prevailing wisdom was that childbed fever was because the mother had an imbalance of the liquids.
One day Semmelweis’ friend Jakob Kolletschka died after accidently getting a cut while doing an autopsy on a corpse. The symtoms were strikingly similar to mothers with childbed fever. Semmelweis hypothesised that the fever could be caused by something in the corpse being transmitted by the physician to the blood stream of the mother.
To test this, Semmelweis asked everyone to wash their hands in a solution of chlorine before handling childbirth. Immediately mortality rates dropped down to 1%, even going to 0% for some months.
Presented with the data, the medial community completely rejected Semmelweis’ ideas. The idea that physicians themselves were responsible for the deaths was completely unacceptable. Further, the idea that diseases could be carried through invisible particles on the hand were rejected for having no basis in reality.
Doctors only had to try washing hands in chlorine for a month to see the effects for themselves, but the idea was considered so loony that nobody bothered with it.
Semmelweis went mad trying to convice others and was forcefully admitted to a mental assylum where he died 14 days later. It was only 20 years after his death that the germ theory explained his observations.