When Should You Release Before You Test?

Posted on November 10th, 2009 in Agile by siddharta || 4 Comments

Mark Levison wrote an nice post called My Challenge to Agile Tool Vendors which caught my attention for obvious reasons, since we are agile tool vendors ourselves. I was going to post a comment, but it became a whole post by itself.

In it he says (italics mine)

I keep on seeing announcements for the next great Agile Task Tracking tool. I just saw one posted to Scrum Development where it’s author said: “I haven’t done much testing, so if you find a bug and want me to fix it let me
know :-)”.

My reply: Congrats I’m sure you have an excellent application. I’m wondering if you see the irony – you’re posting to an Agile group and say that your app has hardly been tested? What is your definition of Done? Do you use TDD? At least Unit Testing? What was your approach to acceptance testing?

The implied message here is that releasing an app un-tested is un-agile. I disagree.

Releasing something new and unknown is a unique situation where Agile needs to be tweaked. In fact this was one of the topics discussed in the Mumbai Agile User Group last month.

“Regular” scrum assumes that the PO knows the market or customer well enough to make decisions on their behalf. This is true for certain kinds of applications.

However, startups regularly face situations where even the PO has a limited amount of knowledge, and the only way to learn is to put things out and see the reaction and figure out whether you are in the right track at all. In such circumstances, it makes sense to validate the idea first, before spending further effort on it. The answer to the question “What was your approach to acceptance testing?” could well be “release it and find out”. That’s a valid answer in certain contexts – as always the context matters.

The tool in question, Taskboardy, is a little taskboard app written for Google Wave. It’s a single developer tool. There are a lot of questions to be answered here – Are people using Google Wave? Is a taskboard tool on Wave something people want? What about the unique replay feature?

In this particular circumstance, I think it makes a lot of sense to get it released as soon as possible so that those questions get answered. Think of it as a big public alpha version. Once the questions get answered, then you can go back and redirect your efforts based on the learning you got – either go back, add tests and pay back the technical debt or scrap the feature and start over.

As a team, you need to assess what your risk areas are, and then tailor the process to suit it.

Doing Distributed Agile?

Share and collaborate with distributed teams with our electronic agile board tools. Get all the benefits of electronic tools without sacrificing the benefits of physical boards. Supports Scrum taskboards, Kanban boards and user story maps. Check it out!

4 Responses to “When Should You Release Before You Test?”

  1. Mark Levison Says:

    I mis-understood the author’s intent. He’s not releasing a tool, so much as testing the waters to see if there is any interest – that wasn’t clear from what he said on Scrum Deve. For these purposes I think he did the right thing. I still stand behind the statement I would show a client a released tool where the vendor couldn’t answer the above questions.

  2. Mark Levison Says:

    Which begs the question – how would Silver Catalyst answer my questions?

  3. siddharta Says:

    “I still stand behind the statement I wouldn’t show a client a released tool where the vendor couldn’t answer the above questions.”

    Why is that? What if you came across the perfect agile tool.. but it was developed with waterfall? I don’t see how that makes it less valuable for the customer.

    I’m pretty sure all the agile tool vendors could give good answers to the questions. But at the end of the day, no one want to be agile for the sake of being agile. What really counts is whether you are adding business value for your customer.

    A more interesting question to ask is ‘In what way does xyz tool allow my customer to be more agile?’ :)

    To answer your questions,

    Our main product, Silver Catalyst, started with Scrum but we use Kanban for this project now. Why? As the project context changes, the process needs to change accordingly.

    # Your Definition of Done

    Coded, unit tested (for the backend), integration tested (for the UI), manually tested

    # Whether you use TDD? Or at least Unit Testing?

    Backend code is TDD. Some UI code is unit tested in Jsunit. Interaction code is functional tested in Selenium.

    # What do you do for Acceptance Testing?

    Code has to pass the automated functional tests plus a manual test

    # How often do you release?

    Once a week. We make lots of small releases. But that doesnt mean that a story takes a week. Since we use Kanban, development is decoupled from the release cadence.

    # What did you learn in your last retrospective?

    Action point to improve the error reporting capability of the selenium tests so that we get notification failures even while the tests are still running.

  4. Mark Levison Says:

    It is possible to develop good software, with great quality that meets a customers needs using waterfall – but it seems akin to throwing darts randomly at dart board (the way I play incidentally). In the end it sends the wrong message to your customer – claim to be in the agile software business but don’t use Agile. Odd.

Leave a Reply