Don’t fall into the Acceptance Criteria trap

Posted on November 3rd, 2011 in Agile by siddharta || 13 Comments

There has been a lot of talk in the agile community about acceptance criteria. How having defined acceptance criteria beforehand makes development a lot easier. And it does. I’m a big fan of acceptance criteria. But just because someone write an acceptance criteria on the back of a card, doesn’t mean that what they’ve written is the correct criteria.

That might sound like a dumb thing to say. After all, if someone writes a wrong acceptance criteria, then what can we do about it? It’s their fault, they’re just going to get the wrong software, and they have no reason to complain… Right?

The new requirement specification

Remember the old days? Customer gives you a requirement spec, you implement the spec. Customer gets the software, and then requests changes. “But, thats not in the spec” you say.

Fast forward a decade, and what do we have

Old days Today
What do we build? Look at the requirements Look at the acceptance criteria
How do we test Conformance to requirements Conformance to acceptance criteria
When are we done? Requirements met Acceptance criteria pass
The hitch Requirement were flawed Acceptance criteria were flawed

I love acceptance criteria

I earlier said that I love acceptance criteria, and now I’ve just gone and bashed it, comparing it with that waterfall document. Whats going on?

Well I love acceptance criteria at the story level. Over there, acceptance criteria do a wonderful job. It lets us know when a story is done and when we can move on to the next story.

The problem comes when you roll it up and use acceptance criteria to steep the ship. Just because we are meeting acceptance criteria doesn’t mean that we are building something useful. We build software, not to meet acceptance criteria, but to solve some business or customer goal. And lets face it, goals are nebulous beings, which most of the time cannot be translated into acceptance criteria. Look at these examples

  • I’m building a search engine. A key goal is to deliver more relevant hits than Google. What are “relevant hits”?
  • I’m making a game. A key goal is that the game should be fun. What is “fun”?
  • I’m writing an iPhone app. A goal is that it should be easy to use. What exactly is “easy to use”?

None of these goals can be translated into acceptance criteria. The only way to get there is to do something, make changes, and repeat.

Build with acceptance criteria, steer with goals

Now, you might think this is obvious. After all, being agile is all about responding to feedback. So it is rather surprising that many agile teams have their sprints and acceptance criteria and definition of done, but have no techniques to validate the application against goals and change direction. In essence, they are playing the “requirements specification is God” game all over again.

If you are a product owner, then your job is NOT to simply to accept or reject stories, but to keep validating the product being built against goals and steer the ship.

Doing Distributed Agile?

Share and collaborate with distributed teams with our electronic agile board tools. Get all the benefits of electronic tools without sacrificing the benefits of physical boards. Supports Scrum taskboards, Kanban boards and user story maps. Check it out!

13 Responses to “Don’t fall into the Acceptance Criteria trap”

  1. Andy Till Says:

    But who pays for the changes required when the specification does not meet the goals? This is simpler if the customer is paying on an agile contract but almost all software that I am aware of is done on a fixed contract.

  2. siddharta Says:

    Hi Andy,

    We simply have to plan for changes.

    For example, lets say I’m getting a website built. The vendor knows that I will not accept the first design they come up with. So they will show me some options, and there will be a lot of back and forth. At the same time the project will be a fixed price. The vendor knows that there will be this feedback process for every customer, and incorporates that into the plan and the price.

    Similarly, in complex software development, we have to assess the uncertainty. Which parts are we certain about and which features have a lot of uncertainty. Then we have to buffer accordingly and incorporate that into the price.

    Of course, if there is a LOT of uncertainty then fixed price is probably not a good idea.

  3. siddharta Says:

    The other option of course is to compensate the scope. Every project has some smaller low priority features. We can choose to take more time to tackle the uncertainty in important features and we can drop some low priority features from the scope in exchange.

  4. Andy Till Says:

    So is it fair to say that you would budget in for some failed iterations, in that the customer rejects the work for whatever reason: goals changed, had a better idea, software not good enough?

    This would make agile overall more expensive to the customer but more likely to build something useful as you say in your article.

  5. siddharta Says:

    Yes thats one option, but like mentioned above thats not the only one. Here is how I would look at it

    1) Plan for changes: When you are working on a domain where you know beforehand that there are going to have changes in certain parts. eg: If you do website design for many clients, you already know before hand how many feedback loops are generally needed

    2) Features trading: In typical software dev, you have some important features, some low priority features. So you can trade doing more feedback on important features and cut scope on less important features. You basically trade scope keeping time & budget fixed.

    3) Agile contracts: For high uncertainty projects (eg: new innovations, untested products etc) it is perhaps better to use a more agile suited contract, like time & material instead of fixed price, since there is likely to be very high amount of churn

  6. Product Owner should deliver Enabling Specifications | bekinin.com Says:

    [...] Govindraj stresses on regularly validating the work against product goals. So it is rather surprising that many agile [...]

  7. Mr Stressed Test Says:

    Who’s job is it to define and write the acceptance criteria down. A project manager I currently have believes it should be a testers job?

    Many thnkas

  8. Product Owner should deliver Enabling Specifications | Scrum Alliance Says:

    [...] Govindraj stresses on regularly validating the work against product goals. So it is rather surprising that many agile [...]

  9. bepenfriends Says:

    Hi Now let me put a scenario here.

    As a product owner, We are asking to have a logout button in the mobiel application (to be used in iPhone) that logs you out from application. now during development team is saying iPhone has default way to do that so no need to have logout button.

    As a product owner you received the feedback and validate it and then say you don’t need it. When released the exact product (built properly) customers are now asking where is the logout button. in this scenario both team and PO is not able to build a product that customer wants.

    We can consider acceptance criteria as a minimal requirement and during conversation we can better it.

    Also, The story goal that you provided is very vast. If we give this as a story then we are giving our vision as a story. Also these stories cannot be fit in the story definition. we might need to split the story into a sizable chunk by using the concept of INVEST.

    Just a thought

  10. Alain Says:

    I don’t agree with the whole statement that goals are foggy and can’t be put into acceptance criteria. especially the examples given.

    Acceptance criteria are all about the question “When is the result good enough and how can we prove it”.

    Take the example of the game must be fun. Define fun. It right you can’t, but you can define acceptance criteria: When asking a 100 people how “fun” the game is on a scale of 1 to 10, 90% should give it a score of 8 or above.

    The ease of use one is even a classic. 90% of new users must be able to use the main functionalities (and here you list them) of the application within 15 minutes, without having to refer to the manual.

    The relevant hits could have an acceptance criteria like: 50% of the users only click on the first 5 hits returned or 80% of users, click the same links again when searching for the same keywords.

    The whole point of acceptance criteria is just to make the goal clear. To know what the customer wants. To manage the expectations.
    By trying to define acceptance criteria and discussing them with the customer, you actually get a lot of feedback. It helps ensure that everyone knows what to expect and provides the basis to prove it by doing tests.

    The acceptance criteria are not the tests since you can device different ways of testing each of the ones mentioned above, but they are the basis for the tests.
    You can (should) even define acceptance criteria for your tests. (How lagre a group of testers do I use? How often do they have to repeat the test? all at once or separatly? Under what conditions? …)

  11. siddharta Says:

    Hi Alain,

    I agree with what you say. You are talking about a higher level of acceptance test, where you test with the end user or in production. This is what I meant by Goals.

    That is a different kind of acceptance test than the acceptance test that the testers use to validate the development of a story.

    When it comes down to it, its the product owner who has to sit with UX folks and run these kinds of tests with the regular releases with real users, and use that information to steer the product.

    Many times the development level acceptance criteria is used as a sign off step that the feature was developed correctly, and never check against the goals.

  12. Notun Bazar Says:

    Can anybody help me with a sample detailed test document please?

  13. siddharta Says:

    Notun, do you mean an acceptance test document? A good structure is the “Given, When, Then” format for each of the acceptance test cases. The added advantage being each one can be easily converted into an automated test

Leave a Reply