What Are Acceptance Tests?

Many developers are confused with the jargon used by test and software engineers when developing tests. Even test developers (TE/SET/SDET) are confused with these terms.

In general, test suites occur in the following varieties:

• Unit tests
• Integration tests
• End-to-end tests

To add to the confusion, there are:

• Functional tests
• Acceptance tests

We understand that it is good practice to run a product against a set of Acceptance Test suites before releasing code, or before a feature can be considered complete.  As our team performs Test-driven development (TDD) and the Behavioral-driven Development (BDD), developers will need to create an acceptance test first, and are initially confused on what the tests should look like.

I refer you to Jonas Bandi’s blog. Jonas provided a good explanation of what an acceptance test is.

To summarize, an Acceptance Test is a way to represent a test case; a test case that meets the acceptance criteria of a story.  An Acceptance Test is basically the contract between the product owner and the developers, when feature development starts in a development life cycle (be it Sprint or Waterfall model).  A story in this context refers to a feature set that product management wants delivered to the customer.

So, the question would be: What would be the best approach to write acceptance tests?

My answer would be: It depends…

To clarify my answer, let me provide 3 scenarios. You see, unit tests, integration tests and end-to-end tests can all be written as Acceptance Tests; depending on what the feature is.

• A unit test may be appropriate if we are delivering something which is simple, doesn’t require complex dependencies and strict behavior between interfaces.
• Integration tests are commonly utilized when the acceptance criteria are restricted to interactions between well-known modules.
• Most of the time, features are shipped based on its usability and the ability to perform a certain task or function. This would involve user scenarios that may involve multiple feature sets to work together to achieve that wanted result. This is when an end-to-end test would be handy. Such tests are normally the closest to real world scenarios.

The key point here is that the feature specification should be known beforehand. If there is an unknown, either that has to be fleshed out, or the story needs to be sliced into smaller pieces and addressed independently.

Integration of Acceptance Tests into Development Cycle

Be it the Waterfall or Agile model, Acceptance Tests are an integral part of the TDD and the BDD methodology applied in a software development process.

The Ancestry API team uses Agile for their development process. In order to fully benefit from Acceptance Testing, the act of designing and writing has to be part of the Sprint cycle. This means that everyone (stake-holders, dependent parties, developers, testers etc.) has to participate in a grooming meeting to vet out the requirements of a proposed feature which will be developed.

The grooming/planning meeting for vetting out the acceptance criteria, which will in turn be used to construct the acceptance test; provides an avenue for the customer/product owner to convey the requirements and expected scenarios to the development team. The important concept is that there should be a two-way communication between the development team and product owner (e.g. stake holders, program managers etc.). If there are any ambiguities during the conversation (related to implementation, dependencies, testability etc.), those have to be fleshed out before a story can be considered READY for development to start.

Catching Problems in Specification Phase

The READINESS of the story is critical because unknown variables may impact the velocity of the team. Problems such as incorrect architecture may throw up the entire sprint cycle and render the implementation unusable.

Our team has benefited greatly from this approach when unknowns were identified by upstream dependencies. This resulted in a conclusion that there is not enough features and data provided by library methods in order to implement the required scenarios. As a result, the story was considered not ready and was sent back to the drawing board. This saved the team plenty of time compared to not detecting the problem and much effort being wasted in designing the wrong implementation which does not meet the requirement.

Acceptance Criteria to Test Code

The development of the acceptance test is an integral part of the development process. One approach that we have taken is to represent an acceptance test in a form which is readable to the product owner. In our case, we utilize Gherkin in .NET via the SpecFlow framework.

Benefits of using the SpecFlow framework:

• Reusability of GIVEN, WHEN and THEN statements
• Isolation of responsibility statement makes execution and validation easier in the test
• Regular expressions in statement definition allows flexibility in reusing statements for various scenarios
• Good integration with Visual Studio (debugging etc.)

An Acceptance Test would ideally be a direct translation from an acceptance criteria define by the stake holders of the feature. There is a melting of skill sets between stake holders and development during this stage. Stake-holders would write acceptance criteria in the Gherkin form, being aware of:

• Initial states of the scenario (GIVEN statements)
• Feature to implement (WHEN statements)
• Expected outcome (THEN statements)

What results is a specification that ensures:

• Development team is building the feature correctly
• Feature team is building the right feature
• Product is testable – which allows automation
• All dependencies and ambiguities have been fleshed out

The development team would rewrite this in SpecFlow, automate the tests and introduce the new acceptance tests into the test suite. When all acceptance tests pass, the feature would be considered completed in terms of meeting the requirements of stake holders.

Is That The Only Testing Needed For a Feature?

Of course not. There are plenty of other tests to ensure the quality of the product; a topic which is not the focus of this article. However, acceptance testing drives the behavioral-driven design approach of the development process and is critical in ensuring a healthy communication channel between stake holders and the development team.

Future Talks

I will be giving a presentation on Continuous Delivery at Ancestry.com on June 5, 2013 at Caesars Palace, Las Vegas, NV during the Better Software & AGILE Development Conference WEST.