Many developers are confused with the jargon used by test and software engineers when developing tests. Even test developers (TE/SET/SDET) are confused with these terms.
In general, test suites occur in the following varieties:
To add to the confusion, there are:
We understand that it is good practice to run a product against a set of Acceptance Test suites before releasing code, or before a feature can be considered complete. As our team performs Test-driven development (TDD) and the Behavioral-driven Development (BDD), developers will need to create an acceptance test first, and are initially confused on what the tests should look like.
I refer you to Jonas Bandi’s blog. Jonas provided a good explanation of what an acceptance test is.
To summarize, an Acceptance Test is a way to represent a test case; a test case that meets the acceptance criteria of a story. An Acceptance Test is basically the contract between the product owner and the developers, when feature development starts in a development life cycle (be it Sprint or Waterfall model). A story in this context refers to a feature set that product management wants delivered to the customer.
So, the question would be: What would be the best approach to write acceptance tests?
My answer would be: It depends…
To clarify my answer, let me provide 3 scenarios. You see, unit tests, integration tests and end-to-end tests can all be written as Acceptance Tests; depending on what the feature is.
The key point here is that the feature specification should be known beforehand. If there is an unknown, either that has to be fleshed out, or the story needs to be sliced into smaller pieces and addressed independently.
The Ancestry API team uses Agile for their development process. In order to fully benefit from Acceptance Testing, the act of designing and writing has to be part of the Sprint cycle. This means that everyone (stake-holders, dependent parties, developers, testers etc.) has to participate in a grooming meeting to vet out the requirements of a proposed feature which will be developed.
The grooming/planning meeting for vetting out the acceptance criteria, which will in turn be used to construct the acceptance test; provides an avenue for the customer/product owner to convey the requirements and expected scenarios to the development team. The important concept is that there should be a two-way communication between the development team and product owner (e.g. stake holders, program managers etc.). If there are any ambiguities during the conversation (related to implementation, dependencies, testability etc.), those have to be fleshed out before a story can be considered READY for development to start.
The READINESS of the story is critical because unknown variables may impact the velocity of the team. Problems such as incorrect architecture may throw up the entire sprint cycle and render the implementation unusable.
Our team has benefited greatly from this approach when unknowns were identified by upstream dependencies. This resulted in a conclusion that there is not enough features and data provided by library methods in order to implement the required scenarios. As a result, the story was considered not ready and was sent back to the drawing board. This saved the team plenty of time compared to not detecting the problem and much effort being wasted in designing the wrong implementation which does not meet the requirement.
The development of the acceptance test is an integral part of the development process. One approach that we have taken is to represent an acceptance test in a form which is readable to the product owner. In our case, we utilize Gherkin in .NET via the SpecFlow framework.
Benefits of using the SpecFlow framework:
An Acceptance Test would ideally be a direct translation from an acceptance criteria define by the stake holders of the feature. There is a melting of skill sets between stake holders and development during this stage. Stake-holders would write acceptance criteria in the Gherkin form, being aware of:
What results is a specification that ensures:
The development team would rewrite this in SpecFlow, automate the tests and introduce the new acceptance tests into the test suite. When all acceptance tests pass, the feature would be considered completed in terms of meeting the requirements of stake holders.
Of course not. There are plenty of other tests to ensure the quality of the product; a topic which is not the focus of this article. However, acceptance testing drives the behavioral-driven design approach of the development process and is critical in ensuring a healthy communication channel between stake holders and the development team.
I will be giving a presentation on Continuous Delivery at Ancestry.com on June 5, 2013 at Caesars Palace, Las Vegas, NV during the Better Software & AGILE Development Conference WEST.
This blog is focused on the technology used behind the scenes at Ancestry.com. It’s a place to learn about the experiences we have, the challenges we face and the solutions we use on our engineering and tech teams to create the Ancestry.com experience.Visit Ancestry.com