Documenting Azure DevOps Test Plans
Document your Azure DevOps Test Plans from your project!
Let’s face it, building your documents for your Azure DevOps Test Plans can be a heavy task.
The same thing is true for requirements that live within your Azure DevOps project, but Azure Test Plans poses a unique challenge. Because Modern Requirements4DevOps adds the “Smart Doc” module to the lineup of Requirements Management features that you can use inside your project, you can create documents out of all your requirements. This, if you’re a BA, is nothing less than a delight.
But whether you’re a BA building a Test Strategy for your team to outlines your testing approach, or you’re a Test Lead documenting the specific Test Plans that will run and what they cover, the task of connecting your Testing Work Items to your documents can be… interesting.
That’s why in this article we cover a brief overview of how Azure DevOps Test Plans module works, as well as how you can document your test outcomes easily. As we cover these two topics you will also learn a bit about the architecture of Azure DevOps itself and we’ll dive into how understanding this can make you better at understanding Azure DevOps as a whole.
So stay tuned and read until the end!
Looking for the quick overview and don’t care about the specifics? Skip down to the video!
Otherwise, this is what you can expect to learn in this article:
- What is Azure DevOps Test Plans?
- How does Test Plans work?
- How do I choose a Test Suite type?
- Building my first Test Plan
- Running my first Test Plan
- Seeing the outcomes of a Test Plan
- Documenting my Test Plan
What is Azure DevOps Test Plans?
Azure DevOps (ADO) is a great place for your Quality Assurance team to build out their Test Plans for a given project. The important thing to note here is that by bringing your Testing Team onto the Azure DevOps platform, you can effectively use your ADO project as one single-source of truth.
You can have your Devs using the Repos and tying development work to requirements, you can have your business analysis teams building requirements and attaching test cases, and you can have your QA team using those Test Cases, and requirements made by the BA team, to run all of your Testing.
Now Azure DevOps Test Plans can be used for both Automated and Manual testing.
I, however, do not come from an automated testing background and can therefore only speak to Azure DevOps Test Plans as it pertains to Manual Testing. Therefore, in this article, the Azure DevOps Test Plan hub will be discussed as the place for manual testing.
What I will say, however, is that Test Plans are an incredible place for your team to do their manual testing needs for many reasons.
The main benefit is that by deploying your manual testing operations from here, you can directly interact with the Test Cases that were already built inside of your Azure DevOps project. This means if you have individuals tasked with building Test Cases who fall into your Business Analysis team, their contributions will be easily pulled into your testing workflow. This capability adds a great benefit for teams who have BAs that are capable of building Test Cases for the QA team – if you’re a BA who does this, you will see how it works later in the article.
Test Plans also offer some incredibly helpful manual testing features that will help your testers provide meaningful input on the outcome / results of their tests. This output goes far beyond the ability to mark a specific test step as passed or failed, but also provides an interactive window for both marking the outcome of individual test steps, capturing live screenshots and recordings, as well as provide comments that are directly tied to the run of that test case.
This means when you review any Test Runs which you have completed, you will be able to easily identify what your tester was experiencing when they marked an individual test step as blocked / failed. The level of insight that comes with using Azure Test Plans means that even an external manual testing team can contribute thoughtfully, and allow teams to identify and interpret failed results easily.
That’s why Azure Test Plans are a great place for your manual testing.
How does Azure DevOps Test Plans work?
There is one obvious reason for Azure DevOps Test Plans to have thousands of Google searches a month.
That reason, is that having your Testing team working in the same environment as your Business Analysis team and Development team, you create a single-source of truth for everyone working on a project. But many of those searches are about “How does ADO testing work?!”.
It really isn’t complicated to any degree, but in my experience many teams are coming from a proprietary, in-house built application they use for Testing. For this reason, teams often need an intro to how the magical Azure DevOps Test Plans work.
Let’s get into the details.
Test Plans is a hub that allows you to create, you guessed it, Test Plans.
I’m not talking about the Documents that you might be thinking of, but we will get into these later.
Test Plans works by allowing you to create a container (Test Plan) to group your Test Suites. You first create a Test Plan, then break it down into the associated Test Suites. The terminology here can be a bit confusing, as many teams choose to use the word “Test Plan” in ways that sound egregious to other teams… so let’s cover them both.
What is a Test Plan?
A Test Plan is a container to group Test Suites in.
What is a Test Suite?
A Test Suite is a collection of Test Cases.
In Azure DevOps, you will first create your Test Plan and then you will create your Test Suites inside of it.
Now, when you create a Test Suite inside your Test Plan there are three options.
Take a look at the graphic below to explain the differences between each:
Time to Read: 20 minutes
Test Plan 1:
Test plans contain test suites.
Test suites can be of three types. Read each of the following suites to see the three different types.
- Test Suite 1
- Test Suite 2
- Test Suite 3
Test Suite 1 – Req-based
Requirements-based test suites are the simplest and most traceable. They pull in all of the Test Cases for a given Requirement.
- Test Case 1
- Test Case 2
Test Suite 2 – Query Based
Query-based test suites pull in a group of tests from your project, irrespective of what requirements the test cases are linked to.
- Test Case 3
- Test Case 4
Test Suite 3 – Static Based
Static-based test suites are used either as containers to group other test suites, or to group a specific set of test cases.
- Test Case 5
- Test Case 6
- Test Suite 1
- Test Suite 2
Depending on which Test Suite you have chosen to create, building a Test Suite will allow you to pull in all of the Test Cases that you want to Test. Once you have your Test Suites created, and the Test Cases inside, you will be able to execute the entire Suite and that will become a Test Run.
But before you get to running your Test Suites (therefore creating a Test Run), you need to really understand what each of these Suites offer.
How do I choose a Test Suite type?
If you just found yourself asking this question, don’t worry… you aren’t lost. It’s easy at this point, and some options can simply be better than others.
So which Suite do I choose? It’s a great question and the answer is situation dependent.
Let’s quickly discuss the Use Cases and the benefits of each of these Suite types.
Remember: Test Suites (of any type) can only include Test Cases.
1) Requirements-based Test Suite
A Requirements-based suite is one which pulls in all of the Test Cases for that requirement.
This means when you go to build a Suite you will not be choosing Test Cases. You will instead simply choose the Requirement.
What happens next is the real magic.
Any Test Cases that have already been built and connected to that requirement get pulled in and populate your Test Suite. You can then run all of the Tests for that given requirement and see from each Test Run (found under the Test Plans tab on the left side of your screen) and conclude that the requirement has been fully met and has passed or that it needs work due to it failing.
Why is that magic?
Remember earlier how I said that Requirements-based Test Suites pull in all of the created Test Cases for a given requirement? That means if your BA was able to add in the relevant Test Cases for a requirement when they created it, or if your developers were able to construct the relevant Test Cases when tying in their development work to a requirement… your QA team doesn’t need to redo this work. This is the benefit of a single-source of truth model wherein your teams can benefit from the work of another team easily.
What’s more is that if your Test Lead decides while building a Requirement-based Suite that they would like to create more Test Cases for that requirement, then the Test Cases they create will be automatically linked to the requirement itself. That means whether Test Cases are being built from the BA or Dev side, or whether they are being added to a Requirement by a Test Lead, every team can contribute to the same group of Work Items.
If you use Requirements-based Test Suites, your team can create Traceability Matrices that show you not only how your Test Cases align with your requirements, but can also show you whether a Test Case has passed or failed for any given requirement.
For this reason, and a few more, I often tell people that Requirements-based testing is the simplest and most traceable.
As for the Use Case for Requirements-based testing, I would suggest using this method of Testing when you have the Test Cases for any given requirement tied directly to that requirement. This will allow you to easily bring in the available Test Cases that have already been contributed.
2) Query-based Test Suite
Next up is Query-based Testing which has a slight difference from the Requirements-based Testing mentioned above. Query-based Test Suites afford teams the ability to create a Test Suite using any criteria available.
What this means is that when you are creating this type of Test Suite, it asks you how you want to pull in Test Cases, and allows you to construct an ad-hoc query to do so. This means you are not concerned with how Test Cases are connected to specific requirements, and instead allows you to pull in the Test Cases that match a specific criteria.
It is important to note, that this does not use the queries your Business Analysis team would create in Azure DevOps; however, the interface for constructing this ad-hoc query is nearly identical.
When would you use a Query-based Test Suite?
In some cases, teams will run tests on all of the Test Cases that are currently in a given iteration, or run tests on everything with a given tag, and these conditions are not necessarily tied directly to a given requirement.
The idea behind Query-based Test Suites is that you can easily include any Test Cases from your project that you would like to test, irrespective of their affiliations with Requirements.
3) Static Test Suite
In Azure Test Plans you can have nested Test Suites.
Static Test Suites are used as containers for other Test Suites, or as a place to create Test Cases ad-hoc.
Most of the time you will be using Static Test Suites as a means of grouping Test Suites together.
Building your first Test Plan
Building your first Test Plan is simple.
The Test Plan you create in Azure DevOps Test Plans will allow your team to begin structuring how Tests will be run to ensure that your project fulfills the quality standards. As described above, you will be creating a Test Plan, which will include your individual Test Suites.
Perhaps you want to build a Test Suite for all of the requirements in a given iteration.
This will be the example we cover in this article.
One way of tackling this is to build your Test Plan called “Given Iteration Test Plan”. To do this go to the Test Plan tab on the left of your Azure DevOps interface, and select “+ New Test Plan”.
Next, you want to name this Test Plan “Test Plan for Iteration 1”.
You can then select if you would like this specific Test Plan to be for a given area path or iteration, otherwise you can leave this blank (in these example images I have left those two fields as the default options.)
We now have our Test Plan created and are brought to the Test Plan to create our Test Suites.
Recall that our purpose here is to assure ourselves that all of the requirements from a given iteration are tested fully. So let’s add in the Test Suites for each of these requirements (these will be Requirements-based Suites).
Here we need to point out two things:
1) The goal is to create a Test Plan that tests all of the requirements for a given iteration. Therefore, we want to pull in the Test Cases that already exist for these requirements, or attach new Test Cases to this requirement if we add to our Test Suite.
2) We don’t want to bring in requirements, we want to bring in the Test cases associated to requirements. This point is just a reminder that even when you build a Requirement-based Suite, you are creating a group of Test Cases not requirements.
So let’s create Test Suites (groups of Test Cases) for each of the requirements in iteration 1.
Above you can see the query you would need to build to pull in the Iteration 1 requirements, then you can simply click “Run Query” as shown above.
Now let’s just stop for one moment.
When we run the query we will see all of our Iteration 1 Work Items, and we will only select a few.
But the question you should ask yourself before moving on is: “how many Test Suites will I create if I select 4 requirements from this list?”
The answer to the above question is 4.
When you are selecting the requirements, remember that you are simply saying I want to have a Requirements-based Test Suite for each of these requirements’ Test Cases.
Let’s take a look at the outcome of clicking the blue “Create Suites” button above.
If you click the image above, you can open it and see that Test Suite for the work items 306 and 307 Test Cases in them. This is shown by the (1) and the (3) respectively.
Running your first Test Plan
Now that you have a Test Suite for all of the necessary requirements, you can either begin assigned Testers, or add in Test Cases to fill out the Test Suites for those requirements.
If you read the article above, this is a great time to reiterate the benefits of Requirements-based Test Suites.
Requirements-based Test Suites use the existing Test Cases that might have been built for a given requirement, and connect any new Test Cases created in the Test Suite to your requirement.
This means more traceability and a more connected workflow between your teams.
Now how do you run your Test Plans?
You do a Test Run.
I know that for some people another “Test something” added to their vocabulary might feel like overkill.
But it’s this simple. Test Plans have Test Suites which are a group of Test Cases.
When a manual tester chooses to run those Test Suites to see if it passes or fails, that action of “running the Test Case(s)” will create a Test Run. A Test Run is the outcome of running one or multiple Test Cases.
Let’s do a Test Run of one of our Test Suites.
For our Test Run, we will be running all three Test Cases in this suite.
We will do our Test Run for the web application we are currently building.
I am launching this from the Chrome Browser, and I would suggest you to do the same.
If you do not have the extension enabled and connected to your project, I would encourage you to install the chrome extension here.
Once you select “Run for web application” a new window will be launched for your Test Run (see below).
This window can be dragged beside the window containing the web application you are currently testing.
You can then see the individual Test Steps that must be run and pass or fail each as you go through your Test Steps. If you fail a given Test Step, you will be able to add your comments as to why a given step failed.
You can even record the screen, capture screenshots, or do a video showing the failure.
This information will be attached to the Test Run. Remember that – it will be important to know in the future.
By allowing your team to provide meaningful contributions, you will be able to easily identify and address the issues with an application. You will be able to see exactly how the functionality is different from the expected results.
Once a tester has completed a Test Case, regardless of whether it passed or failed, they are then ready to go to the next Test Case in their Test Run. To do this they simply click “next >” shown below.
Once a tester has completed their Test suite they can simply Save and Close their Test Run and you will be able to see their outcomes in the “Runs” tab of Azure DevOps Test Plans.
Seeing the outcome of a Test Plan
Let’s say you’ve ran the following Test Plan (Requirements-based Test Suite 307) from our example above with the following results:
You can choose to use this same screen to identify what passed and what failed, but if you want to see the contributions of your testers you will be using the Runs view on found under Test Plans (see left hand side of above picture).
Here you can see the Test Run that “Needs Investigation”.
Click on the Test Run you see in “Recent test runs” and you will see the following the Test Run page with three tabs. Those tabs are “Run summary”, “Test results”, and “Filter” and you can see them below.
One thing to note about Test Runs, is that this is the place wherein you will find the attached screenshots, recordings, videos and/or comments that are added by your testers. As you will notice above, those items though are not found in the “Run summary” tab.
To see the contributions of your testers, you will need to navigate to the “Test results” tab shown below.
From here you will select your failed Test Case, and you will be given all of the information that was collected by a given tester for that Test Case during their Test Run. You can see that information below:
That’s how you would perform your basic manual testing in Azure DevOps Test Plans and see the contributions of your testers within your Azure DevOps project.
Now if you want to use the feedback that has been given by your testers, you can create bugs by going to the previous screen showing a failed Test Case and “Create a Bug”. Doing so will allow you to build a bug that is has a relationship called “Test Result” (shown below) that links directly to the Test Run failed Test Case.
Documenting your Test Plan
We’ve laid the foundation for manual testing above: You build your Test Plan, bring in your Test Suites, then run your Test Suites and view the results in the Runs tab of Azure DevOps Test Plans.
This will give you all of the necessary control over doing manual testing and enacting bugs and changes through the results of your Test Cases. But what if you need to document all of this?
The reason we cover an example of building a Test Plan, Test Suites, doing a Test Run, and showing the results is because there’s more to the picture here when it comes to documenting.
Azure DevOps doesn’t have a great reputation when it comes to taking the work items you create within your project and allowing you to turn those into documents. That’s why in the rest of this article we are going to cover moving all of this content you’ve seen into a document format using our Smart Docs module.
The first thing to point out is that much of the deeper-dive information regarding a given failure of an individual Test Suite, or the Test Plan as a whole is found in the Azure DevOps project itself. But, Azure DevOps will help you create a document for your Test Plan that will include all of the links to the necessary places (like the Test Run showing the details of a failure) inside your project.
Below is how you export a given Test Suite’s results in document format:
When you export this document you can choose “Print” to create a PDF that looks similar to the screenshot below:
Note all of the different links you can see in the above screenshot.
What’s important about these links is that they will help you step through to the full details of a given outcome.
For example, you can see where Test case 1578 (found near the bottom of the screenshot above) has the Latest Test Outcome as a link titled “Failed”. This link will take you the Test Runs page where that Test Case failed, and you can see the contributions of your testers there.
But maybe this document provided by Azure DevOps is not enough.
Maybe you would like to create the common version of a Test Plan document that contains all of the context surrounding a given Test Plan.
This can be done using Modern Requirements4DevOps.
If you need to build a document that outlines the Scope, Risks, Purpose, Strategy, and more… you might think to use Microsoft Word. But the Modern Requirements4DevOps Smart Docs feature will allow you to build up all of the context surrounding an individual Test Plan, while also allowing you to tie in your Test Results too.
The below video will show you how.
Building Test Documents from Azure DevOps
In this video we cover how you can build the documentation surrounding your Test Plans from within your Azure DevOps project using Modern Requirements4DevOps Smart Docs module.