Glossary – Verification and validation (V&V)
Verification checks a product against its requirements; validation checks it...
Test Hub Reporting and Trace Analysis with Modern Requirements4DevOps
Modern Requirements4DevOps includes the ability to generate test hub information reporting and trace analysis. Users can generate Smart Reports and Horizontal Traceability Matrix on Test Plans, Test Suite, Test Cases, Test Runs, Test Case Run, and Test Case Step Run details. Since much of the information in Test Hub is not based on work items and links, Modern Requirements introduced the concept of virtual work items and links to abstract the data into work items and links on which reports can be generated. This is a quickly evolving set of functions with new data being exposed from Azure DevOps as virtual Work Items and Links so that users can generate Smart Reports & Traceability Reports. This reduces the testing effort, improves resource utilization, speed, and quality of the tests.
Introduction
In the simplest form, the test case management software consolidates several aspects of testing into one place. This
reduces the testing effort, improves resource utilization, speed, and quality of the tests.
Most often, the tools differ in their scope, ability, effectiveness, flexibility, and more. While some organizations settle on one comprehensive solution, others opt to use a combination of different tools.
The normal testing hierarchy supported by Azure DevOps is as following:
Whereas organizations normally have a more comprehensive Test Case Management as illustrated below:
The Requirement work item will be available when Requirement suite is created from the Test Hub.
Note: The Requirement work item is supported in Trace Analysis
What does Modern Requirements4DevOps have to offer?
In order to fill the gap between what test strategy companies usually follow and what Azure DevOps has to offer,
Modern Requirements4DevOps now offers Virtual work items.
Modern Requirements4DevOps now offers Virtual work items.
In Azure DevOps, the data for these virtual work items can be fetched from TFS using the following tabs:
Test Run
Test Result
Test Step Run
These Virtual work items are linked with existing work items using Virtual Link.
Virtual Work Items Properties
Support for Attachment and Comments Field
Attachments and Comments are now supported in Trace Analysis and Smart Report module (Trace Analysis, Test Plan Hub, Queries tab and Smart Doc)- Comment added on run test case is mapped to “Detail Comments”
- Attachment added on run test case is mapped to “Result Attachment”
Comment and attachment added from test run are mapped in test results virtual work item.
Smart Report Configuration
Output
- Comments added from update analysis will be mapped to “Comment” field in test
- Comment added from update analysis is added to the “Comment” field in Test Results.
- Comments added on fail steps on run test case are mapped to the “Comment” field in Test Step Run.
Comment for fail step added in “Comment” field of Test Step Run in Trace Analysis and Smart Report.
- Configuration is mapped to “Config Name” field in Test Results
Configuration can be seen mapped below in test results.
- Captured screenshot from Test Run are mapped as an attachment in Test Step Run virtual work
User can take screenshot from the Capture screenshot option as shown in the image below.
The captured screenshot can be seen in ADO mapped as an attachment. Users can download the captured screenshot by clicking on it.
The captured screenshot can be seen as an attachment in the Smart Report. Users can download the attachment by clicking on it.
The options Capture user actions, Record Screen, and Record screen with audio are supported in the Test Result virtual work item. Once a user adds these attachments from ADO, they will be visible in the Smart Report/Trace Analysis section of the Test Result work item under the “Result attachment” property.
For multiple iterations, Comments and Attachments added in each iteration is shown. On run test case, if there are multiple iterations, user can add comments in each iteration which can be seen by adding “Details” field from column options in Trace analysis and through “Report Designer” in Smart Report.
ADO
Comments and attachments added against each iteration can been seen in the “Summary” section.
Trace Analysis
Comments and Attachments added by the user against each of the iterations will be visible in the Traceability Matrix.
Smart Report
By adding “Details” field from the column options, comments and attachments for each iteration can be seen in “Report Designer” in Smart Report.
Test Parameter
Parameter added in Test case steps can be seen by adding “Test Parameter” field from column options in Trace analysis and through “Report Designer” in Smart Report.
ADO
Trace Analysis
Comments and Attachments added by the user against each of the iterations will be visible in the Traceability Matrix.
Smart Report
By selecting the “Test Parameter” field from column options, user can see “Parameters” column in Smart Report.
Linking of Virtual Work Items
In Smart Report, the virtual linking is automatically applied. However, in Trace Analysis you need to manually select the Virtual Link option to link the Virtual work items as shown in the image below:
Hierarchy
The hierarchy below can be created and shown in Smart Report as well as while working in Traceability module:
Test Plan vs Test Suite vs Test Case/Test Point vs Test Run vs Test Result vs Test Step Run vs Bugs
Same hierarchy can be created while showing the work in Trace Analysis.
Paths to Generates Output
Following are the different paths you can define to generate the output:
- Test plan vs Test suite
- Test suite vs Test case
- Test plan vs Test suite vs Test case
- Test run
- Test run vs Test result
- Test run vs Test result vs Test step run
- Test run vs Test result v––s Bugs
- Test run vs Test result vs Test step run vs Bugs
- Test run vs Test result vs Bugs vs Test step run
- Test suite vs Test case vs Test run vs Test result vs Test step run
- Test plan vs Test suite vs Test run vs Test result vs Bugs
- Test suite vs Test case vs Test run vs Test result vs Test step run vs Bugs
- Test plan vs Test suite vs Test run vs Test result vs Test step run vs Bugs
- Test suite vs Test case vs Test run vs Test result vs Bugs vs Test step run
- Test plan vs Test suite vs Test run vs Test result vs Bugs vs Test step run
- Test plan vs Test suite vs Test case vs Test run vs Test result vs Test step run vs Bugs
- Test Plan vs Test Suite vs Test Point vs Test Run vs Test Result vs Test Step Run vs Bugs
- Test Plan vs Test Suite vs Test Case vs Test Point vs Test Run vs Test Result vs Test Step Run vs Bugs
- Test Plan vs Test Suite vs Test Point vs Test Case vs Test Run vs Test Result vs Test Step Run vs Bugs
- Test Plan vs Test Suite vs Test Point
- Test Plan vs Requirement vs Test Case vs Test Run vs Test Result vs Test Step Run Vs Bugs
Usability of the Virtual Work Items
These work items can be used in the following modules/sections:
Smart Report
Virtual work items can be used in Smart Report available under the Queries tab, Test Hub and Smart Docs.
Trace Analysis
In the Traceability module, these virtual items can be used to create a horizontal matrix using:
- Work item type
- Query type
Using Virtual Work Items to Create a Horizontal Matrix Using Query Type
Using Virtual Work Items to Create a Horizontal Matrix Using Work Item Type
Creating Test Plan / Test Suite from Test Hub in Azure DevOps
Limitations
Smart Report | Queries Tab
- The linking of virtual work items is supported only when the report is generated from the
- In Smart Report (Queries Tab), all the sections should be hierarchical except for the ones that do not have any subordinates and, for these, the hierarchy would be
- If you have work items other than Test Plan, Test Suite and Test Case, you cannot generate a test case If other work items are also a part of the query then the smart report will be generated on its normal functionality.
Smart Report | Smart Docs Module
- While using Smart Report in Smart Docs, the default section should have work item type “Any” or root work item
- The Report will only be generated on adding Test Plan / Test Siute / Test Case work items from Insert Work Item These work items are to be created from Test Plan Hub.
Table of Contents
Recent Articles
Glossary – Use Case
A use case describes how a user interacts with a...
Glossary – Traceability in Azure DevOps
Traceability in Azure DevOps links work items to code and...
