Use this tag create “Top Blogs” on the main blog page.
Microsoft Azure customers worldwide now gain access to Modern Requirements4DevOps to take advantage of the scalability, reliability, and agility of Azure to drive application development and shape business strategies.
Continue reading
Good requirements management is essential for successful product and software development.
Continue reading
Today’s rapid technology sector growth puts development and IT operation teams under constant pressure to meet increased customer expectations for different applications.
Continue reading
Building your documents for your Azure DevOps Test Plans as a Business Analyst or Test Lead can be a daunting task.
The same thing is true for requirements that live within your Azure DevOps project, but Azure Test Plans poses a unique challenge.
In this article you get a brief overview of what the Azure DevOps Test Plans module is how it works, and how you can document your test outcomes easily. You will also learn about the architecture of Azure DevOps itself.
For a quick overview, you can skip down to the video
Azure Test Plans is a test management module within Azure DevOps that lets users manage test plans, test suites, and test cases for everyone in the software development process. Using test plans, you can Azure Test Plans also provides a browser extension for exploratory testing and gathering feedback from stakeholders.
Azure DevOps (ADO) is a great place for your Quality Assurance team to build out their Test Plans for a given project. By bringing your Testing Team onto the Azure DevOps platform, you can effectively use your ADO project as your single source of truth.
You can have your developers using the repos and tying development work to requirements, you can have your business analysis teams building requirements and attaching test cases, and you can have your QA team using those Test Cases, and requirements made by the BA team, to run all of your Testing.
Now Azure DevOps Test Plans can be used for both automated and manual testing. But this article only focuses on manual testing on ADO test plans.
Test Plans is an incredible place for your team to do their manual testing needs for many reasons:
Stakeholder Feedback
The biggest advantage of Test Plans is letting your testing teamwork with your Business Analysis and development teams in the same environment. By doing this, you create a single source of truth for everyone working on a project.
By deploying your manual testing operations from here, you can directly interact with the Test Cases that were already built inside of your Azure DevOps project. So the contributions of your Business Analysis team when building test cases will easily be pulled into your testing workflow. This capability adds a great benefit for teams who have BAs that are capable of building Test Cases for the QA team – if you’re a BA who does this, you will see how it works later in the article.
Robust Manual Testing Features
Test Plans also offers some incredibly helpful manual testing features to help your testers provide meaningful input on the outcome or results of their testsbeyond simply marking a specific test step as passed or failed. It also provides an interactive window for both marking the outcome of individual test steps, capturing live screenshots/recordings, and providing comments tied directly to the run of that test case.
So, when you review any Test Runs that you have completed, you can identify what your tester was experiencing when they marked an individual test step as blocked or failed. This level of insight means that even an external manual testing team can contribute thoughtfully and allow teams to identify and interpret failed results easily.
Easy Extensibility
With the right test case management tools, you can generate Smart Reports and Horizontal Traceability Matrices on the details of Test Plans, Test Suites, Test Cases, Test Runs, Test Case Runs, and Test Case Step Runs.
Tools like this can change the functionality of Test Plans to reflect how companies manage test cases through virtual linking.
Every month, Google gets thousands of searches asking how ADO testing works. One reason may be that most teams use proprietary, in-house solutions that are built for testing that are separate from ADO Test Plans.
Before learning how ADO Test Plans works, it’s worth knowing the relevant terminology. Azure Test Plans gives you three main types of test management artifacts:
Sometimes the terminology can be confusing, as many teams say “test plan” when they mean “test suite.”
If your team creates a Test Suite, they must choose between three different kinds of test suites, as shown below:
Test plans contain test suites.
Test suites can be of three types. Read each of the following suites to see the three different types.
Test Suite 1 – Req-based
Requirements-based test suites are the simplest and most traceable. They pull in all of the Test Cases for a given Requirement.
Test Suite 2 – Query Based
Query-based test suites pull in a group of tests from your project, irrespective of what requirements the test cases are linked to.
Test Suite 3 – Static Based
Static-based test suites are used either as containers to group other test suites, or to group a specific set of test cases.
epending on which Test Suite you create, building a Test Suite allows you to pull in all the Test Cases that you want to test.
Once you create your Test Suite with their respective Test Cases, you can execute the entire Suite, which is known as a Test Run.
Before beginning your Test Run, you must choose between three different kinds of Test Suites to execute: Requirements-based, Query-based, and Static.
Before starting your Test Run, you must choose between three different kinds of Test Suites to execute: Requirements-based, Query-based, and Static. Remember: Test Suites (of any type) can only include Test Cases.
1) Requirements-based Test Suite
A Requirements-based Test Suite is one in which you associate your Test Cases to a requirement to define its acceptance criteria.
Creating a Requirements-based Test Suite will pull in all the Test Cases for that requirement. When building a suite, you automatically import all its associated Test Cases. You will instead simply choose the Requirement.
You can then run all the Tests for that given requirement and see from each Test Run and conclude if the product you’re developing fully meets that requirement.
As creating a suite automatically pulls in all the associated tests, your QA team doesn’t need to redo the work of populating all requirements. If your BAs and developers added the Test Cases for a requirement when they created it, the QAs can use it. This is the benefit of a single source of truth model. Your teams can benefit from each other’s work.
Even if your Test Lead decides to create more Test Cases for that Requirements-based Suite, the Test Cases they create will be automatically linked to the requirement itself. So regardless of whether BAs, devs, or Test Leads are adding Test Cases, every team can contribute to the same group of Work Items.
If you use Requirements-based Test Suites, your team can create Traceability Matrices that show youhow your Test Cases align with your requirements and whether a Test Case has passed or failed for any given requirement.
With more advanced solutions, you can even export and share explanatory reports of your tests.
2) Query-based Test Suite
When you make a Work Item query and select which test cases to include into your suite, you create a Query-based Suite. Azure DevOps automatically adds all test cases that meet these criteria to the suite.
When do you use a Query-based Test Suite?
In some cases, teams run tests on all the Test Cases that are currently in each iteration, or run tests on everything with a given tag, or some other criteria. These conditions are not necessarily tied directly to a given requirement and you can only satisfy them through a specific query.
Unlike Requirements-based Suites, which pull in all tests associated with a requirement, Query-based Suites give teams the ability to create a Test Suite using any criteria available.
If you are creating this type of Test Suite, it asks you how you want to pull in Test Cases and allows you to construct an ad-hoc query to do so. It allows you to pull in the Test Cases that match specific criteria. It is important to note that this does not use the queries your Business Analysis team would create in Azure DevOps. But the interface for constructing this ad-hoc query is nearly identical.
3) Static Test Suite
Static suites are logical containers where you can add any tests you like. They aren’t connected to any requirements or queries.
They are useful when you may need to create multiple levels of nested Test Suites. Or you may need a place to create and experiment with ad-hoc tests.
Test Plans help your team to structure how tests are run so that your project fulfills the quality standards. You will create both Test Plans and Test Suites for all the requirements for every iteration below.
The steps are as follows:
Step 1:
To build a Test Plan called “Given Iteration Test Plan,” go to the Test Plan tab on the left panel of your Azure DevOps interface. Select “+ New Test Plan.”
Name this Test Plan “Test Plan for Iteration 1.”
Select if you would like this specific Test Plan to be for a given area path or iteration. Otherwise, you can leave this blank (in these example images those two fields are left blank as the default options.)
With the test plans created, it’s now time to create a suite.
Step 2: Add Test Suites
Since the aim is to test all the requirements from a given iteration of a product are tested, you will add Requirements-based Suites.
When creating a Requirements-based Suite, you will pull in Test Cases that already exist for the given requirements. Otherwise, you will have to add new Test Cases to a requirement before adding it to a Requirements-based Test Suite.
We don’t want to bring in requirements, we want to bring in the Test cases associated to requirements. This point is just a reminder that Even when you build a Requirement-based Suite, you are creating a group of Test Cases, not requirements.
So, let’s create Test Suites (groups of Test Cases) for each of the requirements in iteration 1.
Step 3: Run Query
Above you can see the query you would need to build to pull in the Iteration 1 requirements, then you can simply click “Run Query” as shown above.
When you run the query and see your Iteration 1 Work Items, select a few.
Since each Work Item above is a functional requirement and you have chosen to create Requirements-based Suites, you will create four Test Suites, one for each requirement.
Let’s look at the outcome of clicking the blue “Create Suites” button above.
In the image above, you see that Test Suites for Work Items 306 and 307 have numbers in parentheses next to them. They represent the number of Test Cases they contain, one and three respectively.
Now that you have a Test Suite for all the necessary requirements, you can either run the tests or add in Test Cases to fill out the Test Suites for those requirements.
The action of “running the Test Case(s)” on a Test Suite to see if they pass or fail is known as a Test Run. A Test Run is the outcome of running one or multiple Test Cases.
Let’s do a Test Run of one of our Test Suites.
Right click on the Test Cases within Work Item 307 and click “Run for web application.” With the Chrome browser extension, you can run these Tests within Chrome itself.
Azure will launch a new window for your Test Run (see below).
If you drag this window beside the window containing the web application you are currently testing, you can see the individual Test Steps and pass or fail them as you test them. If the application or function fails one of the Test Steps, you can add your comments on why it failed.
You can even record the screen or take screenshots to better illustrate the failure. This information is now attached to the Test Run.
By testing and commenting on how the application or product you are testing works, you allow your team to provide meaningful contributions to fix the issues you spotted.
Once you have completed a Test Case (whether passed or failed) you’re ready to go to the next Test Case in their Test Run. To do this, click “next” as shown below.
When you complete your Test Suite, you can Save and Close your Test Run. You will be able to see their outcomes in the “Runs” tab of Azure DevOps Test Plans.
Let’s say you’ve ran the following Test Plan (Requirements-based Test Suite 307) from our example above with the following results:
You can choose to use this same screen to identify what passed and what failed, but if you want to see the contributions of your testers, you can use the Runs view under Test Plans (see left side of the picture above).
Here you can see the Test Run that “Needs Investigation.”
Clicking on the Test Run, you see in “Recent test runs” and you will see the following Test Run page with three tabs. Those tabs are “Run summary,” “Test results,” and “Filter” and you can see them below.
Test Runs is where you find the attached screenshots, recordings, videos, and/or comments added by your testers. They aren’t available in the “Run summary” tab.
To see the contributions of your testers, navigate to the “Test results” tab, as shown below.
Select your failed Test Case to access the information collected by a given tester during their Test Run for that Test Case.
To use the feedback given by your testers, you can create bugs by going to the previous screen, which shows a failed Test Case and lets you “Create a Bug.” Doing so will allow you to build a bug that is has a relationship called “Test Result” (shown below) that links directly to the Test Run failed Test Case.
Most large companies doing technically intensive work also need to document the work employees are doing.
You can find a lot of deeper-dive information regarding a given failure of an individual Test Suite, or the Test Plan in the Azure DevOps project itself. Azure DevOps will help you create a document for your Test Plan that will include all the links to the necessary places (like the Test Run showing the details of a failure) inside your project.
How you export a given Test Suite’s results in document format is shown below.
When you export this document, you can choose “Print” to create a PDF that looks like the screenshot below:
The links shown above help you step through to the full details of a given outcome.
For example, Test Case 1578 (found near the bottom of the screenshot above) has the “Latest Test Outcome” value of “Failed” with a link. This link will take you to the Test Runs page where you can see the contributions of your testers.
But this documentation is not enough. Many users like to create the common version of a Test Plan document that contains all the context surrounding a given Test Plan. If you need to build a document that outlines the scope, risks, purpose, strategy, etc., you might want to use Microsoft Word.
Documentation isn’t known as one of Azure DevOps’ greatest strengths. But with a solution like SmartDocs, you can turn work items, Test Cases, Test Suites, Test Plans, and a lot of associated information into readable and shareable documentation.
The below video will show you how.
In this video we cover how you can build the documentation surrounding your Test Plans from within your Azure DevOps project using Modern Requirements4DevOps Smart Docs module.
Your organization’s NFRs can impact time overruns, user satisfaction, compliance, and many other issues. But knowing how to make an NFR is difficult.
It is hard because building non-functional requirements can be a major point of contention for different teams with different backgrounds and methodologies.
Even then, <customers> can struggle to manage the demands or multiple teams, third-party software limitations, organisational quirks, and other business priorities that can get in the way.
For that reason, this article covers what NFRs are and how to build them, with examples and templates to guide you along.
If you open the texting app on your phone, you will notice a few functions. They include texting, emojis, editing contacts, and others. They define what the app does. But these functions work because of several requirements that define how they will work. For example, texting only works if your phone is compatible with all major carriers.
These aspects that define how a system will work by laying out its requirements and limitations are called Non-functional Requirements (NFRs).
They are contrasted with functional requirements, which set the primary function of a system. If the functional requirement of your car is to seat four people in comfort, a non-functional requirement is the size of each headrest.
As time goes on, teams, companies, and stakeholders will notice that their list of non-functional requirements will grow and change. This is expected as the needs of an organization adapt and change.
You may need to add, remove, and change NFRs in response to organizational changes, implementation changes, or even changes in business needs.
Regardless of how small or extensive these changes may be, a team can easily identify how implementing a change might affect their success measures by re-evaluating their NFRs. Teams may notice that a small implementation change could lead to a completely new NFR that will help them identify a project’s success.
As teams continue to expand their list of non-functional requirements, you might be wondering how do they keep this expanding NFR list organized?
Teams keep their NFR’s organized by using categories with which they can evaluate overall success. A typical grouping of non-functional requirements would be:
Operational NFRs, like security, accessibility, and usability
Revisional NFRs, like flexibility and scalability
Transitional: NFRs, like portability, reusability, and interoperability
When setting up a project’s requirements, you must set the following conditions:
A best practice when setting NFRs is establishing measurement benchmarks. But what do your measurement benchmarks (e.g., headrest size) work towards? Ultimately, any requirement has to work towards fulfilling larger business objectives.
These objectives can be very specific, like hitting quarterly revenue targets. Sometimes they can also be less empirical, like building a brand. In both cases, your NFRs should work towards fulfilling your company’s business needs.
Unlike their functional counterparts, non-functional requirements cover an incredibly broad scope. In defining the overall qualities, a project, system, or process should exhibit, the list of non-functional requirements can grow very large
While these non-functional requirements might not “do anything specifically,” they do specify the attributes a system, process, or project must have on completion. You can then use these non-functional requirements to measure the overall success of a given project, process, or system, and provide measurable insights into how close to completion our project might be.
When building non-functional requirements, the first thing a team must consider is whether they are tackling the NFR’s that are relevant to the project. This makes the first step of the process simple.
Identify the project attribute for which you want to build success indicators. Here’s a more exhaustive
breakdown some of the Operational, Revisional, and Transitional aspects of a project.
Operational attributes:
Revisional attributes:
Transitional attributes:
Typically, business analysts, developers, and project stakeholders develop and refine NFRs.You can tune NFRs to be product oriented, addressing specific attributes such as performance (like processor speed on a phone) or the project’s costs.
Some NFRs are created to address criteria like product longevity and growth, which are based on attributes like the modifiability of a product. Other non-functional requirements could be process facing attributes that define aspects of a product such as its reusability.
Each of these non-functional requirements will fall into either the Operational, Revisional, or Transitional categories respectively.
The next step after deciding the actionable attribute is learning how to make an NFR.
You can also build a non-functional requirement using an operational attribute like reliability.
As always, operational attributes of a project are concerned directly with the operational functions of the project from both the system and the user’s perspective. Reliability here refers to a system’s ability to consistently perform to its specification within its intended environment.
As we are trying to establish an NFR for our system, understanding that this is an operational attribute helps us understand how we can make our NFR measurable to make it achievable.
This requires the application of both a measure and a metric where possible. Some common examples of a metric that would apply to reliability is mean time to failure, probability of failure, or a level of uptime per degree of time.
If we wanted to create a reliability non-functional requirement using our NFR template, we can ask the following questions with their respective answers
“Our measure is 100% uptime, our metric is uptime/first year.”
You can condense these answers into one statement:
“The system should be operational so that we see a 100% uptime for the first year of operation in order to meet our 1-year performance guarantee.”
Adding some more information about why this NFR has these criteria as above is option. But it can be helpful to link an NFR to an initiative in order give it more context.
Despite the template, it’s always possible to forget to add non-functional requirements. You should take appropriate caution in that case.
There are many instances where regulatory and compliance issues require the creation of NFRs.
For instance, in the highly regulated medical devices industry, confidentiality (an operational attribute) is paramount .
Specific requirements have been well defined by The International Organization for Standardization (ISO). These well-defined non-functional requirements must be strictly adhered to during the development of any medical device.
Let’s take a look at how these ISO-mandated requirements are categorized, and discuss some tools used to build compliant NFR’s quickly and easily.
ISO 13485 – Medical Devices –Quality Management Systems – are comprised of NFRs based on:
ISO 14971: 2007 – Medical Devices – Application of Risk Management to Medical Devices – are requirements related to:
Due to the broad and complex nature of NFRs, even creating examples of non-functional requirements is a difficult task. This can be especially true for novice business analysts (BAs) and even for veteran BAs who are gathering non-functional requirements in an unfamiliar space.
The elicitation of the correct NFRs is integral to asking the right questions. To do so, BAs must use social science derived data-gathering techniques like interviews, focus groups, surveys, brainstorming, and more
The next step isn’t much easier. Coming up with an example is an issue that many novice business analysts (BAs) face. Even veteran BAs are gathering non-functional requirements in an unfamiliar space can struggle.
But by using cutting-edge tools like Modern Requirements4DevOps, you can assist users with elicitation, authoring, and management of non-functional requirements.
Modern Requirements4DevOps’ FAQ module enables users with the ability to review focused question lists based on specific attributes of each of the four aspects of a project. These question lists provide the user with examples of questions that can be asked to elicit strong non-functional requirements based on each attribute.
The tool is preloaded with over 2500 questions found within 29 topics. By answering questions, users are building their non-functional requirements. Stakeholders can answer questions using existing backlog work items. Or users can create new backlog items right from within the module. They also have the freedom to make new questions and build their own custom question lists.
Try Modern Requirements4DevOps today and build non-functional requirements using pre-built question lists that address topics such as ISO compliance, and more.
Experience for yourself how our Modern Requirements toolbox can empower Microsoft’s industry leading Azure DevOps into a single application requirements management solution.
The Requirements Traceability Matrix (RTM) is a planning tool to help ensure that a project’s scope, requirements, and deliverables remain “as is” when compared to the baseline.
Continue reading
More than ever before, medical device manufacturers are hard-pressed to deliver products that enable faster, cost-effective and more efficient patient care.
Continue reading
If you work in an agile environment, you probably hear terms like “user story” and “requirements” being thrown around all of the time, but have you ever stopped to think about what they really mean?
Continue readingWant to see how ModernRequirements4DevOps works?
We’ll give you a quick Demo!
