Home
This repository contains a solution for enabling postman tests as first class citizens in Azure DevOps pipelines, meaning that each postman test case is represented by an Azure DevOps (ADO) work item of type "test case", and can thus participate in Test Plans.
MSTest is a Microsoft based unit test type framework, and is equivalent to many other unit test framework, like nunit, junit, Google test, etc. The source code for a test class typically looks like
When compiling a MSTest project in Visual Studio (VS), you can execute test cases in the Test Explorer window. If you run use these automated test cases in a Test Plan, you would need to do the following:
- Manually create a workitem of type Test Case in ADO.
- From Test Explorer in VS, map the coded test case to the work item Test Case in ADO. Doing this will fill out the "Associated Automation" tab in the work item Test Case you mapped to, enabling ADO to know where to find that automated case, when you want to run it.
First of all, it is relevant to understand how Postman test cases are structured, and how you save Postman test cases into ADO. In Postman, a collection has a layout like:
So, a collection can contain folders, which in turn can contain requests. This is very similar to the layout of MSTest described above. Since this implementation use MSTest as a wrapper to execute Postman tests, the following, natural mapping has been chosen:
MSTest TestClass <-> Postman Collection
MSTest TestMethod <-> Postman Folder
Of course, each collection needs to be checked into the ADO repository. This is done by exporting the collection file from Postman into a file, and checking this file into the ADO repository. Collection files (and other relevant Postman files) need to follow the following naming convention:
CollectionName must be unique.
Collection files may be saved in different directories in the repository.
Postman files of type: environment, globals and data files files belonging to a specific collection file must be saved in same directory as the corresponding collection file.
This guide assumes the source code is checked into an ADO repository, following the same path structure as shown below
There are three projects in the solution:
The Common project contains various helper classes/methods.
The Wrapper project handles code-generation Postman Collection/folder wrappers, as well as the exeucution of these tests.
The WorkItemGenerator project ensures creating and updating mapping of wrapped test cases to ADO work item test cases.
When the Wrapper project is built in a pipeline (or locally), a code generation using T4 templating will be carried out before the actual project is being compiled. This code generation will perform the folling sequence:
- Search for all collection files in the Git repository fulfilling the standard Postman naming convention as described earlier.
- For each collection file, a class with the MSTest [TestClass] attribute on it will be generated, and for each folder in the collection file, a method with MSTest [TestMethod] attribute will likewise be generated. The actual code content of this method is a one line call to a handler for executing the actual Postman test case.
After the code generation step, the project is compiled. After a successful project compilation, the project output binaries now contains up-to-date MSTest test cases representing all Postman test cases in the repository.
Whem the project WorkItemGenerator is built in the pipeline (or locally), a postbuild step executes the mapping engine, which ensures mapping between wrapped MSTest test cases and ADO work items. The mapping is created or updated as follows:
Case 1
Given: A MSTest case represents a Postman Folder "MyFolder1" in CollectionFile "MyCollection1". The Postman Folder Description does not contain a element.
Then: ADO is checked to see if an existing work item named "MyCollection1 - MyFolder1" of type "Test Case" exists already. If this work item exists, relevant automation fields are updated on this work item if needed.
Notice if you subsequently change the name of the ADO Test Case, then if cannot be found when building the project next time, and then a new ADO Test Case will be created.
Case 2
Given: A MSTest case represents a Postman Folder "MyFolder2" in CollectionFile "MyCollection2". The Postman Folder Description contains a element, with one of four following layouts:
In the first case, the Postman Folder is mapped to ADO Test Case with id 1001, and relevant automation fields are updated on this work item if needed.
In the second case, the Postman Folder is mapped to three ADO Test Cases with id 1001, 1002 and 1003, and relevant automation fields are updated on these ADO Test Cases if needed. This covers the scenario where one Postman Folder represents multiple ADO test cases. If an ADO Test Case with the specified id cannot be located, the build of this project will fail.
In the third and fourth case, a data file of type csv is expected to be present. The purpose of the attribute "dataline" is to map specific rows in the data file to specific ADO test cases. In the third case, the Postman folder is linked to ADO Test Case with id 1001, and each row in the data file containing the value of the attribute "dataline" in the first column, will be executed. In the fourth case, the Postman folder is linked to two different ADO Test Cases, and the attribute value again determines which rows in the corresponding csv file are used for execution. Notice the use of the separator ";" to add multiple values in the attribute "dataline".
Notice that the element must be valid XML data, otherwise the test build will fail.
Notice that text before and after the element can be inserted - it is ignored when the WorkItemGenerator project is parsing the the Postman Folder Description for a element.
The file AzureDevOps.xml in the project needs to be updated with relevant ADO setup information:
The "Url" and "Project" values are necessary to create the connection for the WorkItemGenerator project to connect to ADO.
The "AreaPath" value is used to specify the Area Path field value in the ADO Test Case.
Notice it is also possible so specify custom fields with belonging default values in this xml file. If specified, these custom fields will be given the values when creating or updating the ADO Test Case. This is needed in case of customized ADO Test Cases with mandatory fields. If these are not set when saving the ADO Test Case, the save will fail.
There are in general two ways of running test cases in ADO:
- In a build pipeline. Test source code is checked into the repository. It might need to be compiled, depending upon the test framework. A build task in the pipeline can compile the test source code. Another build task in the pipeline can execute the tests, and the test results can be viewed as part of the build pipeline result. This is convenient for e.g. unit tests, for which you typically are not interested in setting up test plans etc.
- In a release pipeline. In order to use a release pipeline for testing, you setup a Test Plan (TP) in ADO, and add test cases to the TP. Then you can run automated tests in a TP by specifying which release pipeline to use. In the release pipeline setup, you specify which build artefacts to use (like test code/binaries and related dependencies, test execution prerequisites, etc). An automated test must contain metadata about the location and type of the test.
- In the ADO project, navigate to Pipelines/Pipelines, and select "New pipeline".
- Select "Use the classsic editor to create a pileline without YAML".
- Select a source for your code.
- Select "Empty job" when selecting a template.
- Specify Pipeline properties, using "Agent Pool" = "Azure pipelines", "Agent specification" = "Windows-2019", "Allow scripts to access the OAuth token" = Yes.
Notice the Agent specification will not work with "vs2017-win2016" due to an older version of Msbuild on that image, which does not support the T4 template feature being used in the project.
6 Add build tasks as in below picture.
Specify properties for the MsBuild build task as in below picture:
Furthermore, tick the "Restore NuGet Packages" box under Advanced section. This ensures Nuget packages required in the solution are downloaded before build. This works, but is a depracated feature. A better option os to add a separate task before the build (called "NuGet Tool Installer), to retrieve the NuGet packages.
Specify properties for the first Publish Build Artifacts build task as in below picture:
The property "Path to publish" must of course correspond to the relative path in repository containing the project "Common" of the solution.
Specify properties for the second Publish Build Artifacts build task as in below picture:
Specify properties for the third Publish Build Artifacts build task as in below picture:
The property "Path to publish" must be the relative path of the root folder containing all Postman collection files (and associated Postman files).
The reason for using three Publish Build Artifacts, instead of a single Publish Build Artifacts pointing at the Postman foot folder, is of course to reduce the size of the created build artifacts. If just a single Publish Build Artifacts was used, it would also publish all downloaded packages.
Notice, presently this solution only works with a Git repository (due to the implemented algorithm for discovering the root of the repository folder)
- In the ADO project, navigate to Pipelines/Releases, and select "New release pipeline".
- Select "Start with Empty Job".
- Select "Add an artifact".
The source type defaults to "Build", leave it at this default. Specify property "Source (build pipeline" to be the earlier created build pipeline. Select "Add"
- For stages, click on the link in the Stage 1 for adding stage tasks.
For the Agent job, the default settings will work without modification. Notice that the property "Artifact download" defaults to downloading all artifacts from the build pipeline, which means the build artifacts created in the build pipeline will be available for this release pipeline during execution - which is directly what is needed.
- Add three build tasks as shown in below picture:
- For the command line build task, set properties as per below picture:
- For the Visual Studio Test Platform Installer task, leave properties to default values.
- For the Visual Studio Test task, set property "Select test using" value to "Test Run".
This allows you to run specific test cases in a Test Plan.
There is also a value called "Test Plan". This is used triggering a release pipeline and running all automated test cases in a test plan. This is another usage, which can be exploited. The purpose of this wiki is however to execute selected test cases in a Test Plan, why the usage of this value is not investigated further in this wiki.
With the pipelines setup as described above, it is now possible to use the normal flow when working with ADO Test Plans and executing test cases, like:
- Create new Test Plan
- Add existing Test Cases to the Test Plan
- Execute Test Cases by:
a. Selecting the Test Cases to be executed. b. click on the 3-dots, and select Run - Run with options. c. Select "Automated tests using release stage". d. Select an already generated build from the 3-dot button. e. Select the release pipeline. f. Select the stage.
This is what it takes to kick off an automated run of selected test cases. It is possible to look at execution details on the release pipeline when the tests are executing, just as it is possible to look at detailed Test Case results after the run, in the Test Plan.
Test case independency
It is a good practice to write test cases in such a way that there are no dependency on the test case execution order. The reason for that is that there is no documentation on the execution order of automated test in Azure DevOps, and hence automated tests with an interdependency might thus fail randomly if not executed in an expected order.
Notice that Azure DevOps documentation does mention an Order field on test suites, but this only applies to manual test, not automated tests (see https://docs.microsoft.com/en-us/azure/devops/test/reference-qa?view=azure-devops#q-can-i-group-and-reorder-my-requirement-based-test-suites-together).