Home
This repository contains a solution for enabling postman tests as first class citizens in Azure DevOps pipelines, meaning that each postman test case is represented by an Azure DevOps (ADO) work item of type "test case", and can thus participate in Test Plans.
There are in general two ways of running test cases in ADO:
-
In a build pipeline. Test source code is checked into the repository. It might need to be compiled, depending upon the test framework. A build task in the pipeline can compile the test source code. Another build task in the pipeline can execute the tests, and the test results can be viewed as part of the build pipeline result. This is convenient for e.g. unit tests, for which you typically are not interested in setting up test plans etc.
-
In a release pipeline. In order to use a release pipeline for testing, you setup a Test Plan (TP) in ADO, and add test cases to the TP. Then you can run automated tests in a TP by specifying which release pipeline to use. In the release pipeline setup, you specify which build artefacts to use (like test code/binaries and related dependencies, test execution prerequisites, etc). An automated test must contain metadata about the location and type of the test.
MSTest is a Microsoft based unit test type framework, and is equivalent to many other unit test framework, like nunit, junit, Google test, etc. The source code for a test class typically looks like
When compiling a MSTest project in Visual Studio (VS), you can execute test cases in the Test Explorer window. If you run use these automated test cases in a Test Plan, you would need to do the following:
- Manually create a workitem of type Test Case in ADO.
- From Test Explorer in VS, map the coded test case to the work item Test Case in ADO. Doing this will fill out the "Associated Automation" tab in the work item Test Case you mapped to, enabling ADO to know where to find that automated case, when you want to run it.
First of all, it is relevant to understand how Postman test cases are structured, and how you save Postman test cases into ADO. In Postman, a collection has a layout like:
So, a collection can contain folders, which in turn can contain requests. This is very similar to the layout of MSTest described above. Since this implementation use MSTest as a wrapper to execute Postman tests, the following, natural mapping has been chosen:
MSTest TestClass <-> Postman Collection
MSTest TestMethod <-> Postman Folder
Of course, each collection needs to be checked into the ADO repository. This is done by exporting the collection file from Postman into a file, and checking this file into the ADO repository. Collection files (and other relevant Postman files) need to follow the following naming convention:
Collection files may be saved in different directories in the repository.
Postman files of type: environment, globals and data files files belonging to a specific collection file must be saved in same directory as the corresponding collection file.
This guide assumes the source code is checked into an ADO repository, following the same path structure as shown below
There are three projects in the solution:
The Common project contains various helper classes/methods.
The Wrapper project handles code-generation Postman Collection/folder wrappers, as well as the exeucution of these tests.
The WorkItemGenerator project ensures creating and updating mapping of wrapped test cases to ADO work item test cases.
When the Wrapper project is built in a pipeline (or locally), a code generation using T4 templating will be carried out before the actual project is being compiled. This code generation will perform the folling sequence:
-
Search for all collection files in the Git repository fulfilling the standard Postman naming convention as described earlier.
-
For each collection file, a class with the MSTest [TestClass] attribute on it will be generated, and for each folder in the collection file, a method with MSTest [TestMethod] attribute will likewise be generated. The actual code content of this method is a one line call to a handler for executing the actual Postman test case.
After the code generation step, the project is compiled. After a successful project compilation, the project output binaries now contains up-to-date MSTest test cases representing all Postman test cases in the repository.
Whem the project WorkItemGenerator is built in the pipeline (or locally), a postbuild step executes the mapping engine, which ensures mapping between wrapped MSTest test cases and ADO work items. The mapping is created or updated as follows:
Case 1
Given: A MSTest case represents a Postman Folder "MyFolder1" in CollectionFile "MyCollection1". The Postman Folder Description does not contain a element.
Then: ADO is checked to see if an existing work item named "MyCollection1 - MyFolder1" of type "Test Case" exists already. If this work item exists, relevant automation fields are updated on this work item if needed.
Notice if you subsequently change the name of the ADO Test Case, then if cannot be found when building the project next time, and then a new ADO Test Case will be created.
Case 2
Given: A MSTest case represents a Postman Folder "MyFolder2" in CollectionFile "MyCollection2". The Postman Folder Description contains a element, with one of four following layouts:
In the first case, the Postman Folder is mapped to ADO Test Case with id 1001, and relevant automation fields are updated on this work item if needed.
In the second case, the Postman Folder is mapped to three ADO Test Cases with id 1001, 1002 and 1003, and relevant automation fields are updated on these ADO Test Cases if needed. This covers the scenario where one Postman Folder represents multiple ADO test cases. If an ADO Test Case with the specified id cannot be located, the build of this project will fail.
In the third and fourth case, a data file of type csv is expected to be present. The purpose of the attribute "dataline" is to map specific rows in the data file to specific ADO test cases. In the third case, the Postman folder is linked to ADO Test Case with id 1001, and each row in the data file containing the value of the attribute "dataline" in the first column, will be executed. In the fourth case, the Postman folder is linked to two different ADO Test Cases, and the attribute value again determines which rows in the corresponding csv file are used for execution. Notice the use of the separator ";" to add multiple values in the attribute "dataline".
Notice that the element must be valid XML data, otherwise the test build will fail.
Notice that text before and after the element can be inserted - it is ignored when the WorkItemGenerator project is parsing the the Postman Folder Description for a element.
The file AzureDevOps.xml in the project needs to be updated with relevant ADO setup information:
The "Url" and "Project" values are necessary to create the connection for the WorkItemGenerator project to connect to ADO.
The "AreaPath" value is used to specify the Area Path field value in the ADO Test Case.
Notice it is also possible so specify custom fields with belonging default values in this xml file. If specified, these custom fields will be given the values when creating or updating the ADO Test Case. This is needed in case of customized ADO Test Cases with mandatory fields. If these are not set when saving the ADO Test Case, the save will fail.
TODO
TODO mshosted agent, use the win2019 image enable oath token availability
TODO
TODO
TODO Test case independency