Skip to content

Home

Jump to bottom
jbjakobs edited this page Oct 14, 2020 · 28 revisions

Overview

This repository contains a solution for enabling postman tests as first class citizens in Azure DevOps pipelines, meaning that each postman test case is represented by an Azure DevOps (ADO) work item of type "test case", and can thus participate in Test Plans.

ADO pipeline overview

There are in general two ways of running test cases in ADO:

  1. In a build pipeline. Test source code is checked into the repository. It might need to be compiled, depending upon the test framework. A build task in the pipeline can compile the test source code. Another build task in the pipeline can execute the tests, and the test results can be viewed as part of the build pipeline result. This is convenient for e.g. unit tests, for which you typically are not interested in setting up test plans etc.

  2. In a release pipeline. In order to use a release pipeline for testing, you setup a Test Plan (TP) in ADO, and add test cases to the TP. Then you can run automated tests in a TP by specifying which release pipeline to use. In the release pipeline setup, you specify which build artefacts to use (like test code/binaries and related dependencies, test execution prerequisites, etc). An automated test must contain metadata about the location and type of the test.

Test case automation in ADO

How does test cases based on MSTest fit into ADO?

MSTest is a Microsoft based unit test type framework, and is equivalent to many other unit test framework, like nunit, junit, Google test, etc. The source code for a test class typically looks like

When compiling a MSTest project in Visual Studio (VS), you can execute test cases in the Test Explorer window. If you run use these automated test cases in a Test Plan, you would need to do the following:

  1. Manually create a workitem of type Test Case in ADO.
  2. From Test Explorer in VS, map the coded test case to the work item Test Case in ADO. Doing this will fill out the "Associated Automation" tab in the work item Test Case you mapped to, enabling ADO to know where to find that automated case, when you want to run it.

How does test cases based on Postman fit into ADO?

First of all, it is relevant to understand how Postman test cases are structured, and how you save Postman test cases into ADO. In Postman, a collection has a layout like:

So, a collection can contain folders, which in turn can contain requests. This is very similar to the layout of MSTest described above. Since this implementation use MSTest as a wrapper to execute Postman tests, the following, natural mapping has been chosen:

MSTest TestClass <-> Postman Collection

MSTest TestMethod <-> Postman Folder

Of course, each collection needs to be checked into the ADO repository. This is done by exporting the collection file from Postman into a file, and checking this file into the ADO repository. Collection files (and other relevant Postman files) need to follow the following naming convention:

image

Collection files may be saved in different directories in the repository.

Postman files of type: environment, globals and data files files belonging to a specific collection file must be saved in same directory as the corresponding collection file.

Details of the implementation

Overview of source code content of solution

This guide assumes the source code is checked into an ADO repository, following the same path structure as shown below

image

There are three projects in the solution:

The Common project contains various helper classes/methods.

The Wrapper project handles code-generation Postman Collection/folder wrappers, as well as the exeucution of these tests.

The WorkItemGenerator project ensures creating and updating mapping of wrapped test cases to ADO work item test cases.

Generation of MSTest wrapper test methods

When the Wrapper project is built in a pipeline (or locally), a code generation using T4 templating will be carried out before the actual project is being compiled. This code generation will perform the folling sequence:

  1. Search for all collection files in the Git repository fulfilling the standard Postman naming convention as described earlier.

  2. For each collection file, a class with the MSTest [TestClass] attribute on it will be generated, and for each folder in the collection file, a method with MSTest [TestMethod] attribute will likewise be generated. The actual code content of this method is a one line call to a handler for executing the actual Postman test case.

After the code generation step, the project is compiled. After a successful project compilation, the project output binaries now contains up-to-date MSTest test cases representing all Postman test cases in the repository.

Mapping of Postman test cases to ADO work item Test Cases

TODO

Setting up the pipelines

TODO

Setting up a build pipeline

TODO mshosted agent, use the win2019 image enable oath token availability

Setting up a release pipeline

TODO

Running Postman test cases in Azure DevOps Test Plan

TODO

Final notes

TODO Test case independency

Clone this wiki locally