HW1 ‐ Advanced testing techniques

The goal of this homework is to try out advanced testing techniques:

1. combinatorial testing,
2. and integration testing.

Environment setup

Setup a VM with Ubuntu 22.04. If you do not want to install the OS yourself, there are ready-made VMs available to download on OSboxes.

This homework will require Java 17, PostgreSQL, Postman, and an IDE to edit the code. These can be installed via:

sudo apt update
sudo apt install openjdk-17-jdk
sudo apt install postgresql-14
sudo snap install intellij-idea-community --classic
sudo snap install postman
sudo apt install maven

You need to install and configure a PostgreSQL database with the following configuration:

 port: 5432
 username: postgres
 password: password
 database name: postgres

(In a production system you shall use more secure usernames, passwords and authorization. For this lab we can use these non-secure authentication information.)

Note: These predefined attributes can be found in application.properties, in case you need to modify them.

You can change the password by issuing the following command:

sudo -u postgres psql postgres

Then issue the following SQL query:

ALTER USER postgres PASSWORD 'password';

We will use OpenWeatherMap, an external weather API during the lab, for which you need a personal access token.

Sign up here: https://openweathermap.org/guide (the confirmation and activation of the registration can take 20 minutes, thus start with this step).

Combinatorial testing

We will try out combinatorial testing by creating a test input set satisfying n-wise coverage.

Tools

You can use one of the following tools:

• ACTS: Java program, has a GUI; free, but registration required,
• PICT: C++ program (can be built on Windows/Linux/Mac), only CLI; open source.

ACTS can be downloaded from the Moodle repository of this course.

Both tools have detailed user guides.

Exercises (1p)

The method to test is the following:

int calculateEngineInput( GearMode g, bool economyMode, int acceleration )

• the domain of GearMode is {Backward, None, Parking, Drive},
• acceleration can have values between 0-3.

We want to test the combinations of the parameters, but do not want to try all the 32 possible combinations. We first check only the combinations of any two parameters (2-wise or pairwise coverage).

1. How many tests are needed for pairwise coverage? Try to create such a test input set!

2. Create a new model in the tool, and define the above parameters.

3. Generate a pairwise test set! How many combinations did the tool select?

4. After refining the requirements of the system it turns out that not all combinations are possible: economyMode could not be true in case of Backward. Add this constraint to the tool, and generate a new test set! What changed?

   1. ACTS uses similar syntax for constraints (see user guide for details):

      (OS = "Windows") => (Browser = "IE" || Browser = "FireFox")

   2. PICT uses similar syntax for constraints (see user guide for details):

      IF [File system] = "FAT32" THEN [Size] <= 32000;

5. In a new version of the system a new parameter was introduced: BrakeStatus with values {Pressing, Releasing, None}. Modify the model and create a new test set!

6. Modify the configuration of the tool to create a test set with 3-wise coverage! How many combinations were selected?

CHECK Create a screenshot about the last test suite.

This simple example showed how combinatorial testing can be used in practice. The tools and techniques are even more useful when the system under test has many more parameters. Each tool comes with larger sample models, try to load them and generate test sets for them.

Integration testing

Technologies

We will use the following tools and technologies during the homework.

Postman

Postman makes API development faster and easier. Postman is a tool used to send requests and receive responses through a REST API, and you can use it to organize and save your API tests.

The Postman example above shows a simple HTTP GET request to a NASA API with two attributes.

WireMock

WireMock is a tool that can mimic the behaviour of an HTTP API and capture the HTTP requests sent to that API. We can use it

• when we have to implement a feature that uses an HTTP API that is not ready,
• when we write unit tests for classes which use HTTP APIs (external dependency) or
• when we have to write integration, API, or end-to-end tests for features which use external HTTP APIs. For more details, see Introduction to WireMock tutorial.

REST-assured

REST-assured is a library that gives you a DSL (domain-specific language) for firing real HTTP requests against an API and evaluating the responses you receive.

Example project

This homework is created from the article The Practical Test Pyramid written by Ham Vocke. If you don't understand something you can look there for more details later.

We changed and simplified the exercise for the current homework, thus some parts of the code differ.

Use this AVT version: CODE

The application can store data about people (first name, last name), and offers some REST APIs to query people data and the current weather (using an external weather API, OpenWeather).

Here are some useful diagrams showing how the system is structured. The project was built using the Spring framework.

The high level structure of the microservice system:

The internal structure of the microservice:

Take some time to get to know the application:

1. Check out the structure of the source code (main/java/example).
   • person/Person.java is the main entity stored by the application. Check its attributes. PersonRepository manages the persistence of instances (it is currently just an interface, implementation is generated by the Spring framework).
2. ExampleController.java defines the REST endpoints and implements the application logic. Check out how many endpoints have been defined (see @GetMapping).

Exercises

Build & deploy (1p)

1. Clone the repository to your machine.

2. Claim an OpenWeatherMap API key (the external weather API).

   • Get your private API key. (A key like this: a7f0fee465477b551444385b1*******, intentionally redacted)
   • IMPORTANT Modify the env.sample file:

   export WEATHER_API_KEY=a7f0fee465477b551444385b1*******
   

3. Open a terminal in the root of the project folder. Build, test and package your application .

   mvn package
   

4. Now you should deploy the application to your local machine from terminal.

   • Make sure you added your API key to the env.sample file.
   • Make sure, that the database is running, accessible on the right port. See help.
   • Run env.sample to define the API key as an environmental variable:

   source env.sample
   

   • Start and deploy the application:

   java -jar target/springtesting-0.1.0.jar
   

   • Once you see "Started ExampleApplication" in the log, you can use it.

5. Try out the REST services.

   • Open the following URL in a browser: http://localhost:8080/hello
   • Try to query a person. For this, we need to add a person to the person table in the PostgreSQL database.
     • Connect to the database (see connection info above) using:
       • psql -h HOST -p PORT -U USER -d DATABASE command, or
       • Your IDE may have integration (VS Code and IntelliJ IDEA Ultimate does).
     • Add a person with first name "John" and last name "Doe" to the table.
   • Query the person just added: http://localhost:8080/hello/Doe
   • Try to query the weather: http://localhost:8080/weather

6. Try out calling the external weather API directly.

   • Open Postman application. See help here .
   • Create a new request to query the weather by the GPS coordinates of Budapest. See the API documentation here: https://openweathermap.org/current

CHECK Create a screenshot with the REST API response.

Even for deploying and starting such a simple application, numerous other services and configuration steps were necessary. This is the reason why integration and system-level testing is much harder than unit testing. In unit testing, we mostly mock any environmental dependencies — now, we have to consider these dependencies in our tests.

If you are done, stop the application. Let's test.

Existing tests (1p)

The original author made various tests to showcase the different testing levels and approaches. Lets examine some of them.

Tests for the person functionality:

• Open ExampleControllerTest.java. This is a unit test for the Controller, where its methods are called directly, and the calls to the database through PersonRepository are mocked with Mockito. In these tests the database is not called and the application is not started.

• Open ExampleControllerAPITest. This is an integration test, where the integration with parts of the Spring framework is tested (however, without starting a full web server yet). The database is still mocked.

  • The API of the Controller is called using the Spring MVC Test Framework instead of calling the Java methods directly. In this way mappings are also tested.
  • Check out the syntax used by MockMvc (perform, andExpect).

• Finally, open HelloE2ERestTest. This is an end-to-end test testing every component. The application is called using its external REST API visible to the clients, and the database is not mocked. This test is considerably slower, as it starts the web server.

  • For firing up REST calls the code uses RestAssured. Notice the syntax for calling and checking (when/get/then/...).
  • We need some test data to work on. Notice how a person is created and saved in the beginning of the shouldReturnGreeting test, and how the repository is cleared between every tests in the tearDown method.
  • Note: the tests use an in-memory database and not the PostgreSQL used at runtime.

The following table summarizes the differences between these tests.

Test	Public API	Spring web	Controller class	Database	Weather API (external)
ExampleControllerTest	No	No	Yes	Mocked	Mocked (Class)
ExampleControllerAPITest	No	Yes	Yes	Mocked	Mocked (Bean)
HelloE2ERestTest	Yes	Yes	Yes	In-memory	Not used

Tests for the Weather API:

• The main difference when testing a function calling an external API is that we usually do not want to call the external API directly. The external API call can be slow, expensive, and return different responses each time. For this reason, we can mock the request at various levels.
• WeatherClientTest shows an example of mocking the RestTemplate class, making the API request using plain Mockito.
• The next level is when we play back a previously recorded response saved in a JSON file (test/resources/weatherApiResponse.json). (This approach also has drawbacks, as we cannot detect problems or API changes in the external service.)
  • Open WeatherClientIntegrationTest to see this in action. WireMock intercepts URLs and returns predefined responses.
  • Note: the tests use their own hardcoded test-specific API key ("someAppId"). Do not modify this, as all the tests use this string in place of a real API key.
  • WeatherAcceptanceTest is quite similar, but this time, the weather functionality is called through the application's REST API.

CHECK run these tests and create a screenshot of the test results.

New feature (2p)

We extended the person entity with location of their home given with GPS coordinates, and a new endpoint was added to the application (/hello/{lastName}/location). But this part of the code is not tested yet, so extend the existing tests.

1. Add a unit test inside ExampleControllerTest to verify that the location method of the ExampleController class correctly returns the home location of a person.
2. Add an integration test inside ExampleControllerAPITest to verify that the new endpoint correctly returns the home location of a person.
3. Add an E2E test inside HelloE2ERestTest to verify that the new REST API endpoint correctly returns the home location of a person.

For the weather service:

1. Capture a new request with Postman. Replace the saved request in weatherApiResponse.json with the new one. Verify that tests pass the new request, and fix them, if needed.

CHECK: create screenshots about the code of the new test methods.