HW2 ‐ Test generation

The goal of this homework is to try out different test generation methods:

1. model-based testing (MBT) and a model-based test generator tool,
2. and code based test generation.

Environment setup

Setup a VM with Ubuntu 22.04. If you do not want to install the OS yourself, there are ready-made VMs available to download on OSboxes.

This homework will require Java 17, and an IDE to edit the code. These can be installed via:

sudo apt update
sudo apt install openjdk-17-jdk
sudo snap install intellij-idea-community --classic

Model-based testing (MBT)

In this homework we will use the open-source GraphWalker tool. (There are many more MBT tools, see e.g. this overview.)

Overview of GraphWalker

GraphWalker in a nutshell: GraphWalker

• uses an Extended Finite State Machine (EFSM) model for test design,
• can generate paths from the graph model using different algorithms and coverage criteria, and
• uses Java code for the adaptation layer to connect the model to the implementation to support test execution.

Modeling

• Models are stored in JSON format and can be created in GraphWalker Studio.
• The model is a combination of a Usage and System model (according to the terminology used in the lecture). The SUT is modeled from a testing point of view:
  • Edges represent the input events coming from the users or the environment of the SUT. Input events do not have parameters.
  • Output actions of the SUT are not modeled, but the nodes of the model represent the states, where the SUT should move after the given input events.
• The model can contain variables that can be initialized in nodes and manipulated in actions on edges. The variables are to be used in guards on edges.
• GraphWalker can work with several models to create hierarchical models using the SHARED keyword in nodes.
• Many GraphWalker examples use the v_ prefix for nodes (vertex) and e_ for edges, but it is not mandatory.

Paths generated from the model are the test sequences:

• edges represent receiving inputs and performing some action in the SUT, and
• nodes represent performing some validation that the SUT indeed behaved as expected and responded and/or moved to the expected state.

The recommendation of the authors of GraphWalker is to try to model first with simple FSMs, and use only EFSMs (variables) when necessary, as it greatly increases complexity.

Test generation

GraphWalker supports the following generators:

• random and weighted random walk,
• A* (shortest path to a specific vertex or edge),
• quick random.

GraphWalker supports the following stopping conditions:

• edge or vertex coverage [%],
• requirement coverage [%],
• reach a specific edge or vertex,
• time duration.

Tests can be generated offline or online:

• offline: GraphWalker only prints the generated paths. These can be used to validate the model or store them to execute later in some way.
• online: GraphWalker is connected to the SUT and executes each step on the SUT before generating the next step. Test adaptation code is required to map events in the model to the actual implementation.

SEE: Generators and stop conditions

Test adaptation

An adaptation layer is required to concretize the abstract model elements and map them to the elements of the implementation.

GraphWalker can use Java code for adaptation. From the adaptation code any test execution framework or tool can be called (e.g. simple Java code if the SUT is a Java library, Selenium if the SUT is a web application or Appium if the SUT is a mobile app).

The structure of the adaptation code is the following:

• GraphWalker generates an interface from the model, where for each node and edge there is a method without parameters.
• Tests should implement this interface and call the SUT in the methods for edges, and verify the SUT's response or state using assert or verify calls in the methods for nodes.

Usage of GraphWalker

GraphWalker is quite flexible and can be used and configured in several ways.

Command line interface (CLI)

GraphWalker binary (graphwalker-cli-X.Y.Z.jar) needs to be downloaded from the project's website, and called from the command line.

The location of the model, the generator and stopping condition to use can be specified as parameters.

Maven

GraphWalker can be used directly in a Maven build just by adding GraphWalker as a dependency in the pom.xml. We will use this method in the current lab.

The models should be placed inside the src/main/resources or test/main/resources folder in a folder structure similar to the application's package hierarchy.

There are several goals for GraphWalker, e.g.:

• graphwalker:generate-test-sources: generates the interface from the model. The interface is placed inside the target/generated-test-sources/graphwalker/ folder.
• graphwalker:test: will call GraphWalker for all classes annotated with the @GraphWalker annotation.

The generator and stopping conditions can be specified as parameters in the @GraphWalker annotation. In this case no @Test methods are needed. For example:

@GraphWalker(start = "e_Init", pathGenerator = RandomPath.class, stopCondition = VertexCoverage.class, stopConditionValue = "100")

(Note: Another option is to specify generators and stopping conditions imperatively in Java code in JUnit @Test methods. In this case, the usual test goal should be called instead of graphwalker:test. However, this is not the recommended setup.)

SEE: GraphWalker Maven (see the various goals in wiki pages on the right)

Exercises

IMPORTANT: Use the latest version of GraphWalker, as previous versions were not working on newer JDKs.

Try out GraphWalker Studio (1p)

Check out the home page of GraphWalker, and watch the short animations and read the text to understand:

• how models can be created in the Studio web app,
• what is the basic usage process of the tool.

Try out GraphWalker Studio:

• Download the latest GraphWalker studio.
• Start it using the following command:

  java -jar graphwalker-studio-4.3.2.jar
  

• Open the following URL in your browser:

  http://localhost:9090/studio.html
  

• Create a vertex, set it as the start element. Create another vertex, and connect them with an edge.
  • Remember: v should be pressed while clicking for creating a vertex. e should be pressed while holding down the left button and dragging the cursor from one vertex to another for creating an edge. (Yes, really.)
  • Some laptops disable the touchpad when a key is pressed. If you are using a laptop and creating a vertex or an edge does not work, try using a mouse or check the configuration of your touchpad.
• Simulate the model with the Play button.
• Save the model. It will be downloaded as test.json, which can be renamed later.

CHECK: Include a screenshot of your model in GraphWalker Studio running locally on your VM.

Windshield wiper exercise (2p)

In this exercise, we will create the model from a specification and write the adaptation code interacting with a Java implementation. (The system to model is greatly simplified compared to reality to fit the timeframe of the lab.)

Specification

The system under test is an automatic, rain-sensing windshield wiper controller. The controller receives the position the user sets for the wiper switch with possible values off, speed one, speed two or auto. The controller, in response, sets the wiping interval for the wiper motor: for speed one, it is 1 second, and for speed two, it is 0.5 seconds. In the auto mode, the controller acts according to the data received from the rain sensor: if the rain sensor reports light rain, then it sets the interval to the same as used for speed one, while for heavy rain, it sets the interval of speed two. If there is no rain, then it turns off the wiper (interval value 0).

Your task is to test that the controller sets the correct interval values.

Project setup

Use the WindshieldWiper skeleton project from the hw2-test-generation folder in this repository. Clone the avt-labs repository and navigate to the appropriate folder.

• The project consists of three sub-modules:

  • wiper-api: interface of the SUT,
  • wiper-impl: implementation of the SUT only available as a JAR file,
  • wiper-test: project for the MBT model and test code.

• The actual SUT is available in a jar file. In the wiper-impl folder, execute the following command to install it to the local Maven repository:

  mvn org.apache.maven.plugins:maven-install-plugin:2.5.2:install-file   -Dfile=wiper-impl-1.0-SNAPSHOT.jar
  

  (If you are using Windows, use a Command Prompt for these commands and not a PowerShell window, as PowerShell will not interpret the parameters correctly.)

• Change back to the WindshieldWiper folder and install the other projects in the local repository (this way, the test module will always find the api module).

  mvn install
  

Tasks

• Create a model representing the behavior of the SUT.

  • Open the WiperModel.json file from wiper-test/test/resources in Studio as a starter.
  • Pay attention to the fact that the model should be abstract and not necessarily contain every detail. The model should use keywords (events, states) from the specification and the domain, and not from the implementation.
  • Try to work in an iterative way: create a model without the auto mode first.
  • Did you find anything ambiguous or missing in the specification during model development?

• Save the model in the same json file.

• Change to the wiper-test folder and check the syntax of your model with the command

  mvn graphwalker:validate-test-models

• Take a look at the wiper-api project. It consists of the following main parts:

  • Interface of the SUT: WiperController
  • Two enums used in the interface: WiperMode, RainMode
  • The SUT calls the wiper motor, which has the following interface: WiperMotor

• Take a look at the wiper-test project. From now on, work in this project.

  • A sample method in WiperControllerTest shows how the SUT can be instantiated (the implementation for the WiperMotor is not available, therefore some kind of test double needs to be used).

• Generate an interface from the model with

  mvn graphwalker:generate-test-sources

  • It will create an interface in the target folder inside generated-test-sources.
  • Run the mvn graphwalker:test command. It will result in an error eventually ("No start context found"), but will compile the WiperModel interface that can be later used.
  • Take a look at the generated interface: it has the @Model annotation for the class, and @Edge and @Vertex annotations for the model elements.
  • Note: GraphWalker will not generate a method for the start vertex as this is more like an initial pseudo-state. Place your initialization code in the method generated for the edge going out from start.

• The task is now to implement this interface in a test adaptation code.

  • Create a class called WiperModelTest under src/test/java in the proper package hierarchy.

  • Add the following annotation, extends and implements directives to the class:

    @GraphWalker(start = "init")
    public class WiperModelTest extends ExecutionContext implements WiperModel{}

  • Implement the methods defined in the interface:

    • Methods for edges need to call the SUT and perform the appropriate actions.
    • Methods for nodes need to check the state or interactions of the SUT.

  • As WiperMotor has not been implemented yet, use Mockito to mock it.

• Add a suitable generator and stopping condition to the @GraphWalker annotation.

• You can start generating tests with

  mvn graphwalker:test

• Did you find any problems in the implementation?

CHECK: Create a screenshot of your final model.

CHECK: Create a screenshot of your final test code and create a screenshot from the output of GraphWalker.

CHECK: Answer the question whether you found any problems.

Code based test generation

The goal of this homework is to try out tools that can generate tests from source or binary code (code-based or white-box test generators).

These tools select relevant test inputs achieving high coverage or triggering exceptions. Moreover, they record the observed behavior and return values and encode them in assertions (in this way these tests could be used in regression testing).

We will try Randoop, which uses feedback-directed random test generation.

WARNING: Randoop will call the methods in the classes under test with random parameters. Be careful and do not run these tools on code that writes/deletes files, as during test generation it could delete your data!

Randoop

• Read the introduction and stages of test generation sections of Randoop's manual to get a quick overview about the tool.

• Download the latest version of randoop-all jar from the releases. As of now it is randoop-all-4.3.2.jar.

• Set an environment variable pointing to the downloaded jar file.

  export RANDOOP_JAR=/home/.../randoop-all-4.3.2.jar
  

• Run the following command to print the help message for generating tests in order to check that everything is working so far.

  java -cp $RANDOOP_JAR randoop.main.Main help
  

Exercises

Error-revealing tests (1p)

• Use the Randoop tutorial project from the hw2-test-generation folder of the avt-labs repository.

  • Note: the documentation of the tutorial is a bit out of date, the syntax of some commands might have changed. Follow this guide, or check Randoop's manual about the actual syntax if you encounter an error.

• The tutorial contains the implementation of MyInteger.java, which has an add, equals and multiply methods among others. The tutorial has several steps, where different versions of MyInteger is copied to the source folder.

• Move to the first part of the tutorial.

./gradlew first

• Open the project in your favourite IDE, and examine MyInteger and the current tests in MyIntegerTest. The implementation is simple, but it seems to be fine.
• Run the existing manually created tests:

 ./gradlew test

• Generate tests for the MyInteger class. Randoop needs the compiled class files, and you need to set the Java class path correctly, otherwise Randoop will not found it.
• Navigate to the root folder of the tutorial. The MyInteger.class is located inside the folder build/classes/java/main/math, and its fully qualified name is math.MyInteger.
• Call Randoop to generate tests for Stack:
  • The : character is the separator in the class path (-cp). Note that the testclass is given with its fully qualified name, and the .class extension is not needed.
  • The --testclass specifies the class to generate tests for.
  • The --junit-output-dir sets the folder where the generated tests are placed.
  • The --output-limit parameter sets the limit for the number of generated tests.

  java -cp build/classes/java/main:$RANDOOP_JAR randoop.main.Main gentests --testclass=math.MyInteger --junit-output-dir=src/test/java --output-limit=20

• Execute the generated tests (./gradlew test)

• Examine the generated tests! Randoop generated error-revealing tests (test violating some general contract or some implicit test oracles) and regression tests (tests capturing the current behavior for some generated inputs).

• Investigate the error-revealing tests!

  • What are the test checking?
  • Find the problem in the implementation!

• Use the following command to load a fixed version of the implementation.

  ./gradlew second

• See the fix in the source of the implementation and run again the tests to validate the fixes.

CHECK: Describe the issue that caused the tests to fail.

Changing the default parameters (1p)

Explore the functionality of Randoop by changing the parameters of the test generation. More information can be found in the detailed manual.

Some initial ideas:

• Change the limits (time, output). Is Randoop able to increase the coverage?
• Change the values used in tests (nulls, literals...).
• Change classification of tests (e.g. whether exceptions are considered errors).
• Try to specify expected code behavior with pre- and post-conditions.

CHECK: Create a screenshot of the revised test generation commands.