pg-streaming-data-collection

Data collection of performance traces

This folder contains a multi-project build in Gradle for collecting system traces from different APM vendors.

The multi-project contains the following subprojects, each with an own README:

• pg-trace-schema: a Java library that contains our domain model for a system trace using Apache Avro. It is used by all other apps.
• pg-streaming-trace-collection: a Java / Kafka Streams app that can be configured to either collect traces from Nagios or Telegraf and transform the data to our domain model/schema.

Run data collection

The easiest way to run the project is to use Gradle and Docker Compose.

Please make sure to have the following technologies installed:

• Java / JDK (e.g., AdoptOpenJDK)
• Gradle
• Docker (>= 3.2 to support profiles)

We use a Docker Gradle plugin to build Docker images.

Build the multi-project and docker images by running:

gradle build dockerBuildImage

To start all default containers, that is, Zookeeper, Kafka, Schema Registry and pg-streaming-trace-collection, run the following command:

docker-compose up

Alternatively, use profiles with Compose to start additional containers:

• docker-compose --profile telegraf up starts Telegraf to generate test data (see config)

• docker-compose --profile debug up starts Kafdrop

To stop and remove everything, we recommend using the following command to prevent future errors with Apache Kafka:

docker-compose rm -sfv