Skip to content

Add OpenTelemetry distributed tracing #149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
flands wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add OpenTelemetry distributed tracing #149

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d104a78
Add OpenTelemetry distributed tracing
flands May 8, 2020
da1a923
Update README
flands May 18, 2020
0ddd19f
Upgrade to v0.6.0
flands Jul 23, 2020
e8f196d
set newest otel collector dependency
kuba-wu Sep 21, 2020
54a32e6
clean-up, configured with latest splunk otel distro
kuba-wu Apr 7, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ If everything goes well, you can access the following services at given location
* Config Server - http://localhost:8888
* AngularJS frontend (API Gateway) - http://localhost:8080
* Customers, Vets and Visits Services - random port, check Eureka Dashboard
* Otel Collector - http://localhost:55680
* Tracing Server (Zipkin) - http://localhost:9411/zipkin/ (we use [openzipkin](https://github.com/openzipkin/zipkin/tree/master/zipkin-server))
* Admin Server (Spring Boot Admin) - http://localhost:9090
* Grafana Dashboards - http://localhost:3000
Expand Down
68 changes: 68 additions & 0 deletions collector.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
extensions:
health_check:
zpages:
endpoint: 0.0.0.0:55679

receivers:

jaeger:
protocols:
grpc:
endpoint: "0.0.0.0:14250"

otlp:
endpoint: "0.0.0.0:55680"

prometheus:
config:
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: ['localhost:8888']
# Used to query metrics
labels:
environment: test
metric_relabel_configs:
- source_labels: [ __name__ ]
regex: '.*grpc_io.*'
action: drop

processors:

batch:

attributes/copyfromexistingkey:
actions:
- key: environment
from_attribute: env
action: upsert
attributes/newenvironment:
actions:
- key: environment
value: "test"
action: insert

queued_retry:

exporters:

logging:

zipkin:
url: "http://tracing-server:9411/api/v2/spans"
Copy link
Member

@arey arey May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 9411 port should not be hardcoced. It's coming from the OpenZipkin configuration file: https://github.com/spring-petclinic/spring-petclinic-microservices-config/blob/master/tracing-server.yml


service:

pipelines:

traces:
receivers: [jaeger, otlp]
processors: [batch, attributes/copyfromexistingkey, attributes/newenvironment, queued_retry]
exporters: [logging, zipkin]

#metrics:
# receivers: [prometheus]
# exporters: [logging]

extensions: [health_check, zpages]
26 changes: 20 additions & 6 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ services:
mem_limit: 512M
depends_on:
- config-server
entrypoint: ["./dockerize","-wait=tcp://config-server:8888","-timeout=60s","--","java", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
entrypoint: ["./dockerize","-wait=tcp://config-server:8888","-timeout=60s","--","java", "-javaagent:/opt/opentelemetry-auto-0.2.2.jar", "-Dota.exporter.jar=/opt/opentelemetry-auto-exporters-jaeger-0.2.2.jar", "-Dota.exporter.jaeger.endpoint=otel-collector:14250", "-Dota.exporter.jaeger.service.name=discovery", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
Copy link
Member

@arey arey May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it exists a better integration with Spring Boot / Spring Could?

Copy link
Author

@flands flands May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is bytecode (auto) instrumentation -- for Java this is done via -javaagent. This is a common industry practice done by many vendors. The goal is not to be Java framework specific.

ports:
- 8761:8761

Expand All @@ -25,7 +25,7 @@ services:
depends_on:
- config-server
- discovery-server
entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=60s","--","java", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=60s","--","java", "-javaagent:/opt/opentelemetry-auto-0.2.2.jar", "-Dota.exporter.jar=/opt/opentelemetry-auto-exporters-jaeger-0.2.2.jar", "-Dota.exporter.jaeger.endpoint=otel-collector:14250", "-Dota.exporter.jaeger.service.name=customers", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
ports:
- 8081:8081

Expand All @@ -36,7 +36,7 @@ services:
depends_on:
- config-server
- discovery-server
entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=60s","--","java", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=60s","--","java", "-javaagent:/opt/opentelemetry-auto-0.2.2.jar", "-Dota.exporter.jar=/opt/opentelemetry-auto-exporters-jaeger-0.2.2.jar", "-Dota.exporter.jaeger.endpoint=otel-collector:14250", "-Dota.exporter.jaeger.service.name=visits", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
ports:
- 8082:8082

Expand All @@ -47,7 +47,7 @@ services:
depends_on:
- config-server
- discovery-server
entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=60s","--","java", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=60s","--","java", "-javaagent:/opt/opentelemetry-auto-0.2.2.jar", "-Dota.exporter.jar=/opt/opentelemetry-auto-exporters-jaeger-0.2.2.jar", "-Dota.exporter.jaeger.endpoint=otel-collector:14250", "-Dota.exporter.jaeger.service.name=vets", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
ports:
- 8083:8083

Expand All @@ -58,7 +58,7 @@ services:
depends_on:
- config-server
- discovery-server
entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=60s","--","java", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=60s","--","java", "-javaagent:/opt/opentelemetry-auto-0.2.2.jar", "-Dota.exporter.jar=/opt/opentelemetry-auto-exporters-jaeger-0.2.2.jar", "-Dota.exporter.jaeger.endpoint=otel-collector:14250", "-Dota.exporter.jaeger.service.name=api-gateway", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
ports:
- 8080:8080

Expand All @@ -78,7 +78,7 @@ services:
depends_on:
- config-server
- discovery-server
entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=60s","--","java", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
entrypoint: ["./dockerize","-wait=tcp://discovery-server:8761","-timeout=60s","--","java", "-javaagent:/opt/opentelemetry-auto-0.2.2.jar", "-Dota.exporter.jar=/opt/opentelemetry-auto-exporters-jaeger-0.2.2.jar", "-Dota.exporter.jaeger.endpoint=otel-collector:14250", "-Dota.exporter.jaeger.service.name=admin", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
ports:
- 9090:9090

Expand All @@ -97,3 +97,17 @@ services:
mem_limit: 256M
ports:
- 9091:9090

# Collector
otel-collector:
image: omnition/opentelemetry-collector-contrib:latest
command: ["--config=/etc/otelcol-config.yaml", "--new-metrics", "--legacy-metrics=false"]
depends_on:
- tracing-server
volumes:
- ./collector.yaml:/etc/otelcol-config.yaml
ports:
- "13133:13133" # health_check extension
- "14250:14250" # Jaeger gRPC
- "55679:55679" # zpages extension

16 changes: 15 additions & 1 deletion docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,28 @@ ARG DOCKERIZE_VERSION
RUN wget -O dockerize.tar.gz https://github.com/jwilder/dockerize/releases/download/${DOCKERIZE_VERSION}/dockerize-alpine-linux-amd64-${DOCKERIZE_VERSION}.tar.gz
RUN tar xzf dockerize.tar.gz
RUN chmod +x dockerize
RUN wget -O opentelemetry-auto-0.2.2.jar https://github.com/open-telemetry/opentelemetry-auto-instr-java/releases/download/v0.2.2/opentelemetry-auto-0.2.2.jar
Copy link
Member

@arey arey May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it possible to avoid wget the OpenTelemtry artifacts and prefer Maven depencies?

Moreover, the Open Telemetry version 0.2.2 is hardcoded at a lot of places.

Copy link
Author

@flands flands May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it appears the auto-instrumentation is not published on Maven yet -- I will file an issue for this to get it fixed.

RUN wget -O opentelemetry-auto-exporters-jaeger-0.2.2.jar https://github.com/open-telemetry/opentelemetry-auto-instr-java/releases/download/v0.2.2/opentelemetry-auto-exporters-jaeger-0.2.2.jar

# This is the first layer that won't be cached
ARG ARTIFACT_NAME
ADD ${ARTIFACT_NAME}.jar /app.jar
RUN wget -O /opt/opentelemetry-auto-0.2.2.jar https://github.com/open-telemetry/opentelemetry-auto-instr-java/releases/download/v0.2.2/opentelemetry-auto-0.2.2.jar
RUN wget -O /opt/opentelemetry-auto-exporters-jaeger-0.2.2.jar https://github.com/open-telemetry/opentelemetry-auto-instr-java/releases/download/v0.2.2/opentelemetry-auto-exporters-jaeger-0.2.2.jar

ARG EXPOSED_PORT
EXPOSE ${EXPOSED_PORT}

ENV SPRING_PROFILES_ACTIVE docker

ENTRYPOINT ["java", "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap", "-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
ENTRYPOINT ["java", \
"-javaagent:/opt/opentelemetry-auto-0.2.2.jar", \
"-Dota.exporter.jar=/opt/opentelemetry-auto-exporters-jaeger-0.2.2.jar", \
"-Dota.exporter.jaeger.endpoint=otel-collector:14250", \
"-Dota.exporter.jaeger.service.name=config", \
"-XX:+UnlockExperimentalVMOptions", \
"-XX:+UseCGroupMemoryLimitForHeap", \
"-Djava.security.egd=file:/dev/./urandom", \
"-jar", \
"/app.jar" \
]