Correlate Span ID and Trace ID to Kong logs.

[mfischler]

Version of kong: 3.5.0
Platform: kubernetes
ddtrace version: 0.1.2-1
Configuration Type: Specific services
Configuration Details:

{ "instance_name": null, "enabled": true, "updated_at": 1704398691, "route": null, "created_at": 1704398691, "id": "8bba4f3d-7c08-5dc7-a4e7-e77d6cf58e12", "tags": null, "consumer": null, "config": { "resource_name_rule": null, "static_tags": null, "service_name": "{vault://env-vault/KONG_SERVICE_NAME}", "environment": "{vault://env-vault/KONG_ENVIRONMENT}", "initial_sample_rate": null, "agent_endpoint": "{vault://env-vault/KONG_DATADOG_AGENT_TRACING_ENDPOINT}", "initial_samples_per_second": 100 }, "service": null, "protocols": [ "grpc", "grpcs", "http", "https" ], "name": "ddtrace" }

Detailed description of the problem, and if known, the expected behavior.

Not Sure if possible, is there a way to correlate the Span ID and Trace ID to the logs collected from Kong?

Validation:

Kong logs would include the tags "span.id" and "trace.id" generated by the Datadog Trace plugin.

[dmehala]

Hi @mfischler,

Thank you for reporting this issue. Our team is currently reviewing the details, and we'll work on addressing it as soon as possible.

To further enhance the visibility of your request and for future feature suggestions, we encourage you to submit them through our dedicated support platform at Datadog Support Portal.

[mfischler]

Closed a feature implemented using environment variables. Updated technique provided by Datadog.

[danielpoonwj]

can we store the trace id in ngx.ctx? then it can be up to the users to implement the log injection depending on their use case.

if you're using the file-log plugin, you can use custom_fields_by_lua to access the value in ngx.ctx and write it to dd.trace_id in the JSON log

[dmehala]

If ngx.ctx is shared between plugins, then it can be used to share traceID and spanID between plugins, thus making it available for file-log plugin.

On another note, I was able to correlate traces with logs by setting an NGINX variable. However, that solution would requires to manually update the log format which is not that straightforward for Kong.

[danielpoonwj]

If ngx.ctx is shared between plugins, then it can be used to share traceID and spanID between plugins, thus making it available for file-log plugin.

On another note, I was able to correlate traces with logs by setting an NGINX variable. However, that solution would requires to manually update the log format which is not that straightforward for Kong.

yup i think ngx.ctx, ngx.var or kong.ctx.shared can all be potentially used for exposing the trace id outside the plugin.

can we consider setting it on any of those places and documenting it for others to use? we actually maintain a private internal fork just for this but would love to use the upstream if this was implemented

[dmehala]

I opened #61. However, I need to play a bit with log correlation to know if the TraceID and SpanID is enough. Hopefully that will be merged and released by tomorrow. Thank you!