fix(pdk) - respect the trace flags from the parent span (if present) #13015
+7
−4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Kong respects the `should_sample` decision from a parent span if one was provided. Previously it was not doing this resulting in incomplete traces. When using kong as an internal api gateway we are seeing incomplete traces. The logs show that the following appears to be happening: (this is based on a sampling rate of 0.01) - Service A makes a request to Service B with no traceparent header specified - Kong generates a traceparent header and specifies trace flags of `"01"` (i.e. the trace should be sampled) - Service B handles the request from Service A and ingests the traceparent header - Service B then makes a request to Service C, setting the traceparent header appropriately (i.e. using flags `"01"`) - Kong then resets the trace flags to `"00"` (i.e. the trace should *not* be sampled) - Service C handles the request from Service B and, respecting the flags from the traceparent header, does not sample the trace This results in Kong and Services A and B sampling the trace, but Service C does not, and so we get an incomplete trace. I identified the `get_sampling_decision` method as the most likely place where the issue occurs, and since `parent_should_sample` is a tri-state boolean (i.e. it can be `true`, `false`, or `nil`) it makes most sense (to me) to respect the value of `parent_should_sample` if it exists. I tested this locally and it behaves as I would expect; when a traceparent header is included in the request, then the flags are respected, however if not, probabilistic sampling is applied. However rereading the code, I can't work out *why* that should be the case, since the probablistic sampler is deterministic based on trace id (i.e. the same trace-id will always return the same outcome). Even stepping through the code, I couldn't work out exactly what was going on. It seems likely to me that it's something to do with how the root span is constructed, but I couldn't tell for sure.
|
|
github-actions
bot
added
core/pdk
core/tracing
cherry-pick kong-ee
team-eng-enablement
added
the
author/community
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Kong respects the
should_sampledecision from a parent span if one was provided. Previously it was not doing this resulting in incomplete traces.When using kong as an internal api gateway we are seeing incomplete traces. The logs show that the following appears to be happening:
(this is based on a sampling rate of 0.01)
"01"(i.e. the trace should be sampled)"01")"00"(i.e. the trace should not be sampled)This results in Kong and Services A and B sampling the trace, but Service C does not, and so we get an incomplete trace.
I identified the
get_sampling_decisionmethod as the most likely place where the issue occurs, and sinceparent_should_sampleis a tri-state boolean (i.e. it can betrue,false, ornil) it makes most sense (to me) to respect the value ofparent_should_sampleif it exists.I tested this locally and it behaves as I would expect; when a traceparent header is included in the request, then the flags are respected, however if not, probabilistic sampling is applied.
However rereading the code, I can't work out why that should be the case, since the probablistic sampler is deterministic based on trace id (i.e. the same trace-id will always return the same outcome). Even stepping through the code, I couldn't work out exactly what was going on. It seems likely to me that it's something to do with how the root span is constructed, but I couldn't tell for sure.
Checklist
changelog/unreleased/kongorskip-changeloglabel added on PR if changelog is unnecessary. README.md