sentry's baggage header breaks the request

[Wozacosta]

Describe the bug

The header baggage set by sentry (referenced here), when present in the request, breaks it and we get back :

Received an HTTP message that the node could not decode

To Reproduce
Curl with baggage header set:

curl 'https://api.mainnet.hiro.so/v2/fees/transaction'
-H 'Content-Type: application/json'
-H 'baggage: sentry-environment=production'
--data-raw '{"transaction_payload":"000516a66269b5011c8feb0eb8ca6fe39ecef63a8b6c7b000000000000000048686a00000000000000000000000000000000000000000000000000000000000000","estimated_len":180}'
The same without the baggage header, which works:

curl 'https://api.mainnet.hiro.so/v2/fees/transaction'
-H 'Content-Type: application/json'
--data-raw '{"transaction_payload":"000516a66269b5011c8feb0eb8ca6fe39ecef63a8b6c7b000000000000000048686a00000000000000000000000000000000000000000000000000000000000000","estimated_len":180}'

Expected behavior
When present, the baggage header shouldn't make the node unable to decode the message.

Smartphone (please complete the following information):

• Device: iPhone13
• OS: ios18.0
• Browser: react-native
• Version 6.11

[zone117x]

I'm able to reproduce this and confirm it seems to be an issue in this repo's RPC proxy for this endpoint:

Sending through the API proxy:

> curl 'https://api.mainnet.hiro.so/v2/fees/transaction' \
-H 'Content-Type: application/json' \
-H 'baggage: sentry-environment=production' \
--data-raw '{"transaction_payload":"000516a66269b5011c8feb0eb8ca6fe39ecef63a8b6c7b000000000000000048686a00000000000000000000000000000000000000000000000000000000000000","estimated_len":180}'

Received an HTTP message that the node could not decode

Sending directly to a stacks-node:

> curl 'http://localhost:20443/v2/fees/transaction' \
-H 'Content-Type: application/json' \
-H 'baggage: sentry-environment=production' \
--data-raw '{"transaction_payload":"000516a66269b5011c8feb0eb8ca6fe39ecef63a8b6c7b000000000000000048686a00000000000000000000000000000000000000000000000000000000000000","estimated_len":180}'

{"estimated_cost":{"write_length":0,"write_count":0,"read_length":0,"read_count":0,"runtime":0},"estimated_cost_scalar":6,"estimations":[{"fee_rate":9381.786636377206,"fee":56290},{"fee_rate":34399.01538494635,"fee":206394},{"fee_rate":34399.02162053775,"fee":206394}],"cost_scalar_change_by_byte":0.00476837158203125}

[zone117x]

@rafaelcr I'll try to tackle this next week. But feel free to take it if you want, if I haven't yet moved the ticket to in-progress.

[rafaelcr]

I had some time to look into this today and found some interesting things:

1. If I run a local reader API that proxies to a local node, the request works ✅
2. If I run a local reader API that proxies to one of our deployed nodes (through port forwarding), the request works ✅
3. If I proxy to a deployed reader API which proxies to a deployed node, the request fails ❌

Furthermore, I found that the problem is the = character in the baggage: sentry-environment=production header. If I remove it, the request works. I think the forwarding process in k8s is somehow encoding the header value to something like baggage: sentry-environment%3Dproduction which then trips up the Stacks node HTTP server's header parsing code.

Do you have a clue on what could be the cause here @CharlieC3 ? i.e. do you know if there's an internal k8s routing thing that might be rewriting the value of the forwarded header?

[CharlieC3]

I think I may have more info to offer here. I was able to confirm I was seeing the same issue as above, and that changing the = fixed the problem.

Some findings:

• I confirmed that the header was not being encoded/changed on transmit by inspecting the full traffic logs of the Istio sidecar the downstream stacks node uses. This means a header of baggage: sentry-environment=production was reaching the Stacks node exactly as it was provided in the request, with the =.
• I found changing the name of the header from baggage to something else like bbaggage: sentry-environment=production (note the = was kept) also works.
• As Matt noted, I confirmed sending the request as-is directly to a stacks node without the API proxy works fine as well.
• I found that the stacks node is was logging the following error:

  Invalid message preamble: Failed to decode HTTP request or HTTP response (request error: DeserializeError("Invalid HTTP request: duplicate header \"baggage\""); response error: DeserializeError("Failed to parse HTTP response: Version"))
  

• Sending the same header twice yields the same Received an HTTP message that the node could not decode error, regardless of the header.

@rafaelcr @zone117x Does the API happen to use Sentry? Or rather does it have any tracing-adjacent libraries that would intercept the baggage tracing header? If so, I'm wondering if when the API is proxying requests to a backend stacks node and it contains the recognized Opentelemetry baggage header, the API either creates another one or appends to it in some way that breaks the backend stacks node.

Typically Sentry or tracing libraries are only "active" in a deployed production environment. @rafaelcr This might explain why you're not seeing this problem when running it locally.

[zone117x]

Nice cache on the duplicate header. I'm not sure yet where the extra baggage header is added. I'd consider this behavior on stacks-node either a bug or at least highly unusual. I've never seen an http server reject requests with duplicate (non-critical) headers, typically they are merged or ignored. But it's probably easier for us to modify the proxy layer than core.

Before we look into that, @CharlieC3 can Istio be configured to handle this (de-dupe / merge duplicate headers)?

[CharlieC3]

Istio already normalizes headers by default, so I'm not entirely sure how a duplicate is sneaking though. I agree though that this is an issue on the stacks-core end, I'll follow it up with some core devs.
In the meantime I've addressed this by dropping the baggage header.