edge case issue where icap client is not sending body, making icap server stuck in loop

[GlenDC]

Dear. I have a bit of an odd issue with the icap client of g3. There are edge cases where the icap client sends a RESPMOD (for a response WITH body) and where it correctly sends the ICAP header, http req header and http resp header. Sometimes it also sends the beginning of the first chunk (length), but sometimes that beginning of first chunk is missing. At that point when the ICAP server (mine) would try to read the body it will get stuck as the ICAP client is not sending any data, yet neither is it returning an EOF. it just is not sending anything...

So far I cannot find a pattern on why. There are no errors. Sometimes it's a original response which uses transfer encoding (chunked) itself but just as many times the original response was just using content-length (no transfer-encoding).

So I think there is a bug somewhere, where the ICAP client does not send the (response) body as part of the RESPMOD, despite there being one, and just getting stuck somehow.

[zh-jq-b]

Can you capture both the ICAP traffic and the upstream response?
You can also test if it is reproducible with c-icap, which is used in CI.

[GlenDC]

Can you capture both the ICAP traffic and the upstream response

Due to reasons I cannot share such traffic. I could give you very minimal extract from it with some stuff redacted but don't think it would be telling more than what I reported above.

You can also test if it is reproducible with c-icap,

I'll try to see if I can provide you with a reproducible setup with c-icap server. But might not be soon.

[GlenDC]

Ok @zh-jq-b I have been playing around with the c-icap server (https://c-icap.sourceforge.net/install.html) and it seems the bug (or set of bugs) is in the ICAP client implementation used by g3proxy for sure as I see the same "hang" as I see with my own ICAP server.

I also think I've narrowed it down (and yes I am on latest commit of g3 btw).

Seems that the ICAP flow in g3 doesn't handle very well in case the server is returning content chunked encoding style. When that happens, especially with some delay between the chunks, it seems the g3 client code just... hangs

You can try it yourself with two websites, chosen just for demonstration purposes, they are nothing to do with my real traffic:

• https://anglesharp.azurewebsites.net/Chunked for chunked transfer-encoding (domain anglesharp.azurewebsites.net)
• https://www.bbc.com/ for content-length (no chunking) (domain www.bbc.com)

Here is the (headers) dump of both:

• the raw traffic coming into g3proxy for that bbc / GET request (bbc_egress_inc.txt)
• and the ICAP traffic (bbc_icap_inc.txt)

bbc_egress_inc.txt
bbc_icap_inc.txt

If you try this with a g3proxy binary and the c-icap server you'll see all loads fine.

However... now try a website which returns a response in chunked encoding. That's why I've chosen https://anglesharp.azurewebsites.net/Chunked as it always returns chunked encoding, to make sure we trigger this.

Here is the (headers) dump of both:

• the raw traffic coming into g3proxy for that bbc /Chunked GET request (chunked_egress_inc.txt)
• and the ICAP traffic (chunked_icap_inc.txt)

chunked_egress_inc.txt
chunked_icap_inc.txt

If you try this website yourself (or any other which returns a http response with chunked content) you'll see that it just hangs... Try it without g3 proxy and you'll see this page:

But go via g3proxy and you'll just have a page which never loads and stays white.

[zh-jq]

Can you try #625?
The preview data is not consumed if it's a complete chunk, which leads to corrupt data.

[GlenDC]

My own icap server doesn't make use of preview or supports it and it sees the same behaviour FYI

[GlenDC]

That said. I'll try it out in some hours. Thx already upfront!

[GlenDC]

Can you try #625? The preview data is not consumed if it's a complete chunk, which leads to corrupt data.

Seems to have fixed it. Thank you. Want me to close it already, or test for some more days first?

[zh-jq-b]

Let's close this after you are sure that the problem is fixed.

[GlenDC]

@zh-jq-b the problem still seems to be there for me. Despite being up to date. The problem is that the ICAP Server I work with does not work with preview for RESPMOD's (so does not allow 204's). As such it needs to consume the body and returns a 200 OK with the data back.

However, at that point it still gets stuck on transfer-encoding chunked. Content-length responses are fine.

[zh-jq]

@GlenDC You can capture the upstream traffic (via udpdump) and the ICAP respmod traffic to check if the body encoding is broken.

Here is the guide to setup udpdump: https://g3-project.readthedocs.io/projects/g3proxy/en/latest/protocol/setup/wireshark_udpdump.html

[GlenDC]

How do I configure in my g3proxy config to expose this data over udpdump? That doesn't seem to be clear from your docs?

[zh-jq]

How do I configure in my g3proxy config to expose this data over udpdump?

g3/g3proxy/examples/inspect_http_proxy/g3proxy.yaml

Line 12 in c48eeb6

tls_stream_dump: { }

[GlenDC]

Or do you mean I just intercept the udp traffic from the l

How do I configure in my g3proxy config to expose this data over udpdump?

g3/g3proxy/examples/inspect_http_proxy/g3proxy.yaml

Line 12 in c48eeb6

tls_stream_dump: { }

That copies the raw bytes on an interface available for udpdump tools? If so, neat!

[zh-jq]

That copies the raw bytes on an interface available for udpdump tools? If so, neat!

Yes it will dump the remote side traffic by default, to udp port 5555 which should be listen to by wireshark udpdump tool. You can add client-side: true to see client side traffic.

[GlenDC]

Doesn't seem to help me in parrticular @zh-jq . Perhaps I misunderstand what you are trying to do here, but I was getting more info from my regular wireshark captures, where I was also able to see both the ICAP and http traffic. That's what I linked earlier in this issue. Doesn't seem udpdump tool gives me much more info, in fact seems less? And also not that easy to show in wireshark or how do you use it? I can see the frames appear but they seem pretty hard to use and less info than what I had before without udpdump