Docker flow return 400 on POST/DELETE actions if SSL used #410

nohaapav · 2018-01-15T18:32:37Z

Hi, i'm facing strange issue with letsencrypt companion. Everything seems to be setup correctly, i've got new certificate as well and all GET operations works fine. But when i try to send POST or DELETE resource FE end up with "No server is available to handle this request." returned from proxy even though resource is actually created/deleted.

Everything works fine with http so wondering what might be the issue here.

This is proxy response in case of error:

proxy_proxy.1.i800oo238jco@swarm-worker000007 2018/01/15 18:28:01 HAPRoxy: 10.255.0.8:52972 [15/Jan/2018:18:28:02.305] services~ swarmpit_app-be8080_0/swarmpit_app 0/0/1/-1/3 400 658 - - CH-- 3/3/2/2/0 0/0 "DELETE /services/v75pwent0u6jk67yefwc9oyzm HTTP/1.1"

Steps to reproduce the issue:

Use https with docker-flow
Try POST / DELETE operations

Describe the results you received:
400 - "No server is available to handle this request."

Describe the results you expected:
200 - OK

Additional information you deem important (e.g. issue happens only occasionally):

Additional environment details (AWS, VirtualBox, physical, etc.):

vfarcic · 2018-01-15T23:59:00Z

Normally, DFP does not set a GET/POST/PUT/DELETE protocol explicitly. If GET works, other should work as well (at least from DFP perspective). In most cases, certificates offloading is done in DFP. That means that HTTPS request is sent to the proxy which forwards it as HTTP (without S) to the destination service. I'd suggest we start by checking whether DELETE requests work without the proxy. Can you please (temporarily) open a port directly in the service (swarmpit) and send a DELETE request to http://[NODE_IP]:[SERVICE_PORT]/services/v75pwent0u6jk67yefwc9oyzm. Depending on whether it worked or not, we'll plan what to do next.

nohaapav · 2018-01-16T09:03:34Z

Hi yes normally it works like charm, either with opened port or via DFP (HTTP) .. It only happens when HTTPS is used. I'm using DFP companion to get ACME. Can see that certificate gets downloaded for provided serviceDomain and GET operation works fine with HTTPS but if i try to do POST or DELETE 400 is returned from proxy -> no service is available to handle request even though request is actually processed by backend.

vfarcic · 2018-01-19T00:58:12Z

I don't think it's directly related to DFP. I have quite a few services with certificates and haven't noticed a similar problem.

Can you:

Send a DELETE HTTP (not HTTPS) request to the service and confirm that it works? You can do that by entering one of the DFP containers, installing curl (e.g., apk add -U curl), and issuing curl -XDELETE http://[SERVICE_NAME]:[SERVICE_PORT]/....
Post DFP config (e.g. cat /cfg/haproxy.cfg from inside a DFP container)?

nohaapav · 2018-01-28T16:06:37Z

Its kinda strange. I have 3 swarm clusters which 2 of them are using LE with companion and works fine (TEST/CI). The one i have the issue with(PROD) is using 4 custom certificates. These are valid according ssllabs.

GET operation work like charm but once i try POST/DELETE with https it fails in random cases.
I turned off default 400 redirect and response in case of failure is:

<html>
<body>
<h1>400 Bad request</h1>
Your browser sent an invalid request.
</body>
</html>

DFP config looks good. I compared PROD haproxy.cfg with TEST haproxy.cfg and configuration does not differ except cfg frontend section on PROD contains these lines (should be harmless):

http-request set-var(req.scheme) str(https) if { ssl_fc }
http-request set-var(req.scheme) str(http) if ! { ssl_fc }

Given issue is happening only for e.g. kibana, swarmpit service. Others works pretty well even with https. But the same services works on TEST without any problems.

Only difference are the certificates (TEST -> acme, PROD -> custom signed)

Really out of ideas what is going wrong ..

nohaapav · 2018-01-28T16:18:26Z

These occasional (but persistent) 400 errors are pretty much same as here:

#405

nohaapav · 2018-01-28T16:34:23Z

Well problem solved. The issue was indeed caused by http2.

@vfarcic Feel free to close, and many thanks for amazing project ;)

vfarcic closed this as completed Jan 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker flow return 400 on POST/DELETE actions if SSL used #410

Docker flow return 400 on POST/DELETE actions if SSL used #410

nohaapav commented Jan 15, 2018

vfarcic commented Jan 15, 2018

nohaapav commented Jan 16, 2018

vfarcic commented Jan 19, 2018

nohaapav commented Jan 28, 2018

nohaapav commented Jan 28, 2018 •

edited

Loading

nohaapav commented Jan 28, 2018

Docker flow return 400 on POST/DELETE actions if SSL used #410

Docker flow return 400 on POST/DELETE actions if SSL used #410

Comments

nohaapav commented Jan 15, 2018

vfarcic commented Jan 15, 2018

nohaapav commented Jan 16, 2018

vfarcic commented Jan 19, 2018

nohaapav commented Jan 28, 2018

nohaapav commented Jan 28, 2018 • edited Loading

nohaapav commented Jan 28, 2018

nohaapav commented Jan 28, 2018 •

edited

Loading