GET request (client) with default :keepalive times out when the remote end tries to end the connection

[dkharlan]

I'm seeing an issue where a long-running download by the client (several hundred GET requests) will time out.

;; in project.clj
[http-kit "2.2.0"]

;; in my code
(require '[org.httpkit.client :as http])
(doseq [url a-whole-bunch-of-urls]
   (let [data @(http/get url {:as :byte-array})]
      ...)))

After doing a packet capture I noticed that the timeouts come at the end of the download; what appears to happen is that a single TCP connection is used (as expected with the default :keepalive). After a while (and many successful requests) the remote end tries to close the connection with a FIN packet. The remote FIN is ACKed but never answered with FIN from http-kit; the output of ss confirms that the underlying socket has entered the CLOSE_WAIT state. It stays this way until the socket's timeout interval elapses and it times out, at which point I see http-kit send its own FIN and the connection closes. http-kit sets the :error key in its response with the following:

org.httpkit.client.TimeoutException: connect timeout: 60000ms
 at org.httpkit.client.HttpClient.clearTimeout (HttpClient.java:82)
    org.httpkit.client.HttpClient.run (HttpClient.java:433)
    java.lang.Thread.run (Thread.java:745)

I've been able to reproduce this (and its workaround; see below) in multiple environments, so I don't think this is related to any particular hosting / platform issues.

The workaround I've found is to disable Keep-Alive by passing :keepalive -1 as a request option.

[daonsh]

I also have an issue of lots of CLOSE_WAIT sockets on the server. It reaches the maximal amount and after a few days I get an exception "too many open files".

But you write "socket is stuck in the CLOSE_WAIT state. This continues until the socket's timeout interval elapses and it times out"

This sounds good, how can I set the socket's timeout so the CLOSE_WAIT disappears? If they disappear, why is this a problem?

Thanks

[dkharlan]

To clarify, I may very well be misidentifying some key part of the failure case -- the way I noticed this problem at all was that my requests were timing out, and my (perhaps flawed?) understanding of how TCP works is that http-kit should have sent a FIN in response to the remote end trying to close the connection, rather than moving to CLOSE_WAIT and then timing out. That being said the CLOSE_WAIT here may be a red herring since they don't actually persist, so I'll update my description to deemphasize that.