Proposal v4.0 b1 (HTTP/2+3, OS Trust Store, Custom DNS, OCSP, ...)

[Ousret]

This PR showcases how HTTPie could evolve outside of Requests.

Niquests is supposed to be a (mostly) compatible fork of Requests.

Try this preview:

$ pip install "git+https://github.com/Ousret/HTTPie.git@feature-tryout-niquests" -U

Here are the biggest pros of this:

• OS truststore by default, no more certifi!
• Object-oriented headers. Could bring additional features!
• Fully type-annotated!
• HTTP/3 over QUIC.
• HTTP/2 by default.
• Exit Python http.client! in favor of h11.
• Timeout by default.
• Inspect peer certificate, HTTP version, TLS version, cipher, and so on via hook/callback before a request is sent.
• All the features you expose are available in all three protocols.
• Python 3.7+, no sacrifice needed.
• Encrypted DNS support. w/ DNS-over-HTTPS, DNS-over-QUIC, DNS-over-TLS and plain DNS-over-UDP.

Obviously, cons:

• Stricter on emitted requests per RFCs
• Young project but based on solid bases, knowledge, and experiences.
• Need to publish new packages to distro. Easy but time-consuming
• Exit pyopenssl, not supported. more of a pro to me
• Require major bump? Could be. Should be.

Complete list of changes in the fork: https://github.com/jawah/niquests/blob/main/HISTORY.md

• Close httpie refuses to trust a self-signed certificate #576
• Close HTTPie ignores system certificates #480
• Close Support HTTP/2 (httpie-http2 plugin doesn't work) #692
• Close enable quic protocol #523
• Close show me the SSL certificate the server is using? #549
• Close System for meta information output options (timing, SSL info, …) #1023
• Close Include SSL/TLS and other metadata information in --meta/-vv #826
• Close "SSL: CERTIFICATE_VERIFY_FAILED" should show cert at least with --debug #800
• Close Display used server IP in verbose mode #1495
• Close SSL/TLS server certificate validation #1202
• Close Add equivalent cURL --resolve flag #99
• Close binding to an IP #1422
• Close Add -4 and -6 switches to force IP protocol version #94
• Close I would like the option to disable the DNS Cache and do name resolution on every request #1480
• Close Unable to install httpie via Chocolatey #1522
• Close Multiple response headers with same name combined to a single comma-separated list #1413
• Close Digest authentication with qop=auth-int fails "TypeError: expected string or bytes-like object"  #1446
• Close Should use the user-provided Host header for SNI #414
• Close HTTPie could be lead to believe data was passed in stdin when it was not #1551
• Close add local-port flag #1456
• Close can't install chocolatey issues with python error code 1603 #1338
• Close Bad filename when having non ASCII characters #1401
• Close Request is fired to packages.httpie.io on every invocation of http #1527
• Close Cookies with "Domain=localhost" aren't getting stored in session file #602
• Close Change insecure SSL/TLS to deprecated #962
• Close httpie -d does not work with gzip compressed content #1554
• Close --download reports an "incomplete download" with Content-Encoding: gzip and content-length. #423
• Close Deprecate PROTOCOL_SSLv23 #1400
• Close Feature Request: Support special-use domain names like localhost. #1458

---

4.0.0.b1 (unreleased)

• Make it possible to unset the User-Agent, and Accept-Encoding headers. (#1502)
• Dependency on requests was changed in favor of compatible niquests. (#1531)
• Added support for HTTP/2, and HTTP/3 protocols. (#523) (#692) (#1531)
• Added request metadata for the TLS certificate, negotiated version with cipher, the revocation status and the remote peer IP address. (#1495) (#1023) (#826) (#1531)
• Added support to load the operating system trust store for the peer certificate validation. (#480) (#1531)
• Added detailed timings in response metadata with DNS resolution, established, TLS handshake, and request sending delays. (#1023) (#1531)
• Added support for using alternative DNS resolver using --resolver. DNS over HTTPS, DNS over TLS, DNS over QUIC, and DNS over UDP are accepted. (#99) (#1531)
• Added support for binding to a specific network adapter with --interface. (#1422) (#1531)
• Added support for specifying the local port with --local-port. (#1456) (#1531)
• Added support for forcing either IPv4 or IPv6 to reach the remote HTTP server with -6 or -4. (#94) (#1531)
• Removed support for pyopenssl. (#1531)
• Removed support for dead SSL protocols < TLS 1.0 (e.g. sslv3) as per pyopenssl removal. (#1531)
• Dropped dependency on requests_toolbelt in favor of directly including MultipartEncoder into HTTPie due to its direct dependency to requests. (#1531)
• Dropped dependency on multidict in favor of implementing an internal one due to often missing pre-built wheels. (#1522) (#1531)
• Fixed the case when multiple headers where concatenated in the response output. (#1413) (#1531)
• Fixed an edge case where HTTPie could be lead to believe data was passed in stdin, thus sending a POST by default. (#1551) (#1531)
  This fix has the particularity to consider 0 byte long stdin buffer as absent stdin. Empty stdin buffer will be ignored.
• Slightly improved performance while downloading by setting chunk size to -1 to retrieve packets as they arrive. (#1531)
• Added support for using the system trust store to retrieve root CAs for verifying TLS certificates. (#1531)
• Removed support for keeping the original casing of HTTP headers. This come from an outer constraint by newer protocols, namely HTTP/2+ that normalize header keys by default.
  From the HTTPie user perspective, they are "prettified" on the output by default. e.g. "x-hello-world" is displayed as "X-Hello-World".
• Fixed multipart form data having filename not rfc2231 compliant when name contain non-ascii characters. (#1401)
• Fixed issue where the configuration directory was not created at runtime that made the update fetcher run everytime. (#1527)
• Fixed cookie persistence in HTTPie session when targeting localhost. They were dropped due to the standard library. (#1527)
• Fixed downloader when trying to fetch compressed content. The process will no longer exit with the "Incomplete download" error. (#1554) (#423) (#1527)
• Added automated resolution of hosts ending with .localhost to the default loopback address. (#1458) (#1527)

Existing plugins are expected to work without any changes. The only caveat would be that certain plugin explicitly require requests.
Future contributions may be made in order to relax the constraints where applicable.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (4d7d6b6) to head (7cd6579).
Report is 370 commits behind head on master.

❗ Current head 7cd6579 differs from pull request most recent head e3cbc7f

Please upload reports for the commit e3cbc7f to get more accurate results.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #1531       +/-   ##
==========================================
- Coverage   97.28%       0   -97.29%     
==========================================
  Files          67       0       -67     
  Lines        4235       0     -4235     
==========================================
- Hits         4120       0     -4120     
+ Misses        115       0      -115     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Ousret]

So far, I identified this:

• Close httpie refuses to trust a self-signed certificate #576
• Close HTTPie ignores system certificates #480
• Close Support HTTP/2 (httpie-http2 plugin doesn't work) #692
• Close enable quic protocol #523
• Close show me the SSL certificate the server is using? #549

[Ousret]

We can add the following arguments and document them:

• --disable-http2
• --disable-http3
• --http3, e.g. force trying it.

[Ousret]

I've added and documented it.

• --disable-http2
• --disable-http3
• --http3, e.g. force trying it.

Ideally, the connection information (TLS, Cipher, Peer/Issuer Certificate, and OCSP) should be in the request metadata and with a lexer instruction.

[Ousret]

Now, the result is pleasant to the eye.
I have implemented the metadata for Request and the result is as follows:

Niquests now support tracking upload progress, this can also be done here. It should be straightforward enough.

[dwt]

Damn this looks so tantalizing. I'd love for this to get packaged as a tryout / beta. Something installable via e.g. pip install httpie[niquest] perhaps?

[Ousret]

However I noticed the absence of a --http2 flag to force http2. Could you touch on the rationale behind that omission?

You cannot force HTTP/2 using Niquests. It is negotiated via ALPN/TLS. So if the server misinterpret ALPN or discard it, HTTP/2 won't come. And yes, RFC allows it, see https://datatracker.ietf.org/doc/html/rfc9113#name-starting-http-2-with-prior- but I decided to follow the simplest path forward (ALPN/TLS), as browsers do (https://stackoverflow.com/questions/34076231/why-do-browser-implementations-of-http-2-require-tls). I might reconsider it in the future, but the use cases are rare.

[dwt]

I may be missing something here, but does that imply that the client has no way to force http1 2 or 3 according to its wishes? When setting up server support for these protocols, I would like to have the ability to force all of these protocols for debugging purposes?

[Ousret]

I may be missing something here, but does that imply that the client has no way to force http1 2 or 3 according to its wishes?

There's many possibilities, the only missing piece is "force http2" or "disable http1".

• "--disable-http2" HTTP/1.1 or HTTP/3
• "--disable-http2 --disable-http3" HTTP/1.1
• "--disable-http3" HTTP/1.1 or HTTP/2
• "--http3" HTTP/3

Hope that clarify.

[Ousret]

Just updated the docs to include previous comment about protocol combo and the availability of HTTP/3 or not.
Since last time Niquests made HTTP/3 support far more enjoyable with less dependencies and a faster execution time.

Finally, we had to downgrade macos-14 (macos-latest) to macos-13 because it's relatively new and some dependencies like brotlicffi aren't ready for it.

[mshahat]

Hello @Ousret , nice work here ... is there an ETA on when to expect this release to become available? Thank you

[Ousret]

No precise ETA. We are waiting upon the owner approval.

[markstos]

Looks like it could be a big step forward if @jkbrzt approves it. Last week I wanted to test resolving over IPv4 vs IPv6. I found that the released version could not help with this task, but it worked perfectly in the fork, with the same -4 and -6 flags that dig uses.

[ducaale]

Excited to see HTTP/2 and HTTP/3 soon landing in HTTPie 🎉

I may be missing something here, but does that imply that the client has no way to force http1 2 or 3 according to its wishes?

There's many possibilities, the only missing piece is "force http2" or "disable http1".
"--disable-http2" HTTP/1.1 or HTTP/3
"--disable-http2 --disable-http3" HTTP/1.1
"--disable-http3" HTTP/1.1 or HTTP/2
"--http3" HTTP/3

I was wondering if we could have an equivalent to cURL's http-version flags i.e --http1.0,--http1.1,--http2,--http2-prior-knowledge,--http3? Alternatively, it could be a --http-version flag that accepts either 1.0, 1.1, 2, 2-prior-knowledge or 3

[Ousret]

Excited to see HTTP/2 and HTTP/3 soon landing in HTTPie 🎉

Likewise!

I was wondering if we could have an equivalent to cURL's http-version flags

I first need to add a feature into Niquests, which isn't top priority right now.
But it is planed, will be in a future minor. Provided everything goes smooth with this.

regards,

[janbrasna]

Love the idea, and without digging any deeper into the changeset to verify my thoughts in implementation, I'm just raising it here for general discussion:

"This is will default to TLS v1.0 which will negotiate the highest protocol that both […] support."

1. This sounds a bit confusing. It's either "This is will default to TLS v1.0" (which it should not), or "This is will default to a highest TLS version that both […]".

• Because the values don't mean "use as the lowest", but "use the chosen only" (i.e. not a tls12+ or a range, but tls12 only)
• I'd actually propose a magic value like "tls" (roughly meaning an alias to PROTOCOL_TLS_CLIENT) that would use the highest protocol available. Not sure if that's the default thou, not sure if the client can have tls13 capability, but still use tls12 as a default? If default == max always, then there's prolly no need for such value, only way to use max protocol then is to omit the param though. Which would mean no "max" magic value would be listed in help/man among the "available protocols", which in turn might seem a bit confusing then. So an explicit value for max protocol available, even when it really doesn't do anything if that's the default, can still have its value for clarity.
• Do you plan to silently accept the old "ssl2.3" value mapped to default, whatever that ends up to be, not to raise any errors?

3. Are you sure the "tls1.3" value works? It does not in master, and never did (since the openssl apis are different for TLSv13, and the change in flags structure [see 4. ↓] made it impossible to constrain without all *_OP_NO_* used) — it only selects TLSv13 if that would have been the default anyways…

4. Adding ranges, i.e. min=tls11 max=tls12, or just one constraint alone (to match the current python sslcontext, instead of selecting one version, or disabling individual one by one) would also be lovely at some point.

[Ousret]

Indeed, the "This is will default to ..." is kind of confusing. I've addressed that by rephrasing it.

1. It is rephrased accordingly with "If not specified, it tries to negotiate the highest protocol that both the server and your installation of OpenSSL support."
2. "tls1.3" case has been addressed, we are now assuming that "tls1.3" (associated to PROTOCOL_TLS_CLIENT) means strictly do tls 1.3, and appropriate measures have been made.
3. old ssl2.3 value and alike are gone, completely, accepting them is only going to generate noise in GH issues. tls1.0 is the absolute minimum nowadays. nothing in CPython 3.7+ (+whether OpenSSL 1.1.1+ or 3+) permit to support them. so it will ends up with errors no matter what we do.
4. The range of accepted tls version should be dealt with separately, I don't want to further increase this PR size, thus delaying it further on. I agree it can be useful, but it's not top priority as we speak.

regards,

[Ousret]

To demonstrate --ssl=tls1.3 is working, run https --ssl=tls1.3 https://tls-v1-2.badssl.com:1012/