Expose ability to customize ClientHello message #1932
Comments
|
I think the goals of parroting other TLS implementations for the benefit of fingerprinting is at odds with other goals we have, such as not contributing to the ossification of TLS on the internet. That is why we randomise extension orders. I have looked over craftls and I think it looks very good -- I think you should use it, contribute to it, and financially support it if it aligns with your needs. While maintaining a fork is extra work, that work would be borne by someone else if we extended the scope of rustls to cover parroting and that would reduce our ability to deliver on more mainstream features. |
That's great to hear. I have not looked at it with as much care, but I think the crux here is about coming up with the nicest way of supporting that kind of feature set, while minimising the effort on the fork side. To this end, as an example, following a chat with @FiloSottile (one of the maintainers or crypto/tls), we came up with a design for doing a low maintenance fork of crypto/tls called utls-light. My knowledge of rust in general and of rusttls in particular, is not as good as my knowledge of go, so the reason to bring this up here was to understand if there was a way to do something similar for rust too. The main idea behind utls-light, is that we minimize the amount of lines of code that need to change compared to upstream, by using the raw bytes as the API to plug into modifying the clienthello and then hook into the places of the TLS state machine were we need to support the extra features. You can see what I mean by inspecting this diff: ooni/utls-light@2b6c2ef...4dfb1fc OTOH looking at the changeset of the craftls I get the impression that if substantial changes were to happen in rusttls, the effort required to adapt the fork would be also quite substantial: 3andne/craftls@4d1b762...craft-0.22.0. My question is therefore, given that I have gathered there is no interest to natively support this feature set in rusttls, would you be open to collaborating on making the integration of something like that as smooth as possible? |
That sounds interesting, but I'm having a hard time understanding what the shift was on the Go side in the stdlib |
|
As the one who tried integrating this into upstream and currently is maintaining a fork of rustls, I would say it's possible to add this feature with minimal amount of lines of code change before #1475 is merged (5 files and <10 lines modification for each). However, it's not feasible to only have a byte-level API exposed after the construction of ClientHello, because rustls will internally check the integrity of the ClientHello by the hash sent by the server. And I don't think we should violate this behavior. My current implementation modifies the construction process of ClientHello, whose modified values are also recorded and validated against by rustls. However, this piece of code change will be difficult to be made after #1475 is merged. |
Checklist
I believe there are some related issues on the tracker, but none of them fully describe the specific use-case I have in mind. I list them here:
ClientHelloto match JA3 fingerprinting #1421Is your feature request related to a problem? Please describe.
It's becoming a growing problem for servers and TLS middleboxes to perform fingerprinting of the TLS stack used by clients in order to implement various forms of restrictions.
In the case of server-side fingerprinting, services like cloudflare are known to use the TLS fingerprint to implement bot detection and implement different kinds of policies in response to a particular TLS fingerprint. For these cases it's important for the TLS implementation to match whatever is being sent at the HTTP layer. For example if your ClientHello doesn't match exactly the ClientHello of the TLS stack used by whatever you end up sending inside of HTTP (eg. your ClientHello looks like Chrome, but then your User-Agent and HTTP header order is that of Firefox), you will run into blocks.
Relevant research on this:
It has also been seen in the wild that on certain particularly restrictive networks, such as Iran, the filtering rules have specific rules that apply to particular ClientHello messages. As a result of this there are efforts to make encrypt the client hello, however they are not yet widely deployed and they significantly increase the complexity both on the server and on the client (for the Encrypted ClientHello extension to work you depend on having a working encrypted DNS connection, which is also blocked in some places).
As a concrete example to this, we published an article explaining how the default golang tls stack in certain android configurations lead to the cipher suites being sorted in a specific way that leads to blocking in Iran: https://ooni.org/post/making-ooni-probe-android-more-resilient/#changing-our-android-tls-fingerprint.
For some more background on the problem, here is a selection of relevant papers:
In light of all these considerations it would be great if rusttls exposed some way for developers to increase the resilience of the TLS transport by exposing capabilities to customise the ClientHello message which is commonly used for server-side and TLS middlebox fingerprinting.
It's worth noting that in order for this to work and not break during the handshake, the implementer of the ClientHello customization needs to only advertise capabilities that are actually supported by the rusttls stack, so in order to fully parrot say chrome, you will have to support all the extensions it has like certificate compression, all the cipher suits it advertises, etc.
Describe the solution you'd like
Ideally there would be a way from the public rust-tls API to customize the ClientHello message down to the byte level. While I haven't looked at it very carefully, I believe what's being done in #1475 is a step in the right direction.
The only things I would say are that it would be ideal if the API exposed a way to specify the exact ClientHello message being sent down to the byte level. It is my understanding that the API there will allow to specify a subset of fields in the ClientHello message, while leaving other fields out of it. It also does not offer the capability, as far as I can tell, to enforce a specific ordering of fields, which matter from the perspective of fingerprinting.
A good way to check if your fingerprint is unique or not, is using this service: https://tlsfingerprint.io/, which tells you how popular it is in the wild.
Additional context
The most popular way of implementing something of this sort in the go ecosystem is this library: https://github.com/refraction-networking/utls/. Basically it's a complete fork of the crypto/tls standard library with patches applied on top of it to implement a parroting API for popular clients.
However I think that in rust there is the opportunity to have a nicer solution to this problem, by having first class support in a library like rusttls.
In terms of architecture for a potential solution, I think it would be worth taking a look at what I did to implement a simpler version of utls which works by parsing the ClientHello message after it's serialized by the go standard library and it's re-assembled with the desired target ClientHello. See: https://github.com/ooni/utls-light.
This approach has the benefit of not committing a specific API inside of the library itself, since the compatibility is guaranteed by the fact that a ClientHello message shall always match the TLS protocol specification. It's however a dangerous API, because as mentioned before if you advertise some extensions or ciphers not in the rusttls implementation, you might end up breaking the handshake depending on the server.
For this reason it might be worth making use of some of the feature flags in rust to enable or disable this API to prevent people from unintentionally shooting themselves in the foot.
The reason for me to open this issue is to start the discussion about this topic and understand if there is some solution that can be worked on that would be accepted by upstream.
Edit: I was informed of an effort aimed at creating a utls-like thing for rust as well, which can be found here: https://github.com/3andne/craftls, however it works by forking rusttls and I would image the work needed to keep in sync with main would be non-trivial (similar to what utls is doing and why it's so time consuming).
The text was updated successfully, but these errors were encountered: