socket: add MPTCP support

[CrapsDorian]

Multipath TCP (MPTCP), standardized in RFC8684 [1], is a TCP extension that enables a TCP connection to use different paths.

Multipath TCP has been used for several use cases. On smartphones, MPTCP enables seamless handovers between cellular and Wi-Fi networks while preserving established connections. This use-case is what pushed Apple to use MPTCP since 2013 in multiple applications [2]. On dual-stack hosts, Multipath TCP enables the TCP connection to automatically use the best performing path, either IPv4 or IPv6. If one path fails, MPTCP automatically uses the other path.

To benefit from MPTCP, both the client and the server have to support it. Multipath TCP is a backward-compatible TCP extension that is enabled by default on recent Linux distributions (Debian, Ubuntu, Redhat, ...). Multipath TCP is included in the Linux kernel since version 5.6 [3]. To use it on Linux, an application must explicitly enable it when creating the socket. No need to change anything else in the application.

This attached patch adds an --mptcp option which allows the creation of an MPTCP socket instead of TCP on Linux. If Multipath TCP is not supported on the system, an error will be reported. It is important to note that if the end server doesn't support MPTCP, the connection will continue after a seamless fallback to TCP.

---

An HTTP test server indicating whether MPTCP (Multipath TCP) is utilized is now available at the following address: http://test.multipath-tcp.org:5000/.

Attention points:

• This PR replaces First version of Multipath TCP support in curl #9713.
• Sorry for functional option MPTCP with --mptcp #13161, I accidentally opened it while the code was not ready yet, my bad.
• Regarding the documentation, we didn't find a good reference (See-also). We put tcp-fastopen because we had to put something, but we are happy to change.
• It is not clear for us if a test is worth it. According to your comments from First version of Multipath TCP support in curl #9713, it might not be worth it. We successfully tested it with http://test.multipath-tcp.org:5000 .
• The transport is changed to TRNSPRT_MPTCP in Curl_cf_tcp_create(), not to change the logic before ("happy eyeballs", etc.), and because this feature is an extension to TCP, and it should only modify TCP connections. We are happy to change that if there is a better way.
• In the declaration of struct OperationConfig in src/tool_cfgable.h, the struct OperationConfig *next; item has this comment next to it: /* Always last in the struct */, but it is no longer the last item in the structure. Is it normal?

[bagder]

the macOS fails on GitHub Actions are the same weird and mysterious problems we have seen pop up in other PRs recently so not related to this

[piru]

Wouldn't it be more clean to rather CURLE_UNSUPPORTED_PROTOCOL or CURLE_NOT_BUILT_IN if the define is missing?
If the idea is to hide the error quietly, then do so, rather than using hardcoded value (which may or may not be the same on all platforms).

[matttbe]

Thank you for the review!

I hope you don't mind if I reply to these questions instead of Dorian (I worked with him)

IPPROTO_MPTCP is defined in the libc in Linux only. To avoid extra #if and issues when the builder is using an old version of the libc, we simply define it if needed. The value should always be 262.

The goal here is then not to hide an issue. If the system doesn't support MPTCP, a proper error will be returned by the kernel when creating the socket. Please note that the libc, the kernel version and its config can be different at build-time and at run-time (e.g. some builders are using chroot/containers with up to date software, but an old stable kernel). So we think it is better for the kernel to return an error at run-time if it is not supported than trying to guess what the end-user will have at build-time.

If we want an explicit error to say that the feature is not supported, maybe it would be better to do that in an #else after the #ifdef __linux__ we added in Curl_cf_tcp_create(), no?

[matttbe]

If we want an explicit error to say that the feature is not supported, maybe it would be better to do that in an #else after the #ifdef __linux__ we added in Curl_cf_tcp_create(), no?

Something like this maybe?

diff --git a/lib/cf-socket.c b/lib/cf-socket.c
index a852a5231..8bc5423d9 100644
--- a/lib/cf-socket.c
+++ b/lib/cf-socket.c
@@ -1610,10 +1610,14 @@ CURLcode Curl_cf_tcp_create(struct Curl_cfilter **pcf,
     goto out;
   }
 
+  if(data->set.mptcp) {
 #ifdef __linux__
-  if(data->set.mptcp)
     transport = TRNSPRT_MPTCP;
+#else
+    result = CURLE_UNSUPPORTED_PROTOCOL;
+    goto out;
 #endif
+  }
 
   cf_socket_ctx_init(ctx, ai, transport);
 

[matttbe]

FYI, I just pushed a new version have this diff (+ a rebase on top of master)

[piru]

I think it is unwise to limit the MPTCP support for Linux only. AFAIK macOS/iOS already support it and FreeBSD is not far behind.

(Granted at least macOS/iOS will require some platform specific code for this, but that can come later)

[matttbe]

It would be great to support other OS. But unfortunately, it is only easily accessible with Linux, thanks to IPPROTO_MPTCP that is for the moment specific to Linux

On iOS, MPTCP is available, but:

• It is easy only if you use their SDK: doc
• If not, you need to use private libraries (we are not even sure the headers are available) with specific functions to create sockets that are apparently not documented, e.g., OpenSSH for iOS

On FreeBSD, there was an ongoing implementation, but that was years ago, and not working today according to this

There are other implementations, but on specific systems (Citrix load balancer, userspace, etc.): more details here

So it looks like we might have to restrict this to Linux only for the moment. Is it an issue? It is always possible to add a #elif later, no?

[matttbe]

@bagder thank you for the review! I just applied the suggested modification and addressed your 2 other comments.

I also rebased the modifications on top of master, and squashed the last modifications. I hope that's OK to do that here. Or maybe you prefer when modifications are done in separated commits?

[bagder]

I also rebased the modifications on top of master, and squashed the last modifications

That's perfect!

[bagder]

Since there is also a multipath QUIC proposal brewing that could perhaps materialize in a future, should we perhaps rather call this option CURLOPT_MULTIPATH to make it possible to support QUIC (or something else) in a distant future?

[matttbe]

Since there is also a multipath QUIC proposal brewing that could perhaps materialize in a future, should we perhaps rather call this option CURLOPT_MULTIPATH to make it possible to support QUIC (or something else) in a distant future?

Good point! 'multipath' can indeed be used in different transport protocols (MPTCP, MPQUIC, MPDCCP), but also at a lower layer (e.g. IP Route Multipath → equal cost multipath) or higher one (e.g. downloading the same file HTTP in multipath parts, using multiple IPs available on the client side: i.e. half downloaded via WiFi, the other half via the cellular network, similar to Samsung's Download Booster feature that was confusedly called "multiple paths HTTP" by some). I usually don't recommend using this generic term because it can be interpreted differently, but maybe a good documentation can help here? What do you think?

Also, it is important to note that even if MPQUIC is pushed by some of the same people who pushed MPTCP (and who even wrote #9713 :) ), its usage by apps is different compared to MPTCP: instead of simply enabling it by changing one line and letting the kernel doing the rest depending on how the latter has been configured, the app will likely have to do this configuration part to have the different paths created and decide on how to use them. Please see the Quiche PR for more details.

QUIC libs supporting MPQUIC will likely come with a default packets scheduler, and maybe a path-manager, so the app can use multiple paths without too much of an effort. But my point here is that with MPQUIC, the end user might have to pass more info to the app: which paths to use or which path manager to select, what packet scheduler technique to use, etc. Apps like Curl might have to support multiple techniques to be able to serve multiple purposes. In this case, I guess there will be other specific MPQUIC options (or env vars?). Do you then think we should have a generic CURLOPT_MULTIPATH option, plus some MPQUIC specific ones? Or only MPTCP and MPQUIC specific ones?

[matttbe]

(I just pushed a new version: it is just a rebase on top of master because there was a conflict with a362962 -- I didn't rename CURLOPT_MPTCP, but feel free to tell us if we should do that)

[matttbe]

It looks like the issues with 'Linux / hyper' job is not specific to this modification: we can see the same errors in other PRs if I'm not mistaken. I don't know if we need to do something for that.

[icing]

It looks like the issues with 'Linux / hyper' job is not specific to this modification: we can see the same errors in other PRs if I'm not mistaken. I don't know if we need to do something for that.

The CI is using the master branch of hyper and they seem to have borked that.

[bagder]

Yes, #13380 tracks that

[bagder]

When using this option, is there a way to tell it is actually working (has the feature enabled once a connection is established)? Since it falls back transparently it will be hard to tell based on behavior.

[matttbe]

Since it falls back transparently it will be hard to tell based on behavior.

Correct.

Note that if the kernel doesn't support MPTCP, an error will be returned: there is no transparent fallback in this case.

If the kernel supports MPTCP, an easy to check if this feature is working correctly with Curl is to use the test server mentioned above, which will return the current status:

$ ./src/curl http://test.multipath-tcp.org:5000
You are not using MPTCP.
$ ./src/curl --mptcp http://test.multipath-tcp.org:5000
You are using MPTCP.

Or by checking externally that the created socket didn't fall back to TCP, e.g. with ss -Mia.

When using this option, is there a way to tell it is actually working (has the feature enabled once a connection is established)?

From an application, there is a way to check that an established connection didn't fallback from MPTCP to TCP with a getsockopt(), e.g.

#define MPTCP_INFO 1
#define MPTCP_INFO_FLAG_FALLBACK 1
int socket_is_mptcp(int sockfd)
{
    socklen_t len = sizeof(struct mptcp_info);
    struct mptcp_info info = { 0 };

    if (getsockopt(sockfd, SOL_MPTCP, MPTCP_INFO, &info, &len) < 0) {
        /* Client: SOL_MPTCP is not supported: kernel < 5.16: we don't know */
        return -1;
    }
    return (info.mptcpi_flags & MPTCP_INFO_FLAG_FALLBACK) == 0;
}

The main downside is that it requires kernel 5.16. There are more details on the new version of our website: Check for TCP fallback

@bagder : When would you like to have such info? Is it just to check the modification works, or to add the info somewhere, only visible in verbose mode?

[CrapsDorian]

oops, sorry, I accidentally pushed on the wrong branch and closed the PR, my bad