Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enabling TLSv1.3 breaks mbedtls (manually or by upgrading to 3.6.0) #917

Open
jmir1 opened this issue May 11, 2024 · 13 comments
Open

Enabling TLSv1.3 breaks mbedtls (manually or by upgrading to 3.6.0) #917

jmir1 opened this issue May 11, 2024 · 13 comments
Labels

Comments

@jmir1
Copy link
Contributor

jmir1 commented May 11, 2024

I tried compiling mbedtls with TLS 1.3 support in my fork, but for some reason it resulted in mpv not being able to play neither 1.2 nor 1.3 streams.

What I did was adding ./scripts/config.py set MBEDTLS_SSL_PROTO_TLS1_3 in mbedtls.sh, I also tried some other options like MBEDTLS_SSL_TLS1_3_COMPATIBILITY_MODE, but no luck.

I'm interested in this because I recently updated my jellyfin server's nginx config to only offer tls 1.3 but noticed that mpv-android doesn't support it yet.

@jmir1 jmir1 added the question label May 11, 2024
@jmir1
Copy link
Contributor Author

jmir1 commented May 11, 2024

Some relevant logs:

mpv/ffmpeg          V  Opening <stream link>
mpv/ffmpeg          V  https: Setting default whitelist 'http,https,tls,rtp,tcp,udp,crypto,httpproxy'
mpv/ffmpeg          V  tcp: Original list of addresses:
mpv/ffmpeg          V  tcp: Address <ip> port 443
mpv/ffmpeg          V  tcp: Interleaved list of addresses:
mpv/ffmpeg          V  tcp: Address <ip> port 443
mpv/ffmpeg          V  tcp: Starting connection attempt to <ip> port 443
mpv/ffmpeg          V  tcp: Successfully connected to <ip> port 443
mpv/ffmpeg          E  tls: mbedtls_ssl_handshake returned -0x6c00
mpv/stream          E  Failed to open <stream link>
mpv/cplayer         V  Opening failed or was aborted: <stream link>
mpv/cplayer         V  finished playback, loading failed (reason 4)

@sfan5
Copy link
Member

sfan5 commented May 11, 2024

Did you change the mbedtls version to 3.6.0 by chance?
Because I found that upgrading to 3.6.0 would make all TLS connections fail with this exact error, but maybe it's not related to the major version but rather TLSv1.3.

@jmir1
Copy link
Contributor Author

jmir1 commented May 11, 2024

Did you change the mbedtls version to 3.6.0 by chance? Because I found that upgrading to 3.6.0 would make all TLS connections fail with this exact error, but maybe it's not related to the major version but rather TLSv1.3.

You are right, i did upgrade to 3.6.0, and reverting to not using tls 1.3 still gives the same error.

I also upgraded from ffmpeg 6.1.1 to 7.0, I don't know if that's also related

@sfan5
Copy link
Member

sfan5 commented May 12, 2024

and did you test TLSv1.3 with mbedtls 3.5.2?

@sfan5
Copy link
Member

sfan5 commented May 12, 2024

@jmir1
Copy link
Contributor Author

jmir1 commented May 12, 2024

I just tested with 3.5.2 and it gives the same error with TLS 1.3 on and works with it off.
That's why 3.6.0 fails: Mbed-TLS/mbedtls@27eb68d, they enabled it by default.

I tested different ffmpeg versions as well, had no effect.
I think it's probably a bug in ffmpeg.

I don't really know how to enable debug logging, just enabling "MBEDTLS_DEBUG_C" definitely didn't give me any logcat output...

What do you think about openssl btw?

@jmir1 jmir1 changed the title How to enable TLSv1.3? Enabling TLSv1.3 breaks mbedtls (manually or by upgrading to 3.6.0) May 12, 2024
@jmir1
Copy link
Contributor Author

jmir1 commented May 12, 2024

I tried compiling ffmpeg with openssl support and tls 1.3 works as expected.
But it adds a 6MB dependency per arch for libcrypto and libssl

@sfan5 sfan5 added bug and removed question labels May 12, 2024
@sfan5
Copy link
Member

sfan5 commented May 12, 2024

What do you think about openssl btw?

Compiling it in makes ffmpeg unredistributable (--enable-nonfree), it's also quite big.

I don't really know how to enable debug logging, just enabling "MBEDTLS_DEBUG_C" definitely didn't give me any logcat output...

It needs integration, try this:

diff --git a/libavformat/tls_mbedtls.c b/libavformat/tls_mbedtls.c
--- a/libavformat/tls_mbedtls.c
+++ b/libavformat/tls_mbedtls.c
@@ -26,6 +26,7 @@
 #include <mbedtls/platform.h>
 #include <mbedtls/ssl.h>
 #include <mbedtls/x509_crt.h>
+#include <mbedtls/debug.h>
 
 #include "avformat.h"
 #include "internal.h"
@@ -109,6 +110,12 @@ static int mbedtls_recv(void *ctx, unsigned char *buf, size_t len)
     return handle_transport_error(h, "ffurl_read", MBEDTLS_ERR_SSL_WANT_READ, ret);
 }
 
+static void mbedtls_debug(void *ctx, int lvl, const char *file, int line, const char *msg)
+{
+    URLContext *h = (URLContext*) ctx;
+    av_log(h, AV_LOG_ERROR, "%s%d: %s", file, line, msg);
+}
+
 static void handle_pk_parse_error(URLContext *h, int ret)
 {
     switch (ret) {
@@ -185,6 +192,9 @@ static int tls_open(URLContext *h, const char *uri, int flags, AVDictionary **op
     mbedtls_x509_crt_init(&tls_ctx->ca_cert);
     mbedtls_pk_init(&tls_ctx->priv_key);
 
+    mbedtls_ssl_conf_dbg(&tls_ctx->ssl_context, mbedtls_debug, shr->tcp);
+    mbedtls_debug_set_threshold(4);
+
     // load trusted CA
     if (shr->ca_file) {
         if ((ret = mbedtls_x509_crt_parse_file(&tls_ctx->ca_cert, shr->ca_file)) != 0) {

@jmir1
Copy link
Contributor Author

jmir1 commented May 12, 2024

 E  tcp: ssl_tls13_client.c285: client hello: adding key share extension
 E  tcp: ssl_tls13_generic.c1651: Perform PSA-based ECDH/FFDH computation.
 E  tcp: ssl_tls13_generic.c1689: psa_generate_key() returned -27648 (-0x6c00)
 E  tcp: ssl_client.c1012: <= write client hello
 E  tcp: ssl_tls.c4617: <= handshake
 E  tls: mbedtls_ssl_handshake returned -0x6c00
 E  tcp: ssl_msg.c6187: => write close notify
 E  tcp: ssl_msg.c6198: <= write close notify
 E  tcp: ssl_tls.c5521: => free
 E  tcp: ssl_tls.c5583: <= free

These are the logs I got, they don't mean much to me though.

The diff you sent has a typo btw, ssl_context instead of ssl_config, that gave me some trouble lol

diff --git a/libavformat/tls_mbedtls.c b/libavformat/tls_mbedtls.c
--- a/libavformat/tls_mbedtls.c
+++ b/libavformat/tls_mbedtls.c
@@ -26,6 +26,7 @@
 #include <mbedtls/platform.h>
 #include <mbedtls/ssl.h>
 #include <mbedtls/x509_crt.h>
+#include <mbedtls/debug.h>
 
 #include "avformat.h"
 #include "internal.h"
@@ -109,6 +110,12 @@ static int mbedtls_recv(void *ctx, unsigned char *buf, size_t len)
     return handle_transport_error(h, "ffurl_read", MBEDTLS_ERR_SSL_WANT_READ, ret);
 }
 
+static void mbedtls_debug(void *ctx, int lvl, const char *file, int line, const char *msg)
+{
+    URLContext *h = (URLContext*) ctx;
+    av_log(h, AV_LOG_ERROR, "%s%d: %s", file, line, msg);
+}
+
 static void handle_pk_parse_error(URLContext *h, int ret)
 {
     switch (ret) {
@@ -185,6 +192,9 @@ static int tls_open(URLContext *h, const char *uri, int flags, AVDictionary **op
     mbedtls_x509_crt_init(&tls_ctx->ca_cert);
     mbedtls_pk_init(&tls_ctx->priv_key);
 
+    mbedtls_ssl_conf_dbg(&tls_ctx->ssl_config, mbedtls_debug, shr->tcp);
+    mbedtls_debug_set_threshold(4);
+
     // load trusted CA
     if (shr->ca_file) {
         if ((ret = mbedtls_x509_crt_parse_file(&tls_ctx->ca_cert, shr->ca_file)) != 0) {

@jmir1
Copy link
Contributor Author

jmir1 commented May 12, 2024

For now, I will just update to 3.6.0 in my fork and disable TLSv1.3 by default until we find a fix

@sfan5
Copy link
Member

sfan5 commented May 13, 2024

seems to be this Mbed-TLS/mbedtls#8401
ffmpeg is not calling psa_crypto_init

@jmir1
Copy link
Contributor Author

jmir1 commented May 13, 2024

seems to be this Mbed-TLS/mbedtls#8401 ffmpeg is not calling psa_crypto_init

I added psa_crypto_init() and now psa_generate_key() seems to work but now I'm getting a new error when it validates the x509 certificate:

E  tcp: ssl_tls13_generic.c709: x509_verify_cert() returned -9984 (-0x2700)
E  tcp: ssl_tls13_generic.c786: ! Certificate verification flags 00000008
E  tcp: ssl_tls13_generic.c832: <= parse certificate
E  tcp: ssl_msg.c5168: => send alert message
E  tcp: ssl_msg.c5169: send alert level=2 message=48
E  tcp: ssl_msg.c2943: => write record
E  tcp: ssl_msg.c3030: output record: msgtype = 21, version = [3:3], msglen = 2
E  tcp: ssl_msg.c3033: dumping 'output record sent to network' (7 bytes)
E  tcp: ssl_msg.c3033: 0000:  15 03 03 00 02 02 30                             ......0
E  tcp: ssl_msg.c2353: => flush output
E  tcp: ssl_msg.c2369: message length: 7, out_left: 7
E  tcp: ssl_msg.c2374: ssl->f_send() returned 7 (-0xfffffff9)
E  tcp: ssl_msg.c2401: <= flush output
E  tcp: ssl_msg.c3080: <= write record
E  tcp: ssl_msg.c5180: <= send alert message
E  tcp: ssl_tls.c4617: <= handshake
E  tls: mbedtls_ssl_handshake returned -0x2700
E  tcp: ssl_msg.c6187: => write close notify
E  tcp: ssl_msg.c6198: <= write close notify
E  tcp: ssl_tls.c5521: => free
E  tcp: ssl_tls.c5583: <= free

@sfan5
Copy link
Member

sfan5 commented May 13, 2024

So I debugged this further and it seems mbedtls 3.6 has broken the ability to not verify the server certificate. 🤦

Edit: There's quite much going wrong here. You can try this branch with fixes: https://github.com/sfan5/ffmpeg/tree/mbed13

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants