Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Worker crash when webRtcTransport.close() is called #1466

Closed
Lin2879 opened this issue Nov 19, 2024 · 4 comments · Fixed by #1469
Closed

Worker crash when webRtcTransport.close() is called #1466

Lin2879 opened this issue Nov 19, 2024 · 4 comments · Fixed by #1469
Labels

Comments

@Lin2879
Copy link

Lin2879 commented Nov 19, 2024

Bug Report

If both pipe and webrtc transport's listen ip are set to 0.0.0.0, and their portRange intersects, mediasoup worker will crash when webRtcTransport.close() is called.

Your environment

  • Operating system: Linux Ubuntu 20.04
  • Node version: v18.20.3
  • npm version: v10.7.0
  • gcc/clang version: gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
  • mediasoup version: v3.14.15 (with pre-build worker executable)
  • mediasoup-client version:

Issue description

If both pipe and webrtc transport's listen ip are set to 0.0.0.0, and their portRange intersects, mediasoup worker would crash when the following steps were taken:
1.Create two mediasoup workers and a router on each worker
2.Create a pipe transport on each worker and connect them
3.Create and close a webrtc transport on either one of the router (the worker on which the router is created crashes)

  • Audio & video transmission were fine before the crash happens
  • The crash is reproducible with or without the connection from a real browser
  • The crash only happens when pipeTransportOptions.listenInfo.ip is set to '0.0.0.0', it never happens when the value is set to process.env.PRIVATE_IP (tested 10~ times)
  • The crash only happens when pipeTransportOptions.listenInfo.portRange and workerSettings.rtcMinPort/rtcMaxPort intersects, it never happens when the ranges were separated (tested 10~ times)
  • The crash never happens when no pipe transport is created before the webrtc transport (tested 20~ times)

Crash logs:

  mediasoup createWorker() +0ms
  mediasoup:Worker constructor() +0ms
  mediasoup:Worker spawning worker process: /home/lyn/workerCrashTest/node_modules/mediasoup/worker/out/Release/mediasoup-worker --logLevel=debug --logTag=info --rtcMinPort=30000 --rtcMaxPort=31000 +0ms
  mediasoup:Channel constructor() +0ms
  mediasoup:Channel [pid:5488] mediasoup-worker::mediasoup_worker_run() | starting mediasoup-worker process [version:3.14.15] +2ms
  mediasoup:Channel [pid:5488] mediasoup-worker::mediasoup_worker_run() | little-endian CPU detected +1ms
  mediasoup:Channel [pid:5488] mediasoup-worker::mediasoup_worker_run() | 64 bits architecture detected +1ms
  mediasoup:Channel [pid:5488] Settings::PrintConfiguration() | <configuration> +0ms
  mediasoup:Channel [pid:5488] Settings::PrintConfiguration() |   logLevel: debug +0ms
  mediasoup:Channel [pid:5488] Settings::PrintConfiguration() |   logTags: info +1ms
  mediasoup:Channel [pid:5488] Settings::PrintConfiguration() |   rtcMinPort: 30000 +0ms
  mediasoup:Channel [pid:5488] Settings::PrintConfiguration() |   rtcMaxPort: 31000 +1ms
  mediasoup:Channel [pid:5488] Settings::PrintConfiguration() |   libwebrtcFieldTrials: WebRTC-Bwe-AlrLimitedBackoff/Enabled/ +0ms
  mediasoup:Channel [pid:5488] Settings::PrintConfiguration() | </configuration> +0ms
  mediasoup:Channel [pid:5488] DepLibUV::PrintVersion() | libuv version: "1.48.0" +1ms
  mediasoup:Channel [pid:5488] DepOpenSSL::operator()() | openssl version: "OpenSSL 3.0.8 7 Feb 2023" +0ms
  mediasoup:Channel [pid:5488] DepOpenSSL::operator()() | openssl CPU info: "CPUINFO: OPENSSL_ia32cap=0xfefaf383ffcbffff:0x400784239c27ab" +0ms
  mediasoup:Channel [pid:5488] DepLibSRTP::ClassInit() | libsrtp version: "libsrtp2 3.0.0" +0ms
  mediasoup:Channel [pid:5488] DepUsrSCTP::ClassInit() | usrsctp +1ms
  mediasoup:Channel [pid:5488] DepLibUring::ClassInit() | io_uring version: "2.5" +0ms
  mediasoup:Channel [pid:5488] DepLibUring::CheckRuntimeSupport() | kernel version: #4355-Microsoft Thu Apr 12 17:37:00 PST 2024 +0ms
  mediasoup:Channel [pid:5488] DepLibUring::CheckRuntimeSupport() | kernel doesn't support io_uring +0ms
  mediasoup:Channel [pid:5488] DepLibUring::ClassInit() | io_uring not enabled +0ms
  mediasoup:Channel [pid:5488] DepLibWebRTC::ClassInit() | libwebrtc field trials: "WebRTC-Bwe-AlrLimitedBackoff/Enabled/" +0ms
  mediasoup:Worker worker process running [pid:5488] +20ms
  mediasoup createWorker() +22ms
  mediasoup:Worker constructor() +0ms
  mediasoup:Worker spawning worker process: /home/lyn/workerCrashTest/node_modules/mediasoup/worker/out/Release/mediasoup-worker --logLevel=debug --logTag=info --rtcMinPort=30000 --rtcMaxPort=31000 +0ms
  mediasoup:Channel constructor() +10ms
  mediasoup:Channel [pid:5490] mediasoup-worker::mediasoup_worker_run() | starting mediasoup-worker process [version:3.14.15] +2ms
  mediasoup:Channel [pid:5490] mediasoup-worker::mediasoup_worker_run() | little-endian CPU detected +1ms
  mediasoup:Channel [pid:5490] mediasoup-worker::mediasoup_worker_run() | 64 bits architecture detected +1ms
  mediasoup:Channel [pid:5490] Settings::PrintConfiguration() | <configuration> +0ms
  mediasoup:Channel [pid:5490] Settings::PrintConfiguration() |   logLevel: debug +0ms
  mediasoup:Channel [pid:5490] Settings::PrintConfiguration() |   logTags: info +0ms
  mediasoup:Channel [pid:5490] Settings::PrintConfiguration() |   rtcMinPort: 30000 +0ms
  mediasoup:Channel [pid:5490] Settings::PrintConfiguration() |   rtcMaxPort: 31000 +0ms
  mediasoup:Channel [pid:5490] Settings::PrintConfiguration() |   libwebrtcFieldTrials: WebRTC-Bwe-AlrLimitedBackoff/Enabled/ +1ms
  mediasoup:Channel [pid:5490] Settings::PrintConfiguration() | </configuration> +0ms
  mediasoup:Channel [pid:5490] DepLibUV::PrintVersion() | libuv version: "1.48.0" +0ms
  mediasoup:Channel [pid:5490] DepOpenSSL::operator()() | openssl version: "OpenSSL 3.0.8 7 Feb 2023" +0ms
  mediasoup:Channel [pid:5490] DepOpenSSL::operator()() | openssl CPU info: "CPUINFO: OPENSSL_ia32cap=0xfefaf383ffcbffff:0x400784239c27ab" +0ms
  mediasoup:Channel [pid:5490] DepLibSRTP::ClassInit() | libsrtp version: "libsrtp2 3.0.0" +0ms
  mediasoup:Channel [pid:5490] DepUsrSCTP::ClassInit() | usrsctp +0ms
  mediasoup:Channel [pid:5490] DepLibUring::ClassInit() | io_uring version: "2.5" +1ms
  mediasoup:Channel [pid:5490] DepLibUring::CheckRuntimeSupport() | kernel version: #4355-Microsoft Thu Apr 12 17:37:00 PST 2024 +0ms
  mediasoup:Channel [pid:5490] DepLibUring::CheckRuntimeSupport() | kernel doesn't support io_uring +0ms
  mediasoup:Channel [pid:5490] DepLibUring::ClassInit() | io_uring not enabled +0ms
  mediasoup:Channel [pid:5490] DepLibWebRTC::ClassInit() | libwebrtc field trials: "WebRTC-Bwe-AlrLimitedBackoff/Enabled/" +0ms
  mediasoup:Worker worker process running [pid:5490] +16ms
  mediasoup:Worker createRouter() +0ms
  mediasoup:Channel request() [method:WORKER_CREATE_ROUTER] +4ms
  mediasoup:Channel request succeeded [method:WORKER_CREATE_ROUTER, id:1] +3ms
  mediasoup:Router constructor() +0ms
  mediasoup:Worker createRouter() +6ms
  mediasoup:Channel request() [method:WORKER_CREATE_ROUTER] +1ms
  mediasoup:Channel request succeeded [method:WORKER_CREATE_ROUTER, id:1] +3ms
  mediasoup:Router constructor() +6ms
  mediasoup:Router createPipeTransport() +0ms
  mediasoup:Channel request() [method:ROUTER_CREATE_PIPETRANSPORT] +3ms
  mediasoup:Channel [pid:5488] RTC::PipeTransport::PipeTransport() | UDP socket buffer sizes [send:163840, recv:163840] +0ms
  mediasoup:Channel request succeeded [method:ROUTER_CREATE_PIPETRANSPORT, id:2] +1ms
  mediasoup:Transport constructor() +0ms
  mediasoup:PipeTransport constructor() +0ms
  mediasoup:Router createPipeTransport() +4ms
  mediasoup:Channel request() [method:ROUTER_CREATE_PIPETRANSPORT] +2ms
  mediasoup:Channel [pid:5490] RTC::PipeTransport::PipeTransport() | UDP socket buffer sizes [send:163840, recv:163840] +1ms
  mediasoup:Channel request succeeded [method:ROUTER_CREATE_PIPETRANSPORT, id:2] +1ms
  mediasoup:Transport constructor() +3ms
  mediasoup:PipeTransport constructor() +3ms
  mediasoup:PipeTransport connect() +1ms
  mediasoup:Channel request() [method:PIPETRANSPORT_CONNECT] +2ms
  mediasoup:Channel request succeeded [method:PIPETRANSPORT_CONNECT, id:3] +1ms
  mediasoup:PipeTransport connect() +2ms
  mediasoup:Channel request() [method:PIPETRANSPORT_CONNECT] +0ms
  mediasoup:Channel request succeeded [method:PIPETRANSPORT_CONNECT, id:3] +2ms
  mediasoup:Router createWebRtcTransport() +7ms
  mediasoup:Channel request() [method:ROUTER_CREATE_WEBRTCTRANSPORT] +1ms
  mediasoup:Channel [pid:5488] RTC::WebRtcTransport::WebRtcTransport() | UDP socket buffer sizes [send:163840, recv:163840] +1ms
  mediasoup:Channel request succeeded [method:ROUTER_CREATE_WEBRTCTRANSPORT, id:4] +3ms
  mediasoup:Transport constructor() +17ms
  mediasoup:WebRtcTransport constructor() +0ms
  mediasoup:Transport close() +1s
  mediasoup:Channel request() [method:ROUTER_CLOSE_TRANSPORT] +1s
  mediasoup:ERROR:Worker (stderr) terminate called after throwing an instance of 'std::out_of_range' undefined +0ms
  mediasoup:ERROR:Worker (stderr)   what():  vector<bool>::_M_range_check: __n (which is 18446744073709550917) >= this->size() (which is 201) undefined +1ms
  mediasoup:Channel Producer Channel ended by the worker process +5ms
  mediasoup:Channel Consumer Channel ended by the worker process +0ms
  mediasoup:ERROR:Worker worker process died unexpectedly [pid:5488, code:null, signal:SIGABRT] undefined +0ms
  mediasoup:Worker died() [error:Error: [pid:5488, code:null, signal:SIGABRT]] +1s
  mediasoup:Channel close() +2ms
  mediasoup:Router workerClosed() +1s
  mediasoup:Transport routerClosed() +10ms
  mediasoup:Worker worker subprocess closed [pid:5488, code:null, signal:SIGABRT] +3ms

Core dump:
Working in progress, had some technical difficulties setting the core dump path on WSL.

Code snippets that reproduces the crash:

const mediasoup = require('mediasoup');

const config = {
    mediasoup:
    {
        workerSettings:
        {
	    rtcMinPort: 30000,
	    rtcMaxPort: 31000,
            logLevel: 'debug',
            logTags: ['info']
        },
        routerOptions:
        {
            mediaCodecs: [
                {
                    kind: 'audio',
                    mimeType: 'audio/opus',
                    clockRate: 48000,
                    channels: 2,
                },
                {
                    kind: 'video',
                    mimeType: 'video/VP8',
                    clockRate: 90000,
                    parameters:
                    {
                        'x-google-start-bitrate': 1000
                    }
                },
                {
                    kind: 'video',
                    mimeType: 'video/VP9',
                    clockRate: 90000,
                    parameters:
                    {
                        'profile-id': 2,
                        'x-google-start-bitrate': 1000
                    }
                },
                {
                    kind: 'video',
                    mimeType: 'video/h264',
                    clockRate: 90000,
                    parameters:
                    {
                        'packetization-mode': 1,
                        'profile-level-id': '42e01f',
                        'level-asymmetry-allowed': 1,
                        'x-google-start-bitrate': 1000
                    }
                },
                {
                    kind: 'video',
                    mimeType: 'video/h264',
                    clockRate: 90000,
                    parameters:
                    {
                        'packetization-mode': 1,
                        'profile-level-id': '4d0032',
                        'level-asymmetry-allowed': 1,
                        'x-google-start-bitrate': 1000
                    }
                }
            ]
        },
        webRtcTransportOptions:
        {
            listenIps:
                [
                    {
                        ip: '0.0.0.0',
                        announcedIp: process.env.PUBLIC_IP
                    }
                ],
            enableUdp: true,
            enableTcp: false,
            preferUdp: false,
            preferTcp: false,
            enableSctp: false,
            iceConsentTimeout: 0,
            initialAvailableOutgoingBitrate: 1000000
        },
        pipeTransportOptions:
        {
            listenInfo:
            {
                protocol: 'udp',
                ip: '0.0.0.0',
                announcedAddress: process.env.PUBLIC_IP,
                portRange:
                {
                    min: 30800,
                    max: 31000
                }
            },
            enableRtx: true,
            enableSctp: false,
            enableSrtp: false,
        }
    }
}

async function testWorkerCrash() {
	const worker0 = await mediasoup.createWorker(config.mediasoup.workerSettings);
	const worker1 = await mediasoup.createWorker(config.mediasoup.workerSettings);

	const router0 = await worker0.createRouter(config.mediasoup.routerOptions);
	const router1 = await worker1.createRouter(config.mediasoup.routerOptions);

	const pipeTransport0 = await router0.createPipeTransport(config.mediasoup.pipeTransportOptions);
	const pipeTransport1 = await router1.createPipeTransport(config.mediasoup.pipeTransportOptions);

	await pipeTransport0.connect({ ip: pipeTransport1.tuple.localIp, port: pipeTransport1.tuple.localPort });
	await pipeTransport1.connect({ ip: pipeTransport0.tuple.localIp, port: pipeTransport0.tuple.localPort });

	const webRtcTransport = await router0.createWebRtcTransport(config.mediasoup.webRtcTransportOptions);

	setTimeout(() => { webRtcTransport.close(); }, 1000);
}

testWorkerCrash();
@Lin2879 Lin2879 added the bug label Nov 19, 2024
@ibc
Copy link
Member

ibc commented Nov 20, 2024

Thanks. I will look this next week. Just one question: if you use portRange also in WebRtcTransport, does it also crash?

@Lin2879
Copy link
Author

Lin2879 commented Nov 21, 2024

Thanks. I will look this next week. Just one question: if you use portRange also in WebRtcTransport, does it also crash?

Yes, replacing listenIps with listenInfos in webRtcTransport options also lead to worker crash. (portRange.min=30000 portRange.max=31000)

@ibc
Copy link
Member

ibc commented Nov 21, 2024

Thanks a lot @Lin2879, this is fixed in PR #1469. Will release a new version on next week if not before.

@ibc
Copy link
Member

ibc commented Nov 21, 2024

BTW I've been able to reproduce the problem in this very simple way:

import * as mediasoup from '../';

test('test crash issue 1466', async () => {
  const worker = await mediasoup.createWorker({ logLevel: 'debug' });

  const router = await worker.createRouter();

  const pipeTransport = await router.createPipeTransport({
    listenInfo: {
      protocol: 'udp',
      ip: '0.0.0.0',
      announcedAddress: '127.0.0.1',
      portRange: {
        min: 30001,
        max: 30003,
      },
    },
  });

  const plainTransport = await router.createPlainTransport({
    listenInfo: {
      protocol: 'udp',
      ip: '0.0.0.0',
      announcedAddress: '127.0.0.1',
      portRange: {
        min: 30000,
        max: 30003,
      },
    },
  });

  plainTransport.close();

  await wait(200);

  pipeTransport.close();

  await wait(200);

  worker.close();

  await wait(200);
}, 2000);

async function wait(timeoutMs: number): Promise<void> {
  await new Promise(resolve => setTimeout(resolve));
}

@ibc ibc closed this as completed in #1469 Nov 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

Successfully merging a pull request may close this issue.

2 participants