New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Node 20.3 Crashes all the time when executed inside docker #48444
Comments
Another possible ref electron/rebuild#1085 |
@nodejs/libuv |
"Text file busy" means trying to write a shared object or binary that's already in use. My hunch is that node-gyp has some race condition in reading/writing files that wasn't manifesting (much) when everything still went through the much slower thread pool, whereas io_uring is fast enough to make it much more visible. |
Is there a way to disable ioring using a env variable as a temporary workaround when running node-gyp? |
Yes, set |
|
This is actually worse than I thought. Node doesn't run at all with 20.3 |
|
I can reproduce this, and this is quite critical. |
cc @nodejs/tsc for visibility |
FWIW on two systems I have access to (a Red Hat owned RHEL 8 machine and test-digitalocean-ubuntu1804-docker-x64-1 from the Build infra) root@test-digitalocean-ubuntu1804-docker-x64-1:~# docker run -it node:20.3.0 node
Unable to find image 'node:20.3.0' locally
20.3.0: Pulling from library/node
bba7bb10d5ba: Pull complete
ec2b820b8e87: Pull complete
284f2345db05: Pull complete
fea23129f080: Pull complete
9063cd8e3106: Pull complete
4b4424ee38d8: Pull complete
0b4eb4cbb822: Pull complete
43443b026dcf: Pull complete
Digest: sha256:fc738db1cbb81214be1719436605e9d7d84746e5eaf0629762aeba114aa0c28d
Status: Downloaded newer image for node:20.3.0
Welcome to Node.js v20.3.0.
Type ".help" for more information.
> I can reproduce the assertion failure on an Ubuntu 16.04 host with root@infra-digitalocean-ubuntu1604-x64-1:~# docker run -it node:20.3.0 node
Unable to find image 'node:20.3.0' locally
20.3.0: Pulling from library/node
bba7bb10d5ba: Pull complete
ec2b820b8e87: Pull complete
284f2345db05: Pull complete
fea23129f080: Pull complete
9063cd8e3106: Pull complete
4b4424ee38d8: Pull complete
0b4eb4cbb822: Pull complete
43443b026dcf: Pull complete
Digest: sha256:fc738db1cbb81214be1719436605e9d7d84746e5eaf0629762aeba114aa0c28d
Status: Downloaded newer image for node:20.3.0
node[1]: ../src/node_platform.cc:68:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
1: 0xc8e4a0 node::Abort() [node]
2: 0xc8e51e [node]
3: 0xd0a059 node::WorkerThreadsTaskRunner::WorkerThreadsTaskRunner(int) [node]
4: 0xd0a17c node::NodePlatform::NodePlatform(int, v8::TracingController*, v8::PageAllocator*) [node]
5: 0xc4bbc4 node::V8Platform::Initialize(int) [node]
6: 0xc49408 [node]
7: 0xc497db node::Start(int, char**) [node]
8: 0x7f6e8486218a [/lib/x86_64-linux-gnu/libc.so.6]
9: 0x7f6e84862245 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
10: 0xba9ade _start [node]
root@infra-digitalocean-ubuntu1604-x64-1:~# docker run -it node:20.3.0-bullseye node
Unable to find image 'node:20.3.0-bullseye' locally
20.3.0-bullseye: Pulling from library/node
93c2d578e421: Already exists
c87e6f3487e1: Already exists
65b4d59f9aba: Already exists
d7edca23d42b: Already exists
25c206b29ffe: Already exists
599134452287: Pull complete
bd8a83c4c2aa: Pull complete
d11f4613ae42: Pull complete
Digest: sha256:ceb28814a32b676bf4f6607e036944adbdb6ba7005214134deb657500b26f0d0
Status: Downloaded newer image for node:20.3.0-bullseye
Welcome to Node.js v20.3.0.
Type ".help" for more information.
> Our website build is actually broken running |
FWIW I opened an issue about this in the docker-node repo: nodejs/docker-node#1918 TLDR: this is not a problem with Node.js itself, but with the default base OS used by the Docker image, which was upgraded for v20.3.0. |
bullseye works for me as well |
Now I also get the file busy error:
EDIT: Works with |
So to summarize:
|
Should I split the uring problem into a separate issue? |
Can someone post the result of |
On the Ubuntu 16.04 infra machine I cannot run apt in the bookworm based Another datapoint, adding root@infra-digitalocean-ubuntu1604-x64-1:~# docker run --security-opt=seccomp:unconfined -it node:20.3.0 node
Welcome to Node.js v20.3.0.
Type ".help" for more information.
> |
Right, then I can predict with near 100% certainty what the problem is: docker doesn't know about the newish clone3 system call. Its seccomp filter rejects it with some bogus error and node consequently fails when it tries to start a new thread. This docker seccomp thing is like clockwork, it always pops up when new system calls are starting to see broader use. It's quite possibly fixed in newer versions. |
Updating docker to the latest version fixed it (v24.0.2) for me. A few notes:
Here is what I think we should do:
This seems a future-proof solution while keeping the current functionality available. |
UV_USE_IO_URING is (intentionally) undocumented and going away again so don't do that. |
@bnoordhuis Would you just document this as "if you are hit by this bug, update docker"? |
I think there are two different things here. I'm not sure updating docker will help with the uring problem. Or does it? Please confirm. |
If I'm reading this correctly there are 2 separate issues here.
I think we should try to understand better the 2nd issue before disabling it. |
The |
Does this still occur in 20.4.0? |
It shouldn't, libuv now blacklists known-bad kernels. |
I'll go ahead and close this now that it's fixed. To summarize:
|
still no official docker image for 20.4, weird |
Unfortunately the PR for the official build is still open docker-library/official-images#14997 |
Get the same error with image node:18.17.1 and using "ENV UV_USE_IO_URING=0" does not help. |
18.17.1 comes with libuv 1.44.2 which does not include the io_uring based implementation. Can you open a new issue describing what you're observing? |
I observed the same (similar) issue on v12.22.9 and now I upgraded to v20.8.0 and I still have the same issue. When I run node inside a Docker container, I get (this is on v20.8.0):
Docker is pretty old: |
i have the same, what is the correct solution ? |
Try upgrade to latest container runtime (docker, containerd, etc.) to latest. If nothing newer is picked up from your package manager, consider upgrading manually. |
Experiencing this with |
yarn add bufferutil
) fails withText file busy
The text was updated successfully, but these errors were encountered: