Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[diagramsnet container]: Prevent Chromium zombie processes from stacking #1826

Open
al-cheb opened this issue Feb 12, 2025 · 1 comment
Open

Comments

@al-cheb
Copy link

al-cheb commented Feb 12, 2025

Issue:

When specifying low memory limits for a diagramsnet container in Kubernetes, chromium processes could remain in a zombie state if we send a lot of parallel requests.

The diagramsnet service just stopped working with following error, but the container is still alive.

Error 400: : An error occurred while converting the diagram: connect ECONNREFUSED 127.0.0.1:40619

E.g:

$  seq 10 | xargs -P 10 -I -- curl -k https://kroki.local/diagramsnet/svg/eNpLyUwvSizIUHBXqPZIzcnJ17ULzy_KSanlAgB1EAjQ
Error 400: Error: An error occurred while converting the diagram: Navigating frame was detached
Error: Navigating frame was detached
    at #onFrameDetached (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/LifecycleWatcher.js:100:47)
    at file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:36:7
    at Array.map (<anonymous>)
    at Object.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:35:20)
    at CdpFrame.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/common/EventEmitter.js:77:23)
    at #removeFramesRecursively (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:448:15)
    at #onClientDisconnect (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:91:42)Error 400: : An error occurred while converting the diagram: connect ECONNREFUSED 127.0.0.1:43847
Error 400: Error: An error occurred while converting the diagram: Navigating frame was detached
Error: Navigating frame was detached
    at #onFrameDetached (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/LifecycleWatcher.js:100:47)
    at file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:36:7
    at Array.map (<anonymous>)
    at Object.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:35:20)
    at CdpFrame.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/common/EventEmitter.js:77:23)
    at #removeFramesRecursively (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:448:15)
    at #onClientDisconnect (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:91:42)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at listOnTimeout (node:internal/timers:540:9)
    at process.processTimers (node:internal/timers:514:7)Error 400: Error: An error occurred while converting the diagram: Navigating frame was detached
Error: Navigating frame was detached
    at #onFrameDetached (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/LifecycleWatcher.js:100:47)
    at file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:36:7
    at Array.map (<anonymous>)
    at Object.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:35:20)
    at CdpFrame.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/common/EventEmitter.js:77:23)
    at #removeFramesRecursively (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:448:15)
    at #onClientDisconnect (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:91:42)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at listOnTimeout (node:internal/timers:540:9)
    at process.processTimers (node:internal/timers:514:7)Error 400: Error: An error occurred while converting the diagram: Navigating frame was detached
Error: Navigating frame was detached
    at #onFrameDetached (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/LifecycleWatcher.js:100:47)
    at file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:36:7
    at Array.map (<anonymous>)
    at Object.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:35:20)
    at CdpFrame.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/common/EventEmitter.js:77:23)
    at #removeFramesRecursively (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:448:15)
    at #onClientDisconnect (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:91:42)Error 400: : An error occurred while converting the diagram: connect ECONNREFUSED 127.0.0.1:43847
Error 400: : An error occurred while converting the diagram: connect ECONNREFUSED 127.0.0.1:43847
Error 400: : An error occurred while converting the diagram: connect ECONNREFUSED 127.0.0.1:43847
Error 400: : An error occurred while converting the diagram: connect ECONNREFUSED 127.0.0.1:43847
Error 400: Error: An error occurred while converting the diagram: Navigating frame was detached
Error: Navigating frame was detached
    at #onFrameDetached (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/LifecycleWatcher.js:100:47)
    at file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:36:7
    at Array.map (<anonymous>)
    at Object.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/third_party/mitt/mitt.js:35:20)
    at CdpFrame.emit (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/common/EventEmitter.js:77:23)
    at #removeFramesRecursively (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:448:15)
    at #onClientDisconnect (file:///usr/local/kroki/node_modules/puppeteer-core/lib/esm/puppeteer/cdp/FrameManager.js:91:42)

Normal state:

/usr/local/kroki $ ps aux
PID   USER     TIME  COMMAND
    1 kroki     0:00 node src/index.js
   17 kroki     0:00 {chromium} /usr/lib/chromium/chrome --allow-pre-commit-input --disable-background-networking --disable-background-timer-throttl
   19 kroki     0:00 /usr/lib/chromium/chromium --type=zygote --no-zygote-sandbox --no-sandbox --headless --headless
   20 kroki     0:00 /usr/lib/chromium/chromium --type=zygote --no-sandbox --headless --headless
   36 kroki     0:00 /usr/lib/chromium/chromium --type=gpu-process --no-sandbox --disable-dev-shm-usage --disable-breakpad --headless --ozone-platfo
   42 kroki     0:00 /usr/lib/chromium/chromium --type=utility --utility-sub-type=network.mojom.NetworkService --lang=en-US --service-sandbox-type=n
   46 kroki     0:00 /usr/lib/chromium/chromium --type=renderer --headless --first-renderer-process --no-sandbox --disable-dev-shm-usage --disable-b

Hanging Chromium processes in the zombie state:

/usr/local/kroki $ ps
PID   USER     TIME  COMMAND
    1 kroki     0:00 node src/index.js
   19 kroki     0:00 [chromium]
   20 kroki     0:00 [chromium]
   36 kroki     0:00 [chromium]
   42 kroki     0:00 [chromium]
   46 kroki     0:00 [chromium]
   76 kroki     0:00 sh -c clear; (bash || ash || sh)
   83 kroki     0:00 ash
  113 kroki     0:00 sh
  157 kroki     0:00 [chromium]
  162 kroki     0:00 [chromium]
  168 kroki     0:00 [chromium]
  174 kroki     0:00 [chromium]
  181 kroki     0:00 [chromium]
  243 kroki     0:00 ps

What did you expect to see?
The Chromium processes should be properly handled.

@ggrossetie
Copy link
Member

We did something similar in Mermaid: d2ee78b

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants