RuntimeError: can't start new thread

[Ornitier]

Describe the bug
Since last update, the container keep restarting.

Version
Last one : 0.45.7.3

To Reproduce
Do not know, maybe one of the URL is blocking with the last update.

Expected behavior
To run 👍

Logs
#:docker logs
2023-11-21T10:29:50.073892715Z Process Process-1:
2023-11-21T10:29:50.077509735Z Traceback (most recent call last):
2023-11-21T10:29:50.078594835Z File "/usr/local/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
2023-11-21T10:29:50.078652003Z self.run()
2023-11-21T10:29:50.078802075Z File "/usr/local/lib/python3.11/multiprocessing/process.py", line 108, in run
2023-11-21T10:29:50.078822216Z self._target(*self._args, **self._kwargs)
2023-11-21T10:29:50.078943151Z File "/app/changedetectionio/changedetection.py", line 96, in main
2023-11-21T10:29:50.078955261Z datastore = store.ChangeDetectionStore(datastore_path=app_config['datastore_path'], version_tag=version)
2023-11-21T10:29:50.078960663Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-11-21T10:29:50.079141172Z File "/app/changedetectionio/store.py", line 139, in init
2023-11-21T10:29:50.079153929Z save_data_thread = threading.Thread(target=self.save_datastore).start()
2023-11-21T10:29:50.079159234Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-11-21T10:29:50.079404494Z File "/usr/local/lib/python3.11/threading.py", line 964, in start
2023-11-21T10:29:50.079417311Z _start_new_thread(self._bootstrap, ())
2023-11-21T10:29:50.079584604Z RuntimeError: can't start new thread

Additional context
Some watched URLs:
Watching: 1adb4869-6658-4d75-8511-95e5b5b5d9eb https://nginx.org/en/CHANGES
Watching: da11b800-a53d-42d0-b23a-44cddaed2dc0 https://solr.apache.org/security.html
Watching: d8323d88-f3ab-4a5a-bf29-a3e0909c8254 https://www.synology.com/fr-fr/releaseNote/DSM?model=DS415play
Watching: d9f71212-8c6e-42a4-8543-3c0b8ecbad5f https://www.drupal.org/project/search_api_solr

[dgtlmoon]

Quick google says

The "can't start new thread" error almost certainly due to the fact that you have already have too many threads running within your python process, and due to a resource limit of some kind the request to create a new thread is refused.

How many 'fetch workers' do you have set? btw, greetings fellow drupal'er :)

There are 10's of thousands of installs of this app, maybe more, this is the first error of this type ever, so it's got to be something with the setup there, i'm wondering if you have a lot of 'fetch workers' or other setting, can you paste your docker-compose?

https://stackoverflow.com/questions/70087344/python-in-docker-runtimeerror-cant-start-new-thread

what version of docker exactly?

Also you dont say how you're running it, pip docker etcet

[jgupta]

Maybe not related to this directly. We had to raise server limits in sysctl.conf for our production deployment.

kernel.threads-max
kernel.pid_max
vm.max_map_count
fs.file-max
fs.nr_open
fs.inotify.max_user_watches
fs.inotify.max_user_instances

Maybe looking at servers logs will give you clue about the bottleneck.

[dgtlmoon]

@jgupta did you get exactly the same error message "RuntimeError: can't start new thread" or are you mixing up information here?

[Ornitier]

Hi,

Thank you for all your feedbacks.

I can not understand too as there is no change on this server and there is only changedetection running on it.

Docker version

# docker -v
Docker version 18.09.1, build 4c52b90

# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 10 (buster)
Release:	10
Codename:	buster

docker-compose.yml

version: '2'
services:
  changedetection:
    image: ghcr.io/dgtlmoon/changedetection.io
    container_name: changedetection
    hostname: changedetection
    #environment:
    #  - USE_X_SETTINGS=1
    #  - BASE_URL=https://my.server.com:5001/app
    volumes:
      - changedetection-data:/datastore
    ports:
      - 5000:5000
    #networks:
    #  changenet:
    #    aliases:
    #      - changedetection-app
    restart: unless-stopped

  #browser-chrome:
  #  hostname: browser-chrome
  #  image: seleniarm/standalone-chromium:4.0.0-20211213
  #  volumes:
  #    - /dev/shm:/dev/shm
  #  restart: unless-stopped

volumes:
  changedetection-data:

#networks:
#  changenet:

No more clues in logs as suggested by @jgupta

Might be too old server, I should upgrade it.

[dgtlmoon]

What are you running on exactly? the last big update had to update the container and libraries because the older versions no longer received security updates

[Ornitier]

It is running on a Public Cloud OVH server (in France).
I have around 20 URL to check, nothing special and no Headless Chrome use. Just HTML check.

[dgtlmoon]

@Ornitier i repeat, "What are you running on exactly?" what CPU exactly? or it doesnt tell you?

[Ornitier]

x86_64 virtualized Intel processor

[Ornitier]

@dgtlmoon Hi, just to let you know that the upgrade from Debian 10 to 11 solved the problem. It seems that one component of the last update may be incompatible with one of Debian 10.

Docker from Debian 11

# docker -v
Docker version 20.10.5+dfsg1, build 55c4c88

Thank you for your help!

[dgtlmoon]

thanks!

[dgtlmoon]

https://github.com/dgtlmoon/changedetection.io/wiki/Synology-NAS-setup