Django + Gunicorn: How to go about running shared theading.Thread() + subprocess.popen() on startup

[cerahmed]

Hi All,

I have a django backend that, at startup, does two things:

1. Runs a threading.Thread() that calls a function every 10 seconds to read RPi's sensor information and saves it to database.
2. Runs a subprocess.Popen() to call a binary linux program (go2rtc to be specific) that starts the streaming service indefinetely until it's manually stopped.

These calls are performed in each django app's AppConfig def ready() method.

When running django's development server, everything works as expected. However, when running through Gunicorn with multiple workers, things get a bit messy.

I'd like some help in the following:

1. With multiple workers, the def ready() method gets called for each worker, resulting in redundant creation of threads and Popen processes. Using --preload seems to solve the issue. However, the threading.Thread() output to journalctl seems to be blocked; only shows in journalctl after gunicorn is stopped/restared. What is going on here?

2. I need the Thread and Popen instances to be accessible by all workers so that when I send a stop request from the frontend, these threads/processes can halt gracefully. How would I go about sharing these instances between all the workers?

My current gunicorn service is as follows:

[Service]
User=pi
Group=www-data
WorkingDirectory=/home/pi/homecontrol
ExecStart=/home/pi/homecontrol/venv/bin/gunicorn \
          --access-logfile - \
          --workers 3 \
          --bind unix:/run/gunicorn.sock \
          --preload \
          homecontrol.wsgi:application

Appreciate some insights on this.

[pajod]

Sounds like there is nothing in the threads dealing with external sensors/executables that require it to share process/thread context with the HTTP serving, other than managing start/stop of dependent services. Which is one of the key features of systemd, which you already use.

Why have it interfere with your flexibility in controlling HTTP thread/process management, then? Wrap it inside what Django calls "management commands" and have those launched by entirely separate systemd services, declaring dependencies as needed to have systemd manage launching / stopping it.

[cerahmed]

Awsome breakdown for the overall architecture of the issue; really apprecaite it.

One point I'm missing which you slightly touched upon is where (considering best practices) should I store the PID of the external process? Should it be in the DB so that all Gunicorn workers can get it?

[pajod]

should I store the PID

I don't, but have not spent much thought on that. In my case, not a performance issue to query (by name, or to systemd) every time.
Note that retrieving the PID and then separately signalling (potentially since-recycled) that PID is a race condition. If you want to store it, minimize that window in the service crash case by storing the PID in a text file, and instructing systemd to delete that file once the service exits (using PIDFile=).

[cerahmed]

Querying the PID by service name seems to be the best way to go.

If you don't mind helping with the other issue now that the subprocess issue is out of the question.

At startup of my django project, I instantiate a pyhap driver object, which creates homekit accessories and broadcast the service to the network.

Unlike the subprocess binary program, this object must stay within django in order to dynamically exchange information with the front end. Again, in my MVP django project, the driver instance lives in the program as a global variable, which the frontend can access through API endpoints to start/stop/update using the driver's instance methods. Also, the driver object needs to continuously make calls to the django's DB in order to make changes as necessary.

What is the best practice for sharing the driver object between workers? I've read about Celery and Redis, but not sure if that's the easiest way to go about it.

[pajod]

An extra web service that you do want to reconfigure on the python side? That one sounds more like an application for using a threaded worker model in gunicorn, and setting up / tearing down the shared object from the gunicorn server hooks

[cerahmed]

gthreads might actually be the most suitable answer to all my concerns!

I went ahead and read about the different worker types. Ended up specifying only 1 worker, 8 threads, and removed the preload directive; this caused everything to work as expected. I'm quite new to gunicorn so I'm not sure about the side effects this have over running single-threaded multiple workers, but this will do for now.

gunicorn --bind 0.0.0.0:8000 --workers 1 --thread 8 --access-logfile - homecontrol.wsgi:application --graceful-timeout 5

I'm sure that it'd be wiser to adjust my django project for separate django commands for scalability, but that's a problem for another day.

Thanks a lot @pajod for the great support; very much appreciated.