Expose more information about state of worker queue

[matthew-white]

I was watching the pg_stat_activity on the QA server when pm2 was running one instance. I observed the following (copying from Slack):

Even though there was only one pm2 instance, I saw 5 different queries at about the same time from a worker checking for work. I would only expect there to be 1. We spin up a new worker if the existing worker seems to be stuck, but I’m realizing that we never spin down the new worker. So a server could have a growth in the number of workers over time. I’m not sure that’s really the explanation here though, because the 5 queries were all in the same second.

The 5 queries stuck around in pg_stat_activity for a few seconds. I’d expect them to be nearly instantaneous. Their state was idle, so maybe that has something to do with it.

Follow-up idea here: add an option for more verbose logging or other monitoring of the workers. It’d be nice to have more information about the number of workers and how long each of their queries is taking, including the check for work. Right now, it can be hard to gain insight into what’s happening in the worker queue.