Ability to run/queue up workers from web request or another worker

[jjalan]

Often in many projects, two patterns have been identified for which we need to have support.

1. System receives a web request, and wants to queue up a worker to do something. Today, we do npm run <worker>. The problem with this approach is this is not a distributed solution and this would end up running the worker in the same container in which web server is running. When the load is high, system may end up running many workers in the same container causing starvation of resources. We want a distributed system (like Sidekiq in Rails) or something on top of Redis that would easily allow to queue up a worker from a web request.

2. Very similar to above, a worker wants to queue up multiple child workers (ex: system needs to perform some computation for each account). For reasons mentioned above, we need ability to queue up workers that can be run in distributed fashion.