You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Kaspiman opened this issue
Aug 14, 2023
· 1 comment
Assignees
Labels
C-feature-requestCategory: feature requested, but need to be discussedhelp-needed-hardCall for participation: Experience needed to fix: Hard / a lotP-jobsPlugin: Jobs
We in the company actively use JOBS plugin and process a lot of tasks.
Our monolithic application has very different queue handlers with:
a very long locks and small amount of tasks
a very short duration and huge amount of tasks
a regular and predictable number of tasks
an unpredictable task spikes
memory leaks
In the current JOBS implementation, all tasks added into one queue and processed by one pool of workers. Different queue handlers can greatly interfere with others. Problem handlers can take too many workers, requeue tasks too often, and take too long to process.
As a result, the JOBS-cluster performance is unstable and low.
I would like to be able to configure individual pools and assign them individual settings:
the most important queues will always be processed in a separate pools
long tasks will run separately from fast ones
handlers with memory leaks will be restarted more often then non-leaked
other problematic and experimental queues will be processed regardless of stable
The new feature proposes:
configuring many pools
start/destroy pools in parallel
restart certain pool in case of network error
collect metrics from each pool separately (maybe add tag with pool name)
manage each pool via console commands
Here is an example config for solving these problems:
Kaspiman
changed the title
[馃挕 FEATURE REQUEST]: Ability to configure separate pools for different queues with different settings
[馃挕 FEATURE REQUEST]: JOBS: Ability to configure separate pools for different queues with different settings
Aug 14, 2023
Hey @Kaspiman 馃憢馃徎 Thanks for the great idea 馃憤馃徎
I'll plant it (initially) for the 2024.0 milestone (2023.3 is full now).
rustatian
changed the title
[馃挕 FEATURE REQUEST]: JOBS: Ability to configure separate pools for different queues with different settings
[馃挕 FEATURE REQUEST]: Multiple pool support
Aug 14, 2023
C-feature-requestCategory: feature requested, but need to be discussedhelp-needed-hardCall for participation: Experience needed to fix: Hard / a lotP-jobsPlugin: Jobs
Plugin
JOBS
I have an idea!
We in the company actively use JOBS plugin and process a lot of tasks.
Our monolithic application has very different queue handlers with:
In the current JOBS implementation, all tasks added into one queue and processed by one pool of workers. Different queue handlers can greatly interfere with others. Problem handlers can take too many workers, requeue tasks too often, and take too long to process.
As a result, the JOBS-cluster performance is unstable and low.
I would like to be able to configure individual pools and assign them individual settings:
The new feature proposes:
Here is an example config for solving these problems:
The result can be a mechanism for managing many pools in a each plugin. This mechanism can be reused in the future plugins.
Comrades, what do you think of this?
The text was updated successfully, but these errors were encountered: