Mutex wildcards and "don't run this WF at the same time as mutex X"

[winterrobert]

Is there a way to:

1. Use wildcards or regex in mutexes or achieve the same results using some other means?
2. Specify that one workflow shouldn't run at the same time as another set of workflows (without placing a lock itself)?

We have:

• A: a job that syncs data for a specific tenant that runs from time to time during the day (trigged by the tenants) -- we only want one of these jobs to run at the same time for each individual tenant. Currently we give that job a mutex ingest.{{tenantid}}
• B: a batch job that syncs data for all tenants

We don't want any job like A to run at the same time as B and vice versa.
I image we could achieve that if there exists something like this:
Job A:

synchronization:
    mutexes:
      - name:  `ingest.`{{inputs.parameters.tenantid}}``
   dontRunAtTheSameTimeAs:
      - name:  `ingest`

Job B:

synchronization:
    mutexes:
      - name:  `ingest`
   dontRunAtTheSameTimeAs:
      - name:  `ingest.*`

Is something like this possible to achieve with Argo Workflows? Can it be done by using some smart combination of semaphores and mutexes?

[tooptoop4]

wildcards don't exist

[winterrobert]

Wouldn't this be a pretty common use case that people would have run into before?

• I can't put a common mutex like A_B on both Job A and Job B because then I would prohibit multiple workflows of type A to run at the same time even if they have different tenantids.
• I can't use a common semaphore for both Job A and B as that would prohibit multiple workflows of type A to run at the same time even if they have different tenantids.

I guess another way at framing the concept I'm looking for is "workflow categories".

[tooptoop4]

thinking about what feature would be needed to solve this, i can take inspiration from 'quantitative resource' in ControlM which allows you to set how many slots a job consumes. So i think need a way to say one workflow consumes n slots of semaphoreY instead of only consuming 1 slot of a semaphore.

Then if you know you have 1000 tenants what you would do is:

1. Define semaphore "all_tenants" as 1000
2. Make Job A consumer mutex of {{tenant}} AND 1 slot of all_tenants semaphore
3. Make Job B consume 1000 slots of all_tenants semaphore (currently this part is not possible!) @Joibel