DisallowConcurrentExecution in a cluster

[cn-byn]

Hi,

assuming a cluster setup with two schedulers with the same instance name and different instance ids, is the DisallowConcurrentExecution attribute supposed to prevent the concurrent execution of a job across the whole cluster (i.e., both schedulers) or only "inside" one of the nodes?

So if I add two triggers with exactly the same job key, is the execution of the second trigger supposed to be delayed until the first one has finished in this scenario?

Thanks in advance.

[lahma]

Yes, Quartz will block other triggers which target the same job.

quartznet/src/Quartz/Impl/AdoJobStore/JobStoreSupport.cs

Lines 2908 to 2922 in e9d6738

	if (job.ConcurrentExecutionDisallowed)
	{
	state2 = StateBlocked;
	force = false;
	try
	{
	await Delegate.UpdateTriggerStatesForJobFromOtherState(conn, job.Key, StateBlocked, StateWaiting, cancellationToken).ConfigureAwait(false);
	await Delegate.UpdateTriggerStatesForJobFromOtherState(conn, job.Key, StateBlocked, StateAcquired, cancellationToken).ConfigureAwait(false);
	await Delegate.UpdateTriggerStatesForJobFromOtherState(conn, job.Key, StatePausedBlocked, StatePaused, cancellationToken).ConfigureAwait(false);
	}
	catch (Exception e)
	{
	throw new JobPersistenceException("Couldn't update states of blocked triggers: " + e.Message, e);
	}
	}

[cn-byn]

Thanks for the quick response.

However, I'm consistenly seeing that a job with the same job key is being executed simultaneously on two different instances (same scheduler). Both instances are running on the same computer, so there isn't an issue with clocks.

For this test, I'm using the following settings.

{
    "quartz.scheduler.instanceId": "...", // "chndefault" and "ServiceConsole_NB043"
    "quartz.scheduler.idleWaitTime": "10000",
    "quartz.scheduler.batchTriggerAcquisitionMaxCount": "1",
    "quartz.threadPool.threadCount": "1",
    "quartz.threadPool.type": "Quartz.Simpl.DefaultThreadPool, Quartz",
    "quartz.jobStore.type": "Quartz.Impl.AdoJobStore.JobStoreTX, Quartz",
    "quartz.jobStore.acquireTriggersWithinLock": "true",
    "quartz.jobStore.misfireThreshold": "60000",
    "quartz.jobStore.driverDelegateType": "Quartz.Impl.AdoJobStore.SqlServerDelegate, Quartz",
    "quartz.jobStore.clustered": "true",
    "quartz.jobStore.dataSource": "db",
    "quartz.dataSource.db.provider": "SqlServer",
    "quartz.dataSource.db.connectionString": "...",
    "quartz.serializer.type": "json"
  }

I scheduled 4 simple triggers all with the same job key and with the same time.
For testing purposes, the Job itself just awaits Task.Delay for some time.

Executing the following SQL queries...

select * from QRTZ_TRIGGERS;
select * from QRTZ_FIRED_TRIGGERS;

... yields this:

IS_NONCONCURRENT is 1 and the job key is identical, but there are 2 jobs running at the same time on different instances.

Am I missing something?

[lahma]

So if we don't look at the implementation detail (database tables), you can see that the two console apps are running the job simultaneously? I'm just trying to avoid going into a lot of debugging and hair-pulling here...

[cn-byn]

Unfortunately, yes.

Here's my TestJob:

[DisallowConcurrentExecution]
public class TestJob : IJob
{
    public async Task Execute(IJobExecutionContext context)
    {
        Write("BEGIN TestJob.Execute", context);

        for (int i = 0; i < 6; i++)
        {
            Write("Delay 10 s", context);
            await Task.Delay(10000);
        }

        Write("END TestJob.Execute", context);
    }

    private void Write(string msg, IJobExecutionContext context)
    {
        var now = DateTime.UtcNow.ToString("HH:mm:ss.fffffff");
        var scheduler = context.Scheduler.SchedulerName;
        var instance = context.Scheduler.SchedulerInstanceId;
        Debug.WriteLine($"{now} [{scheduler}|{instance}] {msg}");
    }
}

With two instances running (both scheduler "Default"), scheduling two identical triggers one minute in the future produces the following debug output:

11:18:00.5869851 [Default|chndefault] BEGIN TestJob.Execute
11:18:00.5908448 [Default|chndefault] Delay 10 s
11:18:10.5999037 [Default|chndefault] Delay 10 s
11:18:20.6238008 [Default|chndefault] Delay 10 s
11:18:30.6408646 [Default|chndefault] Delay 10 s
11:18:40.6502410 [Default|chndefault] Delay 10 s
11:18:50.6564777 [Default|chndefault] Delay 10 s
11:19:00.6646496 [Default|chndefault] END TestJob.Execute

11:18:02.4044645 [Default|ServiceConsole_NB043] BEGIN TestJob.Execute
11:18:02.4080580 [Default|ServiceConsole_NB043] Delay 10 s
11:18:12.4186936 [Default|ServiceConsole_NB043] Delay 10 s
11:18:22.4274590 [Default|ServiceConsole_NB043] Delay 10 s
11:18:32.4464402 [Default|ServiceConsole_NB043] Delay 10 s
11:18:42.4682424 [Default|ServiceConsole_NB043] Delay 10 s
11:18:52.4892730 [Default|ServiceConsole_NB043] Delay 10 s
11:19:02.4957294 [Default|ServiceConsole_NB043] END TestJob.Execute

Both instances acquire one trigger each, which is fine.
But somehow the functionality that sets other ACQUIRED triggers to BLOCKED before they get executed doesn't work for us (the call to UpdateTriggerStatesForJobFromOtherState in TriggerFired). The second instance still executes its previously acquired trigger.

Maybe it has something to do with isolation levels, but setting txIsolationLevelSerializable to true doesn't help either.

Also, a trigger acquired by the first instance isn't acquired by the second one, so the part that sets a single trigger to ACQUIRED (call to UpdateTriggerStateFromOtherStateWithNextFireTime in AcquireNextTrigger) works, even though the transaction of the second instance may be already waiting for the lock. Once it has the lock, it correctly sees that one trigger has been acquired and acquires the other one.

I think acquiring a non-concurrent job's trigger should immediately block all the others (in the first transaction) to prevent them from being acquired in the first place instead of setting them to BLOCKED in the second transaction. However, that still doesn't explain why the current implementation doesn't work for us.

[Gavinluo]

i have the same question, DisallowConcurrentExecution not work right some time . Quartz 3.3.3

[aleksandar-angelov]

We came up with the same scenario where the DisallowConcurrentExecution was not working. The solution was to create the trigger with the same name on every instance., specifying a Triggerkey. If you don't specify a trigger name, Quartz will automatically generate an ID and even though the trigger is the same, the name will be different in every instance.

Here is an example, Quartz version 3.8.1

[cn-byn]

Thanks, but that's not really a solution since now you can only have a single trigger (with that specific TriggerKey).

DisallowConcurrentExecution is meant to work w.r.t. the JobKey, though. So the number of triggers and their names should not matter.

But maybe that's a hint why the unexpected behavior occurs?

[aleksandar-angelov]

Basically, in your scenario, you have the same trigger that will occur at some time in both instances, so both instances will need to take care of the same trigger and the same job. IMO, there is no need to have multiple triggers that have the same schedule and same job. You need to have one trigger and one of the cluster nodes, randomly chosen, will execute the trigger with the job and will prevent a concurrent execution if the next firing schedule came from another cluster node.

[cn-byn]

IMO, there is no need to have multiple triggers that have the same schedule and same job.

But there is this need in my case. I have several simple triggers (i.e., non-recurring triggers) and those should not cause the job to run concurrently if DisallowConcurrentExecution is set.

Besides, this also happens with multiple triggers with different schedules.