Job is firing multiple times

[Ghyath-Serhal]

I am using quartz version 3.6.2 in my application. It uses PostgreSQL as a database engine.
The application is deployed on Kubernetes and it supports horizontal scaling.

Everything in Quartz is working fine and as excepted, however, I have faced a critical issue 2 times, which is the job is being executed endlessly. This issue happened only 2 times during almost 4 months.

I was not able to reproduce the issue on my local machine.

Below are the properties that are used in the application.

["quartz.jobStore.lockHandler.type"] = "Quartz.Impl.AdoJobStore.UpdateLockRowSemaphore, Quartz",
["quartz.jobStore.type"] = "Quartz.Impl.AdoJobStore.JobStoreTX, Quartz",
["quartz.jobStore.driverDelegateType"] = "Quartz.Impl.AdoJobStore.StdAdoDelegate, Quartz",

["quartz.jobStore.tablePrefix"] = "QRTZ_",
["quartz.jobStore.dataSource"] = "myDS",
["quartz.jobStore.useProperties"] = "true",
["quartz.jobStore.misfireThreshold"] = "60000",
["quartz.jobStore.clustered"] = "true",
["quartz.jobStore.clusterCheckinInterval"] = "20000",
["org.quartz.jobStore.driverDelegateClass"] = "org.quartz.impl.jdbcjobstore.PostgreSQLDelegate",
["org.quartz.jobStore.class"] = "org.quartz.impl.jdbcjobstore.JobStoreTX",

["quartz.dataSource.myDS.connectionString"] = "ConnectionString",
["quartz.dataSource.myDS.provider"] = "Npgsql",
["quartz.serializer.type"] = "json",

["quartz.scheduler.instanceId"] = "AUTO",
["quartz.scheduler.instanceName"] = "InstanceName",

["quartz.threadPool.threadCount"] = "2"

I am using the TriggerJob functionality that creates a simple trigger.

IJobDetail job = JobBuilder.Create("Fullyqualifiedtypename")
                  .WithIdentity(record.Id.ToString(), tenantId)
                  .UsingJobData("BatchJobId", record.Id.ToString())
                  .StoreDurably(true)
                  .Build();

await _scheduler.AddJob(jobDetail, true);
await _scheduler.TriggerJob(jobDetail.Key);

The TriggerJob is executed from a web application, I have a background service that picks these jobs and executes them.

I am setting the LogProvider to a custom ILogProvider that logs all the levels>= Quartz.Logging.LogLevel.Info

I have reviewed all the logs, bug I was not able to identify the root cause.
I appreciate if you can help.

[lahma]

Can't say what might be the root cause, but your configuration looks weird.

["org.quartz.jobStore.driverDelegateClass"] = "org.quartz.impl.jdbcjobstore.PostgreSQLDelegate",
["org.quartz.jobStore.class"] = "org.quartz.impl.jdbcjobstore.JobStoreTX",

You definitely want to use PostgreSQL specific driver, but this looks like Java configuration (you are now probably using the default StdAdoDelegate). You might be better of by using SchedulerBuilder API with compile-time safety and setup.

[Ghyath-Serhal]

Thank you @lahma
I changed the code to use the SchedulerBuilder, I have realized the below change when quartz started

Old:
Using custom data access locking (synchronization): Quartz.Impl.AdoJobStore.UpdateLockRowSemaphore

New:
Using db table-based data access locking (synchronization)

Do you think this might cause such problems?

Another question please, is there any think I can do to help me investigating such problem, such as special tracing or logging in Quartz.

If I include the Debug and Trace levels, it logs all the queries that are being executed. I am not sure if this can help me better understand the problem.

[Ghyath-Serhal]

Hello @lahma

The issue appeared again on a deployed machine.
Clustered, 1 service (no horizontal scaling), 2 threads count.

While running, i checked the quartz tables. The tables qrtz_simple_triggers, qrtz_triggers and qrtz_fired_triggers are empty.
However, the job repeating and repeating. After restarting the service, everything back to normal and the job is not repeating any more.

I am not able to reproduce on my machine.

[lahma]

You need to add logging, add logging from from job's execute. Make sure you don't do anything silly in job's constructor etc.