Graceful shutdown of jobs

[gitterdonewell]

I'm having an issue with the graceful shutdown of jobs. Most likely I'm not doing this right and would appreciate your help.

I have a .NET 6.0 console app with Quartz.Extensions.Hosting (3.7.0). I have a long running job that does something in a loop and potentially runs for like 3 hours at a time. Inside the loop, I monitor for the cancellationToken.IsCancellationRequested value and if it is true, I log and audit the work up to that point and break the loop to exit the job. Example below:

DI

services.AddQuartz(opt => opt.InterruptJobOnShutdownWithWait = true);

MyJob.cs

[DisallowConcurrrentExecution]
public class MyJob : IJob
{
   public async Task Execute(IJobExecutionContext context)
   {
       while(true)
       {
          if(context.CancellationToken.IsCancellationRequested)
          {
              _logger.Information("Job cancellation requested");
             break;
          }
          await Task.Delay(4000);
       }
   }
}

JobSchedule.cs

.
.
   var job = JobBuilder.Create<MyJob>()
                  .WithIdentity(JobKey)
                  .UsingJobData(jobDataMap)
                  .Build();
   
   var trigger = TriggerBuilder.Create()
                       .WithIdentity($"{jobKey.Name}.trigger", jobKey.Group)
                       .WithCronSchedule(cronExpression, misfirePolicy)
                       .Build()

   await scheduler.ScheduleJob(job, trigger, cancellationToken)
.
.

Program.cs

.
.
using (var host = builder.Build())
{
   var cancellationTokenSource = new CancellationTokenSource();

   var hostLifeTime = host.Services.GetRequiredService<IHostApplicationLifeTime>();
   hostLifeTime.ApplicationStopping.Register(() => ShutdownQuartz(host));

   host.Run();
}
.
.

private void ShutdownQuartz(IHost host, CancellationTokenSource cancellationTokenSource)
{
        var scheduler = host.Services.GetRequiredService<ISchedulerFactory>().GetScheduler(cancellationTokenSource.Token);
        await scheduler.Shutdown(waitForJobsToComplete: true, cancellationTokenSource.Token);
        cancellationTokenSource.Cancel();
}

When I debug the console app in VS 2022 and press CTR+C, the ApplicationStopping event is called which in turn calls the ShutdownQuartz() private method above. As soon as the cancellationTokenSource.Cancel() line is hit, immediately the console app exits and VS stops debugging.

So cancelling the token source seems to shutdown the scheduler along with the jobs immediately without waiting. If I have the break point inside the loop in the job, The context.CancellationToken.IsCancellationRequested is never set to true after cancellationTokenSrouce.Cance() is called. At some point like 5 seconds later, the app crashes taking down the job with it. Seems like 5 seconds is the default .NET host shutdown limit. A single CancellationToken from Program.cs is used across the entire app so cancelling it in the ShutdownQuartz() should set it to true in the entire app but the IsCancellationRequested property inside the job is never set to true. I'm not sure if my setup is right for the graceful shutdown of the jobs. I would appreciate your help. Thanks

[lahma]

Have you tried to configure Quartz to do the interrupt for you?

services.AddQuartz(options => ....  options.InterruptJobsOnShutdown = true);

[gitterdonewell]

Didn't make noticeable difference in the behavior. However looks like there may a timing issues somewhere (quartz.NET or .NET 6). One other thing I found is even if I don't cancel the token like CancellationTokenSource.Cancel(), it gets cancelled when the service about to shutdown. I'm only passing the CancellationToken - not the CancellationTokenSource. So somehow whoever is receiving the token manages to recreate the CanellationTokenSource and call the Cancel() method on it. At least that's my theory.

[lahma]

With InterruptJobsOnShutdown you shouldn't need any manual cancellation tokens or sources. The cancellation should flow from the hosted service shutting down to the scheduler's shutdown.

[gitterdonewell]

It works as it supposed to. When the service is interrupted, the token gets cancelled so the job can do whatever it needs to do before shutting down. My issue was there was an uncaught exception from the refit client I used. The exception immediately took down the job.

So it seems you need to have the InterruptJobsOnShutdownWithWait set to true and call the IScheduler.Shutdown(true, cancellationToken) in the IHostApplicationLifetime.Stopping event handler so no new jobs is started.