Getting lot of exceptions SchedulerError #2232

viveknuna · 2023-12-23T07:41:13Z

viveknuna
Dec 23, 2023

I am using quartz.net 3.5.0 in my .net application. We experience misfires and errors in triggers in the production database so to look over this I have added the below event in my code, I can see a lot of these logs, Can someone tell me what the issue is and how to fix it?

public System.Threading.Tasks.Task SchedulerError(string msg, SchedulerException cause, CancellationToken cancellationToken = default)
{
            _logger.LogInfo($"SchedulerError: {msg} and cause {cause.ToString()}");
}

SchedulerError: An error occurred while scanning for the next trigger to fire. and cause Quartz.JobPersistenceException: Failure setting up connection.
---> Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Operation cancelled by user.
---> System.ComponentModel.Win32Exception (258): Unknown error 258
at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) at Microsoft.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest) at Microsoft.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransaction2005(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest) at Microsoft.Data.SqlClient.SqlInternalConnection.BeginSqlTransaction(IsolationLevel iso, String transactionName, Boolean shouldReconnect) at Microsoft.Data.SqlClient.SqlConnection.BeginTransaction(IsolationLevel iso, String transactionName) at Microsoft.Data.SqlClient.SqlConnection.BeginDbTransaction(IsolationLevel isolationLevel) at Quartz.Impl.AdoJobStore.JobStoreSupport.GetConnection() ClientConnectionId:c0c93e66-469e-41d7-99b8-ac5d433c7382 Error Number:-2,State:0,Class:11 ClientConnectionId before routing:226a90cc-f798-4bf6-a976-dce2c62e02b2 Routing Destination:c92b7097a5f4.tr38393.eastus1-a.worker.database.windows.net,11003 --- End of inner exception stack trace --- at Quartz.Impl.AdoJobStore.JobStoreSupport.GetConnection() at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken)
at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken) at Quartz.Core.QuartzSchedulerThread.Run() [See nested exception: Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. Operation cancelled by user. ---> System.ComponentModel.Win32Exception (258): Unknown error 258 at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)
at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at Microsoft.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)
at Microsoft.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransaction2005(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
at Microsoft.Data.SqlClient.SqlInternalConnection.BeginSqlTransaction(IsolationLevel iso, String transactionName, Boolean shouldReconnect)
at Microsoft.Data.SqlClient.SqlConnection.BeginTransaction(IsolationLevel iso, String transactionName)
at Microsoft.Data.SqlClient.SqlConnection.BeginDbTransaction(IsolationLevel isolationLevel)
at Quartz.Impl.AdoJobStore.JobStoreSupport.GetConnection()
ClientConnectionId:c0c93e66-469e-41d7-99b8-ac5d433c7382
Error Number:-2,State:0,Class:11
ClientConnectionId before routing:226a90cc-f798-4bf6-a976-dce2c62e02b2
Routing Destination:c92b7097a5f4.tr38393.eastus1-a.worker.database.windows.net,11003]

viveknuna · 2023-12-23T07:51:02Z

viveknuna
Dec 23, 2023
Author

This is another type of error.

SchedulerError: An error occurred while scanning for the next trigger to fire. and cause Quartz.JobPersistenceException: Couldn't acquire next trigger: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> System.ComponentModel.Win32Exception (258): Unknown error 258
at Microsoft.Data.SqlClient.SqlCommand.<>c.b__208_0(Task1 result) at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Quartz.Impl.AdoJobStore.StdAdoDelegate.SelectTriggerToAcquire(ConnectionAndTransactionHolder conn, DateTimeOffset noLaterThan, DateTimeOffset noEarlierThan, Int32 maxCount, CancellationToken cancellationToken)
at Quartz.Impl.AdoJobStore.JobStoreSupport.AcquireNextTrigger(ConnectionAndTransactionHolder conn, DateTimeOffset noLaterThan, Int32 maxCount, TimeSpan timeWindow, CancellationToken cancellationToken)
ClientConnectionId:3254decb-75bb-42ab-a071-d80f2f3ebef3
Error Number:-2,State:0,Class:11
ClientConnectionId before routing:b317c3d3-7fcd-4f23-8920-a93820b00302
Routing Destination:c92b7097a5f4.tr38393.eastus1-a.worker.database.windows.net,11003
--- End of inner exception stack trace ---
at Quartz.Impl.AdoJobStore.JobStoreSupport.AcquireNextTrigger(ConnectionAndTransactionHolder conn, DateTimeOffset noLaterThan, Int32 maxCount, TimeSpan timeWindow, CancellationToken cancellationToken)
at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken) at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken)
at Quartz.Core.QuartzSchedulerThread.Run() [See nested exception: Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> System.ComponentModel.Win32Exception (258): Unknown error 258
at Microsoft.Data.SqlClient.SqlCommand.<>c.b__208_0(Task1 result) at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Quartz.Impl.AdoJobStore.StdAdoDelegate.SelectTriggerToAcquire(ConnectionAndTransactionHolder conn, DateTimeOffset noLaterThan, DateTimeOffset noEarlierThan, Int32 maxCount, CancellationToken cancellationToken)
at Quartz.Impl.AdoJobStore.JobStoreSupport.AcquireNextTrigger(ConnectionAndTransactionHolder conn, DateTimeOffset noLaterThan, Int32 maxCount, TimeSpan timeWindow, CancellationToken cancellationToken)
ClientConnectionId:3254decb-75bb-42ab-a071-d80f2f3ebef3
Error Number:-2,State:0,Class:11
ClientConnectionId before routing:b317c3d3-7fcd-4f23-8920-a93820b00302
Routing Destination:c92b7097a5f4.tr38393.eastus1-a.worker.database.windows.net,11003]

0 replies

viveknuna · 2023-12-23T07:52:03Z

viveknuna
Dec 23, 2023
Author

This is one more error.

SchedulerError: An error occurred while firing triggers 'System.Collections.Generic.List1[Quartz.Spi.IOperableTrigger]' and cause Quartz.JobPersistenceException: Couldn't commit ADO.NET transaction. Transaction not connected, or was disconnected ---> System.InvalidOperationException: Transaction not connected, or was disconnected at Quartz.Impl.AdoJobStore.ConnectionAndTransactionHolder.CheckNotZombied() at Quartz.Impl.AdoJobStore.ConnectionAndTransactionHolder.Commit(Boolean openNewTransaction) --- End of inner exception stack trace --- at Quartz.Impl.AdoJobStore.ConnectionAndTransactionHolder.Commit(Boolean openNewTransaction) at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken)
at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken) at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken)
at Quartz.Core.QuartzSchedulerThread.Run() [See nested exception: System.InvalidOperationException: Transaction not connected, or was disconnected
at Quartz.Impl.AdoJobStore.ConnectionAndTransactionHolder.CheckNotZombied()
at Quartz.Impl.AdoJobStore.ConnectionAndTransactionHolder.Commit(Boolean openNewTransaction)]

0 replies

viveknuna · 2023-12-23T07:53:14Z

viveknuna
Dec 23, 2023
Author

One more error.

SchedulerError: An error occurred while scanning for the next trigger to fire. and cause Quartz.JobPersistenceException: Failed to obtain DB connection from data source '11111111-1111-1111-1111-e1eee34e3e4e': Microsoft.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the login process. (provider: TCP Provider, error: 35 - An internal exception was caught)
---> System.IO.IOException: Unable to write data to the transport connection: Connection reset by peer.
---> System.Net.Sockets.SocketException (104): Connection reset by peer
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 count)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Stream.Write(ReadOnlySpan1 buffer) at System.Net.Sockets.NetworkStream.Write(ReadOnlySpan1 buffer)
at Microsoft.Data.SqlClient.SNI.SslOverTdsStream.Write(ReadOnlySpan1 buffer) at Microsoft.Data.SqlClient.SNI.SslOverTdsStream.Write(Byte[] buffer, Int32 offset, Int32 count) at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory1 buffer)
at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory1 buffer) at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count) at Microsoft.Data.SqlClient.SNI.SNITCPHandle.Send(SNIPacket packet) at Microsoft.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e) at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection) at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions)
at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry, SqlConnectionOverrides overrides) at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides) at Quartz.Impl.AdoJobStore.JobStoreSupport.GetConnection() ClientConnectionId:771d6b3a-995a-48b9-8208-c2c398802f5c ---> Microsoft.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the login process. (provider: TCP Provider, error: 35 - An internal exception was caught) ---> System.IO.IOException: Unable to write data to the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException (104): Connection reset by peer at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 count) --- End of inner exception stack trace --- at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 count) at System.IO.Stream.Write(ReadOnlySpan1 buffer)
at System.Net.Sockets.NetworkStream.Write(ReadOnlySpan1 buffer) at Microsoft.Data.SqlClient.SNI.SslOverTdsStream.Write(ReadOnlySpan1 buffer)
at Microsoft.Data.SqlClient.SNI.SslOverTdsStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory1 buffer) at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory1 buffer)
at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at Microsoft.Data.SqlClient.SNI.SNITCPHandle.Send(SNIPacket packet)
at Microsoft.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e)
at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)
at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection) at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions) at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry, SqlConnectionOverrides overrides)
at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
at Quartz.Impl.AdoJobStore.JobStoreSupport.GetConnection()
ClientConnectionId:771d6b3a-995a-48b9-8208-c2c398802f5c
--- End of inner exception stack trace ---
at Quartz.Impl.AdoJobStore.JobStoreSupport.GetConnection()
at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken) at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken)
at Quartz.Core.QuartzSchedulerThread.Run() [See nested exception: Microsoft.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the login process. (provider: TCP Provider, error: 35 - An internal exception was caught)
---> System.IO.IOException: Unable to write data to the transport connection: Connection reset by peer.
---> System.Net.Sockets.SocketException (104): Connection reset by peer
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 count)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.Stream.Write(ReadOnlySpan1 buffer) at System.Net.Sockets.NetworkStream.Write(ReadOnlySpan1 buffer)
at Microsoft.Data.SqlClient.SNI.SslOverTdsStream.Write(ReadOnlySpan1 buffer) at Microsoft.Data.SqlClient.SNI.SslOverTdsStream.Write(Byte[] buffer, Int32 offset, Int32 count) at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory1 buffer)
at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](TIOAdapter writeAdapter, ReadOnlyMemory1 buffer) at System.Net.Security.SslStream.Write(Byte[] buffer, Int32 offset, Int32 count) at Microsoft.Data.SqlClient.SNI.SNITCPHandle.Send(SNIPacket packet) at Microsoft.Data.ProviderBase.DbConnectionPool.CheckPoolBlockingPeriod(Exception e) at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection) at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection) at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions)
at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
at Quartz.Impl.AdoJobStore.JobStoreSupport.GetConnection()
ClientConnectionId:771d6b3a-995a-48b9-8208-c2c398802f5c]

0 replies

viveknuna · 2023-12-23T07:55:37Z

viveknuna
Dec 23, 2023
Author

One more error.

SchedulerError: An error occurred while scanning for the next trigger to fire. and cause Quartz.JobPersistenceException: Couldn't acquire next trigger: Invalid operation. The connection is closed.
---> System.InvalidOperationException: Invalid operation. The connection is closed.
at Microsoft.Data.SqlClient.SqlCommand.<>c.b__208_0(Task1 result) at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Quartz.Impl.AdoJobStore.StdAdoDelegate.SelectTriggerToAcquire(ConnectionAndTransactionHolder conn, DateTimeOffset noLaterThan, DateTimeOffset noEarlierThan, Int32 maxCount, CancellationToken cancellationToken)
at Quartz.Impl.AdoJobStore.JobStoreSupport.AcquireNextTrigger(ConnectionAndTransactionHolder conn, DateTimeOffset noLaterThan, Int32 maxCount, TimeSpan timeWindow, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at Quartz.Impl.AdoJobStore.JobStoreSupport.AcquireNextTrigger(ConnectionAndTransactionHolder conn, DateTimeOffset noLaterThan, Int32 maxCount, TimeSpan timeWindow, CancellationToken cancellationToken)
at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken) at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken)
at Quartz.Core.QuartzSchedulerThread.Run() [See nested exception: System.InvalidOperationException: Invalid operation. The connection is closed.
at Microsoft.Data.SqlClient.SqlCommand.<>c.b__208_0(Task1 result) at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Quartz.Impl.AdoJobStore.StdAdoDelegate.SelectTriggerToAcquire(ConnectionAndTransactionHolder conn, DateTimeOffset noLaterThan, DateTimeOffset noEarlierThan, Int32 maxCount, CancellationToken cancellationToken)
at Quartz.Impl.AdoJobStore.JobStoreSupport.AcquireNextTrigger(ConnectionAndTransactionHolder conn, DateTimeOffset noLaterThan, Int32 maxCount, TimeSpan timeWindow, CancellationToken cancellationToken)]

0 replies

viveknuna · 2023-12-23T07:56:18Z

viveknuna
Dec 23, 2023
Author

One more error.

SchedulerError: An error occurred while scanning for the next trigger to fire. and cause Quartz.JobPersistenceException: Couldn't commit ADO.NET transaction. Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> System.ComponentModel.Win32Exception (258): Unknown error 258
at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
at Microsoft.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at Microsoft.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)
at Microsoft.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransaction2005(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
at Microsoft.Data.SqlClient.SqlInternalTransaction.Commit()
at Microsoft.Data.SqlClient.SqlTransaction.Commit()
at Quartz.Impl.AdoJobStore.ConnectionAndTransactionHolder.Commit(Boolean openNewTransaction)
ClientConnectionId:bf29e212-1379-42bc-a15c-0b4ef6c750a0
Error Number:-2,State:0,Class:11
ClientConnectionId before routing:9fc833af-12ea-4144-8d9d-46da16e8b9e4
Routing Destination:c5edb2d25b76.tr34938.eastus1-a.worker.database.windows.net,11017
--- End of inner exception stack trace ---
at Quartz.Impl.AdoJobStore.ConnectionAndTransactionHolder.Commit(Boolean openNewTransaction)
at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken) at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken)
at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable`1 requestorId, CancellationToken cancellationToken)
at Quartz.Core.QuartzSchedulerThread.Run() [See nested exception: Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> System.ComponentModel.Win32Exception (258): Unknown error 258
at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
at Microsoft.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at Microsoft.Data.SqlClient.TdsParser.TdsExecuteTransactionManagerRequest(Byte[] buffer, TransactionManagerRequestType request, String transactionName, TransactionManagerIsolationLevel isoLevel, Int32 timeout, SqlInternalTransaction transaction, TdsParserStateObject stateObj, Boolean isDelegateControlRequest)
at Microsoft.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransaction2005(TransactionRequest transactionRequest, String transactionName, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
at Microsoft.Data.SqlClient.SqlInternalTransaction.Commit()
at Microsoft.Data.SqlClient.SqlTransaction.Commit()
at Quartz.Impl.AdoJobStore.ConnectionAndTransactionHolder.Commit(Boolean openNewTransaction)
ClientConnectionId:bf29e212-1379-42bc-a15c-0b4ef6c750a0
Error Number:-2,State:0,Class:11
ClientConnectionId before routing:9fc833af-12ea-4144-8d9d-46da16e8b9e4
Routing Destination:c5edb2d25b76.tr34938.eastus1-a.worker.database.windows.net,11017]

0 replies

lahma · 2023-12-23T09:15:16Z

lahma
Dec 23, 2023
Maintainer

Make sure you are using SqlServerDelegate, not plain StdAdoDelegate - I'm seeing StdAdoDelegate in the stack trace
Discuss with your database team about the timeouts, which kind of queries are causing them

7 replies

viveknuna Dec 23, 2023
Author

@lahma
Can having a dedicated database for quartz tables help address these issues? It's a multitenant application where each tenant has its own database, including other business tables and quartz-related tables.

In my opinion, having a single database exclusively for quartz-related tables may resolve all these issues.

lahma Dec 23, 2023
Maintainer

You can try that too. But final decision should be based on metrics and hard numbers, please consult the database people.

viveknuna Dec 23, 2023
Author

Thank you for your time, as always helpful.

viveknuna Jan 6, 2024
Author

@lahma May I know what is the default isolation level for the quartz? I mean when how is the default locking behaviour? Could you please share some documentation links around it? And what are the different supported values which can be confirmed?

I could see quartz.jobStore.txIsolationLevelSerializable default value is false, what does it mean? and what does it mean when I set it as true?

lahma Jan 6, 2024
Maintainer

quartznet/src/Quartz/Impl/AdoJobStore/JobStoreSupport.cs

Lines 331 to 366 in 045ac8c

 protected virtual async ValueTask<ConnectionAndTransactionHolder> GetConnection() 

 { 

 DbConnection conn; 

 DbTransaction tx; 

 try 

 { 

 conn = ConnectionManager.GetConnection(DataSource); 

 await conn.OpenAsync().ConfigureAwait(false); 

 } 

 catch (Exception e) 

 { 

 ThrowHelper.ThrowJobPersistenceException($"Failed to obtain DB connection from data source '{DataSource}': {e}", e); 

 return default; 

 } 

 try 

 { 

 if (TxIsolationLevelSerializable) 

 { 

 tx = conn.BeginTransaction(IsolationLevel.Serializable); 

 } 

 else 

 { 

 // default 

 tx = conn.BeginTransaction(IsolationLevel.ReadCommitted); 

 } 

 } 

 catch (Exception e) 

 { 

 conn.Close(); 

 ThrowHelper.ThrowJobPersistenceException("Failure setting up connection.", e); 

 return default; 

 } 

 return new ConnectionAndTransactionHolder(conn, tx); 

 }

TxIsolationLevelSerializable is false by default.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Getting lot of exceptions SchedulerError #2232

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Getting lot of exceptions SchedulerError #2232

viveknuna Dec 23, 2023

Replies: 6 comments · 7 replies

viveknuna Dec 23, 2023 Author

viveknuna Dec 23, 2023 Author

viveknuna Dec 23, 2023 Author

viveknuna Dec 23, 2023 Author

viveknuna Dec 23, 2023 Author

lahma Dec 23, 2023 Maintainer

viveknuna Dec 23, 2023 Author

lahma Dec 23, 2023 Maintainer

viveknuna Dec 23, 2023 Author

viveknuna Jan 6, 2024 Author

lahma Jan 6, 2024 Maintainer

viveknuna
Dec 23, 2023

Replies: 6 comments 7 replies

viveknuna
Dec 23, 2023
Author

viveknuna
Dec 23, 2023
Author

viveknuna
Dec 23, 2023
Author

viveknuna
Dec 23, 2023
Author

viveknuna
Dec 23, 2023
Author

lahma
Dec 23, 2023
Maintainer

viveknuna Dec 23, 2023
Author

lahma Dec 23, 2023
Maintainer

viveknuna Dec 23, 2023
Author

viveknuna Jan 6, 2024
Author

lahma Jan 6, 2024
Maintainer