[QUERY] Service bus Delivery not on receive link error

[akshit-mehta-rubrik]

Query/Question
Trying to complete messages in service bus using ServiceBusReceiverClient and getting error - "Delivery not on receive link". wanted to know the meaning of error, root cause and resolution steps.

Messages received: 28K (500 at a time and 28K/500 total receiveMessage calls)
Messages to be completed: 28K one by one in a loop without custom delay
Message lock duration: 4 minutes
Message auto lock duration: 400 minutes

Expected result: 28K messages deleted from queue
Actual result: 2K messages were deleted and for remaining messages "Delivery not on receive link" error was thrown while completing. The number varies in each error and cannot be confirmed.

  private val receiver =
    new ServiceBusClientBuilder()
      .credential(
        fullyQualifiedNamespace,
        credentials
      )
    )
      .receiver()
      .maxAutoLockRenewDuration(
        400
      )
      .disableAutoComplete()
      .queueName(queueName)
      .buildClient()

// delete logic
def deleteMessages(messageList: List[QueueMessage]): Int = {
    var messagesDeleted = 0
    messageList.foreach { msg =>
      {
        val result =
          Try(
            receiver.complete(
              msg
                .originalMessage
                .asInstanceOf[ServiceBusReceivedMessage]
            )
          )
        result match {
          case Failure(e) =>
            scribe.error(
              s"Error occurred while completing message: ${msg.id}." + //gives id of message
                s" Error message - ",e
            )
          case Success(_) => messagesDeleted += 1
        }
      }
    }
    
    messagesDeleted
  }

Why is this not a Bug or a feature Request?
A clear explanation of why is this not a bug or a feature request?
currently it's not certain that issue is due to SDK or something is missed from my end.

Setup (please complete the following information if applicable):

• Application runs in Azure Databricks and uses scala 2.12.x. Driver node configuration -
• a) Standard Ds3v2 - 14gb, 4 cores, JVM memory: ~6Gb,
  b) Standard D4asv5 - 16gb, 4 cores, JVM memory: ~8Gb
  for simplicity of recreating this error we're just doing service bus receive and delete and no other operation.
• Approximate queue message size: ~500Bytes as seen from Service bus UI
• Library/Libraries: com-azure:azure-messaging-servicebus:7.13.3, 7.10.1

Debug logs

    com.azure.messaging.servicebus.ServiceBusException: Delivery not on receive link.
    	at com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient.lambda$updateDisposition$52(ServiceBusReceiverAsyncClient.java:1478)
    	at reactor.core.publisher.Mono.lambda$onErrorMap$31(Mono.java:3811)
    	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
    	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:278)
    	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:106)
    	at reactor.core.publisher.Operators.error(Operators.java:198)
    	at reactor.core.publisher.MonoError.subscribe(MonoError.java:53)
    	at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
    	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
    	at reactor.core.publisher.Operators.error(Operators.java:198)
    	at reactor.core.publisher.MonoError.subscribe(MonoError.java:53)
    	at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
    	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:263)
    	at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
    	at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
    	at reactor.core.publisher.Mono.block(Mono.java:1765)
    	at com.azure.messaging.servicebus.ServiceBusReceiverClient.complete(ServiceBusReceiverClient.java:174)

   Caused by: java.lang.IllegalArgumentException: Delivery not on receive link.
    	at com.azure.core.amqp.implementation.handler.ReceiverUnsettledDeliveries.sendDispositionImpl(ReceiverUnsettledDeliveries.java:322)
    	at com.azure.core.amqp.implementation.handler.ReceiverUnsettledDeliveries.sendDisposition(ReceiverUnsettledDeliveries.java:153)
    	at com.azure.messaging.servicebus.implementation.ServiceBusReactorReceiver.updateDisposition(ServiceBusReactorReceiver.java:120)
    	at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.updateDisposition(ServiceBusReceiveLinkProcessor.java:133)
    	at com.azure.messaging.servicebus.ServiceBusAsyncConsumer.updateDisposition(ServiceBusAsyncConsumer.java:69)
    	at com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient.updateDisposition(ServiceBusReceiverAsyncClient.java:1455)
    	at com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient.complete(ServiceBusReceiverAsyncClient.java:421)
    	at com.azure.messaging.servicebus.ServiceBusReceiverClient.complete(ServiceBusReceiverClient.java:174)

24/05/10 15:07:40 DEBUG ServiceBusReceiverAsyncClient: {"az.sdk.message":"Update started.","lockToken":"f4048348-c939-424d-bf8c-393e074132b8","entityPath":"<queueName>","sessionId":null,"dispositionStatus":"COMPLETED"}
24/05/10 15:07:40 WARN ServiceBusReactorReceiver: {"az.sdk.message":"Delivery not found to update disposition.","entityPath":"<queueName>","linkName":"<queueName>_d948be_1715353650225","lockToken":"f4048348-c939-424d-bf8c-393e074132b8"}
24/05/10 15:07:40 ERROR ServiceBusReactorReceiver: {"az.sdk.message":"Delivery not on receive link.","exception":"Delivery not on receive link.","entityPath":"<queueName>","linkName":"<queueName>_d948be_1715353650225"}
java.lang.IllegalArgumentException: Delivery not on receive link.
        at com.azure.core.amqp.implementation.handler.ReceiverUnsettledDeliveries.sendDispositionImpl(ReceiverUnsettledDeliveries.java:322)
        at com.azure.core.amqp.implementation.handler.ReceiverUnsettledDeliveries.sendDisposition(ReceiverUnsettledDeliveries.java:153)
        at com.azure.messaging.servicebus.implementation.ServiceBusReactorReceiver.updateDisposition(ServiceBusReactorReceiver.java:120)
        at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.updateDisposition(ServiceBusReceiveLinkProcessor.java:133)
        at com.azure.messaging.servicebus.ServiceBusAsyncConsumer.updateDisposition(ServiceBusAsyncConsumer.java:69)
        at com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient.updateDisposition(ServiceBusReceiverAsyncClient.java:1455)
        at com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient.complete(ServiceBusReceiverAsyncClient.java:421)
        at com.azure.messaging.servicebus.ServiceBusReceiverClient.complete(ServiceBusReceiverClient.java:174)

Information Checklist

• Query Added
• Setup information Added
• Debug logs

[github-actions]

@anuchandy @conniey @lmolkova

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[anuchandy]

@akshit-mehta-rubrik are you looking into delete all the messages in the queue? We are developing a feature that can delete them in bulk, which is in beta state right now.

[mohjohfox]

Hi, I have the same problem at the moment. I want to mark messages as complete after successfully processing them. To achieve this, I use the ServiceBusReceivedMessageContext and do the complete operation via context.complete(). For some messages, the operation is successful. For others, the error described above occurs. After some analysis, I realized that the messages were delivered again because they were still in the queue. After several redeliveries (mostly once or twice), the complete operation is successful, and the message disappears from the queue like I wanted, which I find a little bit strange. Could it possibly be due to a temporary loss of connection of the ServiceBusProcessorClient or the ServiceBusReceiverClient, which causes problems?

[anuchandy]

Hello @mohjohfox, seeing "delivery not on receive link" means that client’s link (connection) to the Service Bus entity (e.g. queue, topic) was lost by the time application was attempting to complete the message. A message that didn't complete will be redelivered by the service (redelivery attempts will be done for max_delivery_count times configured for the entity).

Could you validate this using "ServiceBusProcessorClient" with the current version 7.17.0 to see if experience improves? could you also make sure "prefetch" is not set explicitly when building "ServiceBusProcessorClient" (prefetch - reference). It would be helpful to see how you’re building the processor client (values of the configurations used in the builder)

[akshit-mehta-rubrik]

@akshit-mehta-rubrik are you looking into delete all the messages in the queue? We are developing a feature that can delete them in bulk, which is in beta state right now.

@anuchandy is the bulk delete option applicable for deleting some messages from queue (as this is my usecase) ?
On a side note, i wanted to understand how the bulk delete would resolve the "Delivery not on receive link" error.

could you also make sure "prefetch" is not set explicitly

For this error i've tried that as well, however that too throwed the "Delivery not on receive link" error

[mohjohfox]

@anuchandy We are currently using version 7.14.2, I will update the version to 7.17.0 and test it again as you suggested.
We are building the ServiceBusProcessorClient as follows:

 ServiceBusClientBuilder()
            .transportType(AmqpTransportType.AMQP)
            .retryOptions(AmqpRetryOptions().apply {
                    maxRetries = 5
                    delay = Duration.ofSeconds(2)
                    maxDelay = Duration.ofMinutes(2)
                    mode = AmqpRetryMode.EXPONENTIAL
                }
            )
            .connectionString("connectionString")
            .processor()
            .queueName("queueName")
            .maxAutoLockRenewDuration(Duration.ofMinutes(150))
            .disableAutoComplete()
            .processMessage(
                messageProcessor()::processMessage
            )
            .processError { context: ServiceBusErrorContext ->
                log.error(
                    "Failed to handle message, ${context.errorSource}",
                    context.exception
                )
            }
            .buildProcessorClient()

[anuchandy]

Hi @akshit-mehta-rubrik, you can find details of batch-delete here https://learn.microsoft.com/en-us/azure/service-bus-messaging/batch-delete The feature is in the process of getting ready for GA.

Unlike receive-complete, the batch-delete optimized for bulk delete, for example, a single batch-delete API call can delete ~4000 messages. In case of receive-complete, deleting 4000 messages means client needs to receive 4000 messages over network then there will be 4000 individual network calls for complete. So, in your use of deleting 28K messages, the batch delete can do it with ~7 network calls, while receive-complete approach takes 56K network calls, which means the risk of batch-delete being affected by network interruption is very low.