PIP 94: Support level increment delay for ReconsumerLater interface

• Status: Draft
• Author: Xiaolong Ran
• Pull request:
• Mailing list discussion:
• Release:

The purpose of this proposal is mainly to add ReconsumerLater on the consumer side to retry in an incremental level

Motivation

At present, ReconsumrLater only supports specifying a specific delay time for distribution processing. The usage is as follows:

while (true) {
     Message<String> msg = consumer.receive();

     try {
          // Process message...

          consumer.acknowledge(msg);
     } catch (Throwable t) {
          log.warn("Failed to process message");
          consumer.reconsumeLater(msg, 1000 , TimeUnit.MILLISECONDS);
     }
 }

Its implementation principle is to use Pulsar's built-in delay message to pass in the specified time as the parameter of deliverAfter(), and then push the message to the consumer side again after the time arrives.

This is a good idea, which allows users to flexibly define their own delay time in a specific scenario. But assuming that the message is not processed correctly within the time specified by the user, the behavior of ReconsumerLater has ended at this time. Whether we can consider adding a retry scheme according to the time level. Then when the first specified time range is not processed correctly, ReconsumerLater() can automatically retry according to the time level until the user correctly processes the specific message.

Implementation

As mentioned above, if we can here follow a certain delay level from low to high and allow it to automatically retry, it is a more user-friendly way. For example, we can define the following delay level:

MESSAGE_DELAYLEVEL = "1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h"

In this PIP, we mainly introduce two new API interfaces to users:

1. Specify the delay level

reconsumeLater(Message<?> message, int delayLevel)

This implementation method is consistent with the current reconsumeLater interface, but instead of specifying the delay level, specify the specific delay time. For example, level 1 corresponds to 1s, and level 3 corresponds to 10s.

2. Retry with increasing level

reconsumeLater(Message<?> message)

Different from the above two, it is a back-off retry, that is, the retry interval after the first failure is 1 second, and the retry interval after the second failure is 5 seconds, and so on, the more the number of times, the longer the interval.

This kind of retry mechanism often has more practical applications in business scenarios. If the consumption fails, the general service will not be restored immediately. It is more reasonable to use this gradual retry method.

Compatibility

The current proposal will not cause damage to compatibility. It exposes two new API interfaces based on the original API for users to use, so there will be no compatibility-related issues here.