backoff strategy for the built-in bridge

[reubenmiller]

Is your feature request related to a problem? Please describe.

thin-edge.io should be a first good citizen when it comes to connecting to an external MQTT broker. It should implement reasonable backoff algorithms so that if the external MQTT becomes overloaded, it does not cause a "thundering herd" problem but quickly retrying to connect to the broker.

The new in-built MQTT bridge functionality should support a backoff strategy which will avoid such issues.

Describe the solution you'd like

The following backoff strategy is proposed:

• On each retry, the wait interval should be doubled (with jitter), but cap the max result a configurable value (e.g. 10mins)
• A minimum wait interval should be enforced (and configurable) to ensure that clients do not try to reconnect to quickly after a failed attempt. Default value should be 30 seconds
• The wait interval should only be reset to the minimum wait interval after the MQTT connection has been connected for a minimum period (e.g. 5 mins)

Describe alternatives you've considered

Additional context

[reubenmiller]

@didier-wenzek @albinsuresh @jarhodes314, we can discuss the specifics in the ticket.

[didier-wenzek]

The proposed strategy is not directly aligned to what is proposed by the backoff crate, but one should be able to make it.

[reubenmiller]

The proposed strategy is not directly aligned to what is proposed by the backoff crate, but one should be able to make it.

The linked crate looks good, and I'm sure we can wrap it to utilize the reset logic regarding the "stable connection".

[reubenmiller]

See these comments for information about the backoff algorithm (plus jitter).

[gligorisaev]

QA has througly checked the feature and is pleased to confirm that the described is functioning.