retry() request filter

[Traumeel]

I have a custom Filter which injects additional headers into requests. Occasionally requests fail with 5xx. I would like to have an ability to retry failed requests and return the final successful response to a client.

I want to be able to retry the backend call on specific http status codes. For example I want to retry on status code 502. Example filter setting: retry(3, 502) would retry 3 times if HTTP status code is 502.

[aryszka]

@Traumeel thanks for the input, good idea!

As for the implementation, I believe the filter should 'mark' the request flow to be retried, and the execution itself should rely on the existing loadbalancer fallback mechanism, except that in this case, it could retry the request to the same backend, and not only on dial errors. The filter could also instrument the retry mechanism for the "usual" reliability patterns. @szuecs, @lmineiro, what do you think about this approach?

[aryszka]

this approach would also mean that the filter only instruments the retry, regarless where it is in the filter chain. E.g. the tee() filter calls won't be retried.

[szuecs]

@aryszka they call a backend URL that is not loadbalanced. It's just a single route. Maybe we can have both filters, one that just does the retry and one for the loadblanced routes.

[aryszka]

what i meant was to support all these cases:

1. if not load balanced, but marked as retriable: retry to the same backend
2. if load balanced, and marked as retriable: retry to the next load balanced backend
3. if load balanced, but not marked as retriable: retry to the next load balanced backend, but only when it is a dial error

and I believe this should be implemented by extending the current load balancing mechanism.

[szuecs]

we should work on #556 to make this possible.

[szuecs]

Follow up is #1473