Respect server request to drop decision logs

[ashutosh-narkar]

OPA periodically uploads decision logs to a remote server. If the server responds with a non-2xx status code, OPA will re-queue those decision log event(s) in its buffer and attempt to upload them again after an exponential backoff delay mechanism. Currently there is no way for the server to indicate to OPA to drop those decision log event(s). It would be nice if OPA was able to drop those decision log event(s) along with the exponential backoff if server responded with HTTP 429 status code.

[anderseknert]

Some thoughts I had while reading this: how does the server know that the OPA client needs to drop decisions? "429 Too many requests" would to me indicate that the server currently can't handle the load from potentially many clients, but I'm not sure that automatically translates to a signal for the client to start discarding decision logs. The client could be well-prepared for this with plenty of memory available, in which case I'd expect the client to keep the logs, the exponential backoff to tick up and the client to try again a little later.

Would this behavior be configurable?

[ashutosh-narkar]

HTTP 429 can be used as an indication that the client has sent too many request. Currently OPA will go into exponential backoff when it gets this status from the server. My thought was can we use this specific status code from the server to indicate to OPA to do more that just exponential backoff? WDYT?

[anderseknert]

Like that snarky saying goes: “Poor planning on your part does not constitute an emergency on my part” — i.e. that the server is currently under heavy load, perhaps from a temporary spike, does not automatically mean that a client should need to do something as drastic as dropping decisions if it's capable of keeping them and sending them off at a later point in time. I don't mind the option, but I do think this should be configurable, and not the default behavior.

One concern I have is that there might be machines involved between the client and the decision log server — i.e. gateways or proxies — that could send this status code too. Those kind of scenarios are almost never tested for, so could very well manifest in production making for some truly hard to debug situations.

But as long as this is actively set by the user, at least they've made the choice themselves.

[johanfylling]

Isn't a 429 response more an indication of the server saying "I'm a bit busy right now, please try again later"; rather than "please don't talk to me again, so drop whatever you were trying to tell me"?
Attached to the 429 there might be a Retry-After header, which could also be used to inform our back-off.

Instead of having the server explicitly tell us when to start dropping records, perhaps it's more appropriate to have the client config define under what circumstances it's ok to start dropping data, if ever. I'm thinking the most likely such circumstance might be when the client is starting to run out of memory (or has reached some configured threshold).

If auditing is a concern for the setup, it's probably better for OPA to stop accepting queries, rather than dropping the decision log.

[ashutosh-narkar]

Closing this in favor of #5724. As y'all have commented, OPA automatically dropping logs if it gets a 429 does not seem like the right approach. In #5724, we'll take a holistic look at decision logs plugin.