Client-provided timestamps that are too far in the past are rewritten with the current timestamp; confusion ensues

[tkaemming]

Timestamps that are too far in the past (30 days) are discarded and rewritten as the current timestamp. This can lead to confusing effects when some clients that may legitimately send very old events (mobile SDKs with offline buffering, for example) have their timestamps rewritten.

This can lead to the incorrect conclusion that an issue that previously existed (especially ones that are not isolated to a specific version, such as a client/server incompatibility that was fixed on the server side, DNS routing issues, etc.) and was assumed to be or explicitly marked as resolved has mysteriously started to occur again in the present.

[tkaemming]

One approach here that might make sense (this came up in discussion with @mattrobenolt and @alex-hofsteede) is to stop rewriting old timestamps, set a hard cap at some point after which the event is rejected as invalid, and also allow the user to optionally define their own cap as an inbound filter if they'd like to be more restrictive

[vitaliy-ostapchuk93]

Any chance this can be disabled in the settings? We currently running a trial to switch to sentry and have to move the dumps we already got, which can be really confusing with the updated timestamps.

[BYK]

I think @tkaemming's cap idea is pretty good compared to rewriting as the current timestamp. How do we move forward with that solution? Who else should be involved in making this decision?

[liewegas]

We are also encountering this problem. Any chance of getting a fix or workaround?

[fpacifici]

@untitaker Is Relay resetting the timestamp to the current timestamp during the normalization phase? Snuba is not doing it.

[untitaker]

Yes, Relay is doing it as did the old Python store. What is the behavior we want? Right now there should be a clear error message at the top indicating that the timestamp is rewritten. It does not happen silently.

[getsantry]

Routing to @getsentry/product-owners-ingestion-and-filtering for triage ⏲️