Num of events in an issue condition using large time period is flaky

[realkosty]

Environment

SaaS (https://sentry.io/)

Steps to Reproduce

1. 2 alerts (see customer case) are identical except 1 uses event's {attribute="environment"} value {match="equals"} {value="beta"} filter, and the other one has the same environment selected at the top instead.
2. An issue (see customer case) where ALL events match all other conditions in both alerts and are 100% from the same matching environment.

Expected Result

both alerts fire

Actual Result

only the alert with top-level environment filter fires

Product Area

Alerts

Link

No response

DSN

No response

Version

No response

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[getsantry]

Routing to @getsentry/product-owners-alerts for triage ⏲️

[rachrwang]

@ceorourke - can you help take a look at this one? Thanks!

[ceorourke]

I have spent a fair amount of time looking into this without being able to reproduce it.

First I tried to reproduce it with the simplest parts - I made a rule using the environment picker and a rule using the filters for the environment. Both rules fired.

Then I wrote a test for the same so I could trace what may have been happening, but they both fired as well. We then added in the additional filters the customer's rule had and still didn't encounter any problems.

Next we dug through the rule processing pipeline and how we evaluate the environment in the two different ways shown in the rule but can't find any problem. We dug through the logs and tried to figure out where it may have encountered a problem but could not find anything.

Our best guess for now is that because the rules have the "Number of events in an issue is more than 100 in 1w" condition that means it goes through our delayed processing pipeline (a buffer that's flushed every minute) and perhaps the rules were put into different buckets so that when we got the number of events in the last hour it was on two slightly different timestamps and it failed on that. We can't find any problem with the environment options.

I did notice that the rules were created one minute after the other as if the creator already had some problem with a separate rule, is that the case? It seems unlikely that someone would choose to test the two different environment options. Maybe we can look deeper into the original problem rule, if it exists.

[JParisFerrer]

Really appreciate the detailed notes @ceorourke!

I did notice that the rules were created one minute after the other as if the creator already had some problem with a separate rule, is that the case? It seems unlikely that someone would choose to test the two different environment options. Maybe we can look deeper into the original problem rule, if it exists

Yes, this is the case. @realkosty may have linked it in the customer case you have, it will have been invalidated since then and I could provide him the latest link. We observed a false negative in the slightly more complicated alert rule (it mainly includes more actions, and the tag filter uses in instead of eq), which triggered Kosty to suggest this simplified experiment which also had false negative.

Our best guess for now is that because the rules have the "Number of events in an issue is more than 100 in 1w" condition that means it goes through our delayed processing pipeline (a buffer that's flushed every minute) and perhaps the rules were put into different buckets so that when we got the number of events in the last hour it was on two slightly different timestamps and it failed on that

This is an interesting theory, but I'm not sure if it applies. Based on your description here, it sounds like after 1 minute, this bucket mismatch should no longer be a concern? But it appears that these issues that are triggering one alert and not the other have >120 issues, and receive them spread over hours, not seconds. Does that disprove that, or did I misunderstand the delayed pipeline?