Dashboard widgets timing out after refreshing

[narsaynorath]

Environment

SaaS (https://sentry.io/)

Steps to Reproduce

1. Navigate to a dashboard (best reproduced using a dashboard with multiple widgets and a larger time range)
2. Refresh multiple times, waiting for the dashboard to trigger some requests
3. Alternatively, make a few edits e.g. changing the time range multiple times, or going back and forth between the widget builder and dashboard

Expected Result

The dashboard widgets load successfully.

Actual Result

After a few refreshes, the dashboard widgets will start to timeout and render the following banner text in the widget:

Query timeout. Please try again. If the problem persists try a smaller date range or fewer projects. Also consider a filter on the transaction field if you're filtering performance data.

Product Area

Dashboards

Link

No response

DSN

No response

Version

No response

[getsantry]

Routing to @getsentry/product-owners-dashboards for triage ⏲️

[narsaynorath]

The reason these timeouts get hit is because we enforce limits on how much data a single organization-project combination can request data in a 10 minute window. There are a few problems that can be tackled here.

1. Better error messaging for this issue
2. Investigate whether the queries are necessarily taking up as much resources as they should be and if there is excessive usage, reduce that overhead so we can fit more queries in the request window
3. Improve caching on both the frontend and backend to reduce how often we hit the DB

We need to dedicate some time to investigate and build a plan to concretely tackle these issues. For 3, the occurrence of this when editing the dashboard (e.g. navigating between the dashboard and widget builder) will be less when the redesign (see #81292) fully GAs because users will not have to navigate back and forth anymore.

This ticket serves as a log to track any progress we are making towards this issue. It currently needs to be triaged and assigned a DRI.