Add "Time Consumed" column to "Performance" page

[natematykiewicz]

Problem Statement

Right next to the TPM, P50, P95 columns add a Time Consumed column. This is a feature that Scout has that is missing in Sentry.

In Scout you view Web Endpoints and Background Jobs on separate pages. This Time Consumed column basically multiplies TPM x Average Response Time to get an idea of what endpoints are using your servers the most.

It's really easy to look at a 10 second P50 web request and want to fix that (and you should), but if that 10 second web request occurs once a month, it might actually make way more sense to optimize a 500ms request that happens 100k times a day.

In Scout, if you sort by Time Consumed descending, you essentially get a list of what you should optimize in order to be able to handle more requests with the same amount of servers (or even reduce your server pool size). It's essentially a "Best Bang for the Buck" sort.

Solution Brainstorm

Add a "Time Consumed" column that's a percentage of time per Operation.

So for me, I see that my "Operation" column has 2 different values in it: sidekiq and rails.request. It would want to easily see which sidekiq jobs my servers spend the most time on, and which rails requests my servers spend the most time on. I don't think there's any value in providing an overall percentage (that's sidekiq + rails.request), because they're two different groups of work that get scaled independently.

[getsentry-release]

Routing to @getsentry/visibility for triage. ⏲️

[narsaynorath]

Hi @natematykiewicz, thanks for the feedback! I've filed this as a ticket in the Performance team's backlog. We'll keep this issue open if we need to gather more information about your request or update with progress.

[natematykiewicz]

Thanks!

[natematykiewicz]

Hi @narsaynorath! Just checking back on this. This feature is probably the single most helpful thing about Scout APM. Is there a way to find this information in Sentry today or do I need to continue using Scout as my APM?

[getsantry]

Routing to @getsentry/product-owners-performance for triage ⏲️

[chadwhitacre]

Sorry this fell through the cracks @natematykiewicz. Back on the radar thanks to @nateberkopec, applied some labels to make sure the right team has this in their queue and follows up. 🤞

[brentc]

This feels like something the starfish project is trying to address. @zoesyc @nanoburd ?

[getsantry]

Routing to @getsentry/product-owners-starfish for triage ⏲️

[getsantry]

Routing to @getsentry/product-owners-performance for triage ⏲️

[natematykiewicz]

@zoesyc I know we talked and you showed me the database performance page, which is great, but not what I was looking for to address this issue.

Lack of a "Time Consumed" column on the "Performance" page is one of the top reasons we still pay for Scout and haven't gone all in on Sentry as our APM.

[zoesyc]

hey @natematykiewicz -

This request is still very much on our radar!

We are considering a larger refactor of our performance monitoring solution (which includes this work). I can keep you informed of any updates regarding the larger project or if we implement this feature before then.

[0Calories]

With Sentry Insights, we now have the Time Spent column which effectively is what is being described in this issue. So this is a thing in Sentry now :)

Closing as completed, let me know if you have any questions @natematykiewicz

[natematykiewicz]

Where is this column @0Calories?

[0Calories]

@natematykiewicz From what I can see, you're looking at the Backend view, right? If you switch to the Queries tab, you can see the Time Spent column, so we effectively list out all your queries in the order of the ones that take up the most time in your application. We don't have this column for the Overview tab where your backend transaction are listed. Does your usecase require a column like this specifically for transactions?

[natematykiewicz]

Does your usecase require a column like this specifically for transactions?

Yes. This Query "Time Spent" is far too fine of granularity for what I'm talking about. It's helpful for optimizing a single query, but if I'm trying to increase throughput of my web application, I want to know which endpoint is using my servers the most. This is often the result of application code and/or many queries running in one request, as opposed to a single slow query.

And it's not necessarily the slowest endpoint either. A 10s endpoint that gets requested by an admin once a month won't gain me any throughput to fix. A 300ms endpoint that gets requested 1000 times per second will gain a lot of throughput even with a 50ms improvement.

[0Calories]

@natematykiewicz I definitely see the value in that. I brought this back to the team and we're considering swapping out user misery and apdex with time spent as a potential alternative. Does this sound like it would fit for your use-case, or do you use those sorting fields as well?

[natematykiewicz]

To be honest, I don't really use apex or user misery. They're so abstract and I don't really know what they mean. If I'm trying to improve performance for UX purposes, I sort by P50/P95. If I'm trying to improve performance to lower server hosting costs, I use Scout and sort by the "Time Consumed" column that I'm asking for.

Also @0Calories, I did have a video interview with @zoesyc back in September of 2023 about Project Starfish. I shared my screen and demonstrated how I use Scout to solve performance issues. Perhaps she still has screenshots from that.