[Metrics] Potential issue with grouping gauges w/ dimensions by week/month/year

[jstanden]

I think the issue here is we could be asking two different questions with averages for a gauge metric like cerb.tickets.open, which measures the number of tickets in each bucket at an interval (5min, hour, daily).

(1) What was the avg number of open tickets in the system (across all buckets) at a point in time? We would expect this to be the sum of all bucket averages at that time. e.g. There were 180 open tickets on average yesterday (30 + 50 + 20 + 80 in buckets A,B,C,D).

(2) What was the number of open tickets in an average bucket at a point in time? We would expect this to be the avg of all bucket averages. e.g. The average bucket had 45 open tickets yesterday (45 == (30+50+20+80)/4).

The original reports people wanted that led to the creation of metrics all dealt with (1), and that's a baked-in assumption.

It does seem like (1) is causing problems when we aggregate daily samples as a week/month/year.

I have some new test data going that I can validate the statistics with for that scenario. We probably need an explicit way to refer to magnitude (total) or central tendency (average).