Add logger_name tagging support to LogbackMetrics

[dsu]

I would like the logback_events_total metric to support logger_name as a tag, and not only the log level.
LogbackMetrics might have Map<String, Counter> erorCounter instead of a single counter for all loggers for the log level.
This would increase the value of the metric, as it would tell where and when problem occurred exactly.
It would be helpful for log optimization (optimizing the number of logs).

[jonatan-ivanov]

Thanks for the issue!

I would like the logback_events_total metric to support logger_name as a tag, and not only the log level.

I think I see how this could be useful but the biggest tradeoff I see is that this would significantly increase the number of time-series. I would totally understand some of the users saying this is high cardinality data for them which can really be the case for everyone having a component that uses random/generated logger names. Usually, this is not the case but the cordiality will definitely explode for most apps. For example, this simple logger in Spring Framework is actually 6 loggers by default:

org.springframework.web.servlet.DispatcherServlet

since loggers are hierarchical. If you have a longer package name, you will have a longer logger hierarchy and more loggers. That number times the available levels (5), so the DispatcherServlet logger above can end up having 6*5=30 time series alone.

I just looked into a not too complicated Spring Boot app, right after startup, it registered ~1300 loggers. If all (unlikely) produces all 5 levels, we are looking at 1300*5=6500 time series only for logback for this app alone.

LogbackMetrics might have Map<String, Counter> erorCounter instead of a single counter for all loggers for the log level.

This is an implementation detail but the registry already has such a map. Also, the key needs to encode the logger name, the level and all the other possible user-provided tags, I don't think I would do this.

This would increase the value of the metric, as it would tell where and when problem occurred exactly.

It can explode them. I understand where this can be useful but once you see that there is an issue, cannot you dig deeper by querying the logs themselves?

It would be helpful for log optimization (optimizing the number of logs).

I think if I would like to optimize the number of logs, after looking at their volume (can be this metric), I would query the logs to see the amount of events by name.

I will check with others if we have seen a similar feature request. You can try the above out and see how many time series are created for your app if you copy and modify the LogbackMetrics class. I would also recommend trying out MeterProvider for this implementation.

[dsu]

Hi @jonatan-ivanov ,
Thank you for the feedback.

Maybe the package/loggger names and child logger name level limit that will be captured can be used as a config option?

This way it should not explode.

I am interested only in logs generated by my packages.

I would query the logs to see the number of events by name.

Unfortunately, OpenSearch does not index logger_name, so I am stuck with it, and I am unable to generate the report automatically.

We already use logs metrics in Grafana to monitor the app, so it makes more sense to add it here.

[shakuzen]

Maybe the package/loggger names and child logger name level limit that will be captured can be used as a config option?

This way it should not explode.

I am interested only in logs generated by my packages.

That sounds reasonable, although the cardinality is still potentially an issue. This is the first time this is being requested, so we would wait for others to express interest in the feature before we take on the support burden of the feature here. You could copy LogbackMetrics and modify it to work how you need in the meantime.