carbon-cache (graphitewriter) is not processing queue of performance data after upgrading from 2.15.2 to 2.16

[bertdeknijf]

Describe the bug

A clear and concise description of what the bug is.

Please ensure to read https://github.com/Icinga/icinga2/blob/master/doc/15-troubleshooting.md first. Formatting tips: GitHub supports Markdown: https://guides.github.com/features/mastering-markdown/

To Reproduce

Provide a link to a live example, or an unambiguous set of steps to reproduce this bug. Include configuration, logs, etc. to reproduce, if relevant.

1. upgrade icinga from version 2.15.2 to 2.16 (tested in debian bookworm)
2. carbon cache is still receiving data, but no longer parsing or parsing very slowly
3. restarting icinga2 clears the queue, but it fills up very fast again and not processed nearly fast enough

this causes many checks in 'overdue' state

Expected behavior

i would expect the data to be parsed faster (or at all)

Screenshots

If applicable, add screenshots to help explain your problem.

Your Environment

Include as many relevant details about the environment you experienced the problem in

root@monitoring:/etc/icinga2# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.15.2-1)

Copyright (c) 2012-2026 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later https://gnu.org/licenses/gpl2.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
Platform: Debian GNU/Linux
Platform version: 12 (bookworm)
Kernel: Linux
Kernel version: 6.1.0-44-amd64
Architecture: x86_64

Build information:
Compiler: GNU 12.2.0
Build host: runner-aa2jha3hm-project-575-concurrent-1
OpenSSL version: OpenSSL 3.0.19 27 Jan 2026

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

debian 12 ->

Additional context

Add any other context about the problem here.

[jschmidt-icinga]

I'll spin up my graphite setup next week and see if I can reproduce your issue.
In the meantime, can you provide logs of both your Icinga 2 instance as well as graphite that might help?

[bertdeknijf]

ok, thanks.
i attached a zipfile with some relevant logs

icinga-sanitized.zip

[jschmidt-icinga]

Since you're processing a huge number of check results and there are no errors, it looks like the problem is a performance regression due to the additional overhead introduced to the GraphiteWriter in #10668.

This can be greatly mitigated by batching the individual metrics (plus metadata and thresholds) per check result, since currently every item is its own send, causing its own coroutine to be spawned etc.

I will try to have a fix ready for 2.16.1 (no timeline as of yet). In the mean time, either continue on 2.15.3, or alternatively you can try to mitigate the issue by turning off metadata and/or thresholds (in case you don't need one or both of those).

object GraphiteWriter "graphite" {
  host = "graphite"
  port = 2003
  enable_send_thresholds = false
  enable_send_metadata = false
}

[jogoossens]

This can be greatly mitigated by batching the individual metrics (plus metadata and thresholds) per check result, since currently every item is its own send, causing its own coroutine to be spawned etc.

This is a fix that would be available in 2.16.1 or could we change config to do this? :)

[jschmidt-icinga]

This is a fix that would be available in 2.16.1 or could we change config to do this? :)

No, sorry. It needs some refactoring in lib/perfdata/graphitewriter.cpp.

Some more details: The way GraphiteWriter (and some others) currently work is that it does send/flush IO-Operations per metric. #10668 didn't even touch that, but to make it possible to cancel operations they were put into a coroutine, which adds an overhead to each operation. The proper fix/mitigation is to have this overhead only once per check-result, by collecting all metrics in a buffer and then sending them out in a single IO-Operation.

[jogoossens]

This is a fix that would be available in 2.16.1 or could we change config to do this? :)

No, sorry. It needs some refactoring in lib/perfdata/graphitewriter.cpp.

Some more details: The way GraphiteWriter (and some others) currently work is that it does send/flush IO-Operations per metric. #10668 didn't even touch that, but to make it possible to cancel operations they were put into a coroutine, which adds an overhead to each operation. The proper fix/mitigation is to have this overhead only once per check-result, by collecting all metrics in a buffer and then sending them out in a single IO-Operation.

ok cool, anything that can speed it up would be more than welcome!!

[jschmidt-icinga]

@jogoossens @bertdeknijf Are you willing to try out #10834 and see if that fixes the issue for you? Preferably I'd ask you to build from the branch yourselves, but I can provide packages for your distros if that helps.

[jogoossens]

if you could build debian 12 packages (all required .deb files) we could test it for sure! Is that an option? Thanks so much already for the rather quick fix :=)

[jogoossens]

you did not put them somewhere already right ? (I might have missed them :) )

[jschmidt-icinga]

No sorry, didn't get around to it on Thursday and Friday was a holiday here. I'll see if I can get them built by the end of today.