prometheus-node-exporter-lua: omit zero

[NoctivagusObitus]

Maintainer: @champtar
Environment: OpenWrt 22.03.2, r19803-9a599fee93

Description:

I am using the Prometheus node exporter on OpenWrt for a while now. What I have noticed is, depending on the config, there may be multiple time series always zero (e.g. node_network_transmit_carrier_total{device="lan1"} 0). To me these don't provide any value but increase cpu load measurably. For the time being I have implemented a filter on my devices locally like shown here: master...NoctivagusObitus:packages:feature_omit_zero_export

diff --git a/utils/prometheus-node-exporter-lua/files/usr/bin/prometheus-node-exporter-lua b/utils/prometheus-node-exporter-lua/files/usr/bin/prometheus-node-exporter-lua
index 36f064c5b..a70a7606a 100755
--- a/utils/prometheus-node-exporter-lua/files/usr/bin/prometheus-node-exporter-lua
+++ b/utils/prometheus-node-exporter-lua/files/usr/bin/prometheus-node-exporter-lua
@@ -45,7 +45,9 @@ end
 function metric(name, mtype, labels, value)
   output("# TYPE " .. name .. " " .. mtype)
   local outputter = function(labels, value)
-    print_metric(name, labels, value)
+    if tonumber(value) ~= 0 then
+      print_metric(name, labels, value)
+    end
   end
   if value then
     outputter(labels, value)
lines 1-21/21 (END)

On this Grafana dashboard I disabled my filter. The exported time series where about doubled.

This works fine for me since more then a year now.

I would be happy to contribute a configurable version of this, exporting all series by default. Therefor I would kindly ask for a little push in the right direction for using dynamic arguments in the referenced lua script. From what I understand the node exporter gets called by utthpd in this line

packages/utils/prometheus-node-exporter-lua/files/etc/init.d/prometheus-node-exporter-lua

Line 32 in f9af73d

procd_set_param command /usr/sbin/uhttpd -f -c /dev/null -l / -L /usr/bin/prometheus-node-exporter-lua

But the -L switch seams to only take one argument which is a file and no additional parameters. There may be a better approach?

Looking forward to get feedback on this one.

[NoctivagusObitus]

@champtar any thoughts an that?

[jow-]

You could pass desired options via environment variables, e.g. prior to procd_set_param command ... issue a procd_set_param env SUPPRESS_ZERO_TIMESERIES=1 and within the Lua code check if getenv('SUPPRESS_ZERO_TIMESERIES') == '1' then ...

[NoctivagusObitus]

@jow-

thanks a lot for your suggestion. it worked like a charm out of the box.

It just took me quite some time to test build and install of a real package. which i have accomplished by now. so i have opened my first pull request here: #24016

constructive criticism is very welcome.

[NoctivagusObitus]

@champtar may I ask you to have a look?

[markalex2209]

Am I correct in understanding that proposed change will drop out from the output all metrics that have value 0?

And if so, it might affect other stuff too, especially considering how vector matching in prometheus works.
For example query like node_filesystem_free_bytes/node_filesystem_size_bytes will not yield any value of device have no free space.

Wouldn't it be wiser to implement a way of excluding interfaces from scanning, since this is very much an interface specific issue?

[NoctivagusObitus]

@markalex2209 thanks for your response. First of all your are perfectly right about your understanding. Because of that the configuration design should have been an opt in approach.
But you are also right that in my personal case only network interfaces are relevant. In fact it would make even more sense to filter out unwanted data as early as possible. Of cause this would make for a more elaborate change and more complicated config.
Would netdev.lua be the right place to look into some filtering?

I would still be curious about having some value filtering since node_network_receive_fifo_total is always 0 for me where I would like to keep other metrics of some interfaces. In regards to the example you pointed out I am thinking that it should only be relevant to gauges? In my experience a counter commonly does not even exist with the value of 0 but is only created after its value has reached at least 1 and can only grow from there.

[markalex2209]

Disclaimer: I'm just a passer-by in this repo, so can't speak with any kind of authority on lua, this package, nor openwrt at large.

But using common logic and depending on the vision behind this idea I see two possible ways of implementation:

1. Filter out all interfaces, that were never used since reboot. If I understand this more or less what you personally were trying to solve.
   I think it might be done using regex like ^.+:(?:\s+0){16}. Probably should be OK, since original package already uses regex.
   It still should be an opt-in behaviour, IMO.
2. Filter out interfaces based on list of names provided.

As I understand, netdev.lua is precisely the place to do filtration to keep it limited to node_network_* metrics.

[NoctivagusObitus]

Still thank you a lot for your input.

Maybe @champtar has an opinion on this?
I would be happy to either look into excluding interfaces in netdev.lua or using the mtype of the metric function to limit the exclude 0 logic to counters in here:

packages/utils/prometheus-node-exporter-lua/files/usr/bin/prometheus-node-exporter-lua

Lines 45 to 54 in fca0cec

	function metric(name, mtype, labels, value)
	output("# TYPE " .. name .. " " .. mtype)
	local outputter = function(labels, value)
	print_metric(name, labels, value)
	end
	if value then
	outputter(labels, value)
	end
	return outputter
	end