Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Sensor metrics can have idential names #398

Open
Mikle-Bond opened this issue Oct 3, 2024 · 0 comments
Open

[Bug]: Sensor metrics can have idential names #398

Mikle-Bond opened this issue Oct 3, 2024 · 0 comments

Comments

@Mikle-Bond
Copy link

Symptoms

Prometheus will warn about clashing values in logs on every scrape. Example:

{"caller":"scrape.go:1744","component":"scrape manager","level":"warn","msg":"Error on ingesting samples with different value but same timestamp","num_dropped":2,"scrape_pool":"vmware","target":"http://vmware-exporter:9272/metrics?target=10.10.10.1","ts":"2024-10-03T07:30:54.384Z"}

Culprits

Looking at the output of exporter, there are metrics with the same name/labels repeated several times. Example (emphasis mine):

[...snip...]
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="igb driver 5.0.5.1",type="Software Components"} 2.0
* vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="igb driver 5.0.5.1",type="Software Components"} 2.0
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="Power Supply 1: Running/Full Power-Enabled",type="power"} 2.0
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="Power Supply 2: Running/Full Power-Enabled",type="power"} 2.0
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="System Chassis 1 Chassis Intru: General Chassis intrusion - Deassert",type="Chassis"} 2.0
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="System Chassis 1 Chassis Intru: Drive Bay intrusion - Deassert",type="Chassis"} 2.0
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="System Chassis 1 Chassis Intru: I/O Card area intrusion - Deassert",type="Chassis"} 2.0
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="System Chassis 1 Chassis Intru: Processor area intrusion - Deassert",type="Chassis"} 2.0
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="System Chassis 1 Chassis Intru: System unplugged from LAN - Deassert",type="Chassis"} 2.0
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="System Chassis 1 Chassis Intru: Unauthorized dock - Deassert",type="Chassis"} 2.0
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="System Chassis 1 Chassis Intru: FAN area intrusion - Deassert",type="Chassis"} 2.0
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="System Chassis 1 Chassis Intru: Unknown - Deassert",type="Chassis"} 2.0
* vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="System Chassis 1 Chassis Intru: Unknown - Deassert",type="Chassis"} 2.0
* vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="System Chassis 1 Chassis Intru: Unknown - Deassert",type="Chassis"} 2.0
* vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="System Chassis 1 Chassis Intru: Unknown - Deassert",type="Chassis"} 2.0
* vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="System Chassis 1 Chassis Intru: Unknown - Deassert",type="Chassis"} 2.0
* vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="System Chassis 1 Chassis Intru: Unknown - Deassert",type="Chassis"} 2.0
* vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="System Chassis 1 Chassis Intru: Unknown - Deassert",type="Chassis"} 2.0
* vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="System Chassis 1 Chassis Intru: Unknown - Deassert",type="Chassis"} 2.0
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="Power Supply 2 PS2 Status: Failure status - Deassert",type="power"} 2.0
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="Power Supply 2 PS2 Status: Predictive failure - Deassert",type="power"} 2.0
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="Power Supply 2 PS2 Status: Power Supply AC lost - Deassert",type="power"} 2.0
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="Power Supply 1 PS1 Status: Failure status - Deassert",type="power"} 2.0
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="Power Supply 1 PS1 Status: Predictive failure - Deassert",type="power"} 2.0
  vmware_host_sensor_state{cluster_name="",dc_name="ha-datacenter",host_name="node-207.work.internal",name="Power Supply 1 PS1 Status: Power Supply AC lost - Deassert",type="power"} 2.0
[...snip...]

Getting via pyVim, this is what sensor descriptions look like

   (vim.host.NumericSensorInfo) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      name = 'System Chassis 1 Chassis Intru: Unknown - Deassert',
      healthState = (vim.ElementDescription) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         label = 'Green',
         summary = 'Sensor is operating under normal conditions',
         key = 'green'
      },
      currentReading = 0L,
      unitModifier = 0,
      baseUnits = '',
      rateUnits = <unset>,
      sensorType = 'Chassis'
   },
[...repeats 7 more times...]
   (vim.host.NumericSensorInfo) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      name = 'igb driver 5.0.5.1',
      healthState = (vim.ElementDescription) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         label = 'Green',
         summary = 'Sensor is operating under normal conditions',
         key = 'green'
      },
      currentReading = 0L,
      unitModifier = 0,
      baseUnits = '',
      rateUnits = <unset>,
      sensorType = 'Software Components'
   },
[...repeats 1 more time...]
(vim.host.NumericSensorInfo) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   name = 'igb device firmware 1.2.3',
   healthState = (vim.ElementDescription) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      label = 'Green',
      summary = 'Sensor is operating under normal conditions',
      key = 'green'
   },
   currentReading = 0L,
   unitModifier = 0,
   baseUnits = '',
   rateUnits = <unset>,
   sensorType = 'Software Components'
},
[...repeats 1 more time...]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant