Skip to content

[BUG] metrics collector job bugs #4539

@ylwu-amzn

Description

@ylwu-amzn

1. early exit the for loop

https://code.amazon.com/packages/OpenSearch-aos-machine-learning/blobs/45897346a129c232ca5268173c92f369abe386f0/--/ml-commons/plugin/src/main/java/org/opensearch/ml/jobs/processors/MLStatsJobProcessor.java#L168

This line exits the whole for loop. Seems not correct.

2. null listener

Update setting

PUT _cluster/settings
{
  "persistent" : {
    "plugins.ml_commons.metrics_static_collection_enabled" : true
  }
}

It succeed and returns

{
  "acknowledged": true,
  "persistent": {
    "plugins": {
      "ml_commons": {
        "metrics_static_collection_enabled": "true"
      }
    }
  },
  "transient": {}
}

But see error in log

[2026-01-14T19:46:40,930][WARN ][o.o.c.s.ClusterApplierService] [node1] failed to apply updated cluster state in [2.1s]:
version [76], uuid [yGjrFKG_SVGVsMIjHIDS1Q], source [becoming candidate: clusterApplier#onNewClusterState]
java.lang.NullPointerException: Cannot invoke "org.opensearch.ml.common.settings.SettingsChangeListener.onStaticMetricCollectionEnabledChanged(boolean)" because "listener" is null
	at org.opensearch.ml.common.settings.MLFeatureEnabledSetting.lambda$new$14(MLFeatureEnabledSetting.java:118) ~[?:?]
	at org.opensearch.common.settings.Setting$Updater.apply(Setting.java:1270) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.common.settings.AbstractScopedSettings$SettingUpdater.lambda$updater$0(AbstractScopedSettings.java:694) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.common.settings.AbstractScopedSettings.applySettings(AbstractScopedSettings.java:230) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:590) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:516) [opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:212) [opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916) [opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:299) [opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:262) [opensearch-3.3.2.jar:3.3.2]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:1447) [?:?]

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions