[BUG] TSVB Visualization Broken on Rollup Index

[franco-caylent]

Describe the bug

TSVB returns empty response while line visualization returns the correct response.

To Reproduce

1. Ingest cluster metrics into opensearch using cluster-monitor. Im using a data stream for ingestion.
2. Create a rollup job with the following configuration (modify source and destination accordingly):

{
"rollup": {
    "enabled": true,
    "schedule": {
      "interval": {
        "start_time": 1740417791,
        "period": 5,
        "unit": "Minutes",
        "schedule_delay": 0
      }
    },
    "last_updated_time": 1740417791803,
    "enabled_time": 1740417791798,
    "description": "",
    "schema_version": 21,
    "source_index": "cluster-monitor-ingest",
    "target_index": "cluster-monitor-history",
    "page_size": 1000,
    "delay": 15000,
    "continuous": true,
    "dimensions": [
      {
        "date_histogram": {
          "fixed_interval": "2m",
          "source_field": "@timestamp",
          "target_field": "@timestamp",
          "timezone": "UTC",
          "format": null
        }
      },
      {
        "terms": {
          "source_field": "node_stats.name.keyword",
          "target_field": "node_stats.name.keyword"
        }
      }
    ],
    "metrics": [
      {
        "source_field": "node_stats.jvm.gc.collectors.young.collection_count",
        "metrics": [
          {
            "min": {}
          },
          {
            "max": {}
          },
          {
            "sum": {}
          },
          {
            "avg": {}
          },
          {
            "value_count": {}
          }
        ]
      },
      {
        "source_field": "node_stats.jvm.gc.collectors.old.collection_count",
        "metrics": [
          {
            "min": {}
          },
          {
            "max": {}
          },
          {
            "sum": {}
          },
          {
            "avg": {}
          },
          {
            "value_count": {}
          }
        ]
      },
      {
        "source_field": "node_stats.indices.indexing.index_time_in_millis",
        "metrics": [
          {
            "min": {}
          },
          {
            "max": {}
          },
          {
            "sum": {}
          },
          {
            "avg": {}
          },
          {
            "value_count": {}
          }
        ]
      },
      {
        "source_field": "node_stats.jvm.gc.collectors.young.collection_time_in_millis",
        "metrics": [
          {
            "min": {}
          },
          {
            "max": {}
          },
          {
            "sum": {}
          },
          {
            "avg": {}
          },
          {
            "value_count": {}
          }
        ]
      },
      {
        "source_field": "node_stats.jvm.gc.collectors.old.collection_time_in_millis",
        "metrics": [
          {
            "min": {}
          },
          {
            "max": {}
          },
          {
            "sum": {}
          },
          {
            "avg": {}
          },
          {
            "value_count": {}
          }
        ]
      },
      {
        "source_field": "node_stats.fs.data.free_in_bytes",
        "metrics": [
          {
            "min": {}
          },
          {
            "max": {}
          },
          {
            "sum": {}
          },
          {
            "avg": {}
          },
          {
            "value_count": {}
          }
        ]
      },
      {
        "source_field": "node_stats.jvm.mem.heap_used_percent",
        "metrics": [
          {
            "min": {}
          },
          {
            "max": {}
          },
          {
            "sum": {}
          },
          {
            "avg": {}
          },
          {
            "value_count": {}
          }
        ]
      },
      {
        "source_field": "cluster_health.active_shards_percent_as_number",
        "metrics": [
          {
            "min": {}
          },
          {
            "max": {}
          },
          {
            "sum": {}
          },
          {
            "avg": {}
          },
          {
            "value_count": {}
          }
        ]
      },
      {
        "source_field": "cluster_health.active_shards",
        "metrics": [
          {
            "min": {}
          },
          {
            "max": {}
          },
          {
            "sum": {}
          },
          {
            "avg": {}
          },
          {
            "value_count": {}
          }
        ]
      },
      {
        "source_field": "cluster_stats.indices.store.size_in_bytes",
        "metrics": [
          {
            "min": {}
          },
          {
            "max": {}
          },
          {
            "sum": {}
          },
          {
            "avg": {}
          },
          {
            "value_count": {}
          }
        ]
      }
    ]
  }
}

3. Create a count visualization using line graph
4. Create a count visualization using TSVB
5. Verify the line graph shows events, while TSVB returns an empty response.
6. verify Opensearch logs shows no errors, Dashboards show no errors and the web console shows no errors.

Expected behavior
Both visualizations should yield the same data.

OpenSearch Version
2.17.0

Dashboards Version
2.17.0

Plugins

opensearch01 opensearch-alerting 2.17.0.0
opensearch01 opensearch-anomaly-detection 2.17.0.0
opensearch01 opensearch-asynchronous-search 2.17.0.0
opensearch01 opensearch-cross-cluster-replication 2.17.0.0
opensearch01 opensearch-custom-codecs 2.17.0.0
opensearch01 opensearch-flow-framework 2.17.0.0
opensearch01 opensearch-geospatial 2.17.0.0
opensearch01 opensearch-index-management 2.17.0.0
opensearch01 opensearch-job-scheduler 2.17.0.0
opensearch01 opensearch-knn 2.17.0.0
opensearch01 opensearch-ml 2.17.0.0
opensearch01 opensearch-neural-search 2.17.0.0
opensearch01 opensearch-notifications 2.17.0.0
opensearch01 opensearch-notifications-core 2.17.0.0
opensearch01 opensearch-observability 2.17.0.0
opensearch01 opensearch-performance-analyzer 2.17.0.0
opensearch01 opensearch-reports-scheduler 2.17.0.0
opensearch01 opensearch-security 2.17.0.0
opensearch01 opensearch-security-analytics 2.17.0.0
opensearch01 opensearch-skills 2.17.0.0
opensearch01 opensearch-sql 2.17.0.0
opensearch01 opensearch-system-templates 2.17.0.0
opensearch01 query-insights 2.17.0.0

Screenshots

Host/Environment (please complete the following information):

• OS: Mac OS 15.3.1
• Browser and version: Firefox 135.0 (aarch64)

Additional context
This is only happening with this cluster-monitor dataset, other rollup jobs have no issues.
I have two index aliases, cluster-monitor-ingest and cluster-monitor-history. When running the same query on ingest, it returns an expected response but when changing the alias to history it returns an empty response (while the line graph always returns the expected response)

[franco-caylent]

Quick update, I noticed that when I add a terms aggregation the visualization works. That would indicate that TSVB is working correctly. I'd still like to understand why the line vis shows the count aggregation while tsvb requires a terms aggregation.

[kavilla]

hello @franco-caylent , thanks for opening. i think this could be actually expected behavior. based on this https://opensearch.org/docs/latest/im-plugin/index-rollups/index/ and your responses.

in a regular index (cluster-monitor-ingest), your data looks like this:

{
  "@timestamp": "2024-01-01T10:00",
  "node_stats.name": "node1",
  "node_stats.jvm.mem.heap_used_percent": 75
},
{
  "@timestamp": "2024-01-01T10:01",
  "node_stats.name": "node1",
  "node_stats.jvm.mem.heap_used_percent": 80
}

but in your rolled up index (cluster-monitor-history), the data is pre-aggregated like this:

{
  "@timestamp": "2024-01-01T10:00",
  "node_stats.name.keyword": "node1",
  "node_stats.jvm.mem.heap_used_percent": {
    "avg": 77.5,
    "count": 2,
    "min": 75,
    "max": 80
  }
}

• TSVB needs terms aggregation because it needs to know precalculated value to use (avg, min, max, etc.) for your heap usage metrics
• line charts work because by default are built to handle both structures automatically
• adding terms aggregation worked because you're telling TSVB to use the node_stats.name.keyword grouping that's defined in your rollup

---

if you agree @franco-caylent , i think this issue might be best placed in the doc repo since this took a while for me to get too.

cc: @opensearch-project/admin @ashwin-pc