You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello, ive encountered a strange case where query time increases every 2 hours and leads to lots of nodata alerts and degraded performance
i have a setup of following components
thanos-query-frontend - > thanos query -> fanout to prometheuses inside k8s clusters added via sidecar as a store in thanos query
thanos-query flags ExecStart=/opt/thanos/current/thanos query --http-address 0.0.0.0:19192 --grpc-address 0.0.0.0:19182 --query.replica-label replica --query.replica-label prometheus_replica --query.replica-label receive_replica --log.format=json --store.sd-files="/etc/thanos/stores.yaml" --query.auto-downsampling --query.max-concurrent=1000 --query.max-concurrent-select=1000 --query.timeout=60s --query.default-step=60s --store.response-timeout=120s --store.limits.request-series=10000 --log.level=error --query.promql-engine=thanos
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hello, ive encountered a strange case where query time increases every 2 hours and leads to lots of nodata alerts and degraded performance
i have a setup of following components
thanos-query-frontend - > thanos query -> fanout to prometheuses inside k8s clusters added via sidecar as a store in thanos query
thanos-query flags
ExecStart=/opt/thanos/current/thanos query --http-address 0.0.0.0:19192 --grpc-address 0.0.0.0:19182 --query.replica-label replica --query.replica-label prometheus_replica --query.replica-label receive_replica --log.format=json --store.sd-files="/etc/thanos/stores.yaml" --query.auto-downsampling --query.max-concurrent=1000 --query.max-concurrent-select=1000 --query.timeout=60s --query.default-step=60s --store.response-timeout=120s --store.limits.request-series=10000 --log.level=error --query.promql-engine=thanos
query-frontend
ExecStart=/opt/thanos/current/thanos query-frontend --http-address 0.0.0.0:19195 --query-frontend.downstream-url="http://0.0.0.0:19192" --labels.response-cache-config-file=/etc/thanos/cache.yaml --query-range.response-cache-config-file=/etc/thanos/cache.yaml --log.format=json --query-range.max-query-parallelism=1000 --labels.max-query-parallelism=1000 --query-range.split-interval=12h --query-range.request-downsampled --query-frontend.log-queries-longer-than=30s
thanos version 32.5
it looks like its related to prometheus creating block every 2 hours but i cant seem to find how it can cause such delays
Beta Was this translation helpful? Give feedback.
All reactions