Cache default settings

[jordan-powers]

This PR introduces a mechanism of caching the computed default values for settings in order to fix a performance regression in the many-shards-quantitative benchmark associated with the introduction of some default providers in #135886.

Alternate approach to #138711.

[jordan-powers]

Benchmark results running the elastic/logs many-shards-quantitative benchmark in esbench.

Baseline ref: 22a7491
Contender ref: b4f9c07

Results:

|                                                        Metric |                                   Task |         Baseline |        Contender |        Diff |   Unit |   Diff % |
|--------------------------------------------------------------:|---------------------------------------:|-----------------:|-----------------:|------------:|-------:|---------:|
|                    Cumulative indexing time of primary shards |                                        |    459.276       |    488.942       |    29.6653  |    min |   +6.46% |
|             Min cumulative indexing time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
|          Median cumulative indexing time across primary shard |                                        |      1.66667e-05 |      1.66667e-05 |     0       |    min |    0.00% |
|             Max cumulative indexing time across primary shard |                                        |    276.031       |    294.503       |    18.4719  |    min |   +6.69% |
|           Cumulative indexing throttle time of primary shards |                                        |      0           |      0           |     0       |    min |    0.00% |
|    Min cumulative indexing throttle time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
| Median cumulative indexing throttle time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
|    Max cumulative indexing throttle time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
|                       Cumulative merge time of primary shards |                                        |    232.969       |    191.653       |   -41.3167  |    min |  -17.73% |
|                      Cumulative merge count of primary shards |                                        |   1638           |   1732           |    94       |        |   +5.74% |
|                Min cumulative merge time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
|             Median cumulative merge time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
|                Max cumulative merge time across primary shard |                                        |    173.972       |    140.93        |   -33.0413  |    min |  -18.99% |
|              Cumulative merge throttle time of primary shards |                                        |    105.254       |     78.6038      |   -26.6497  |    min |  -25.32% |
|       Min cumulative merge throttle time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
|    Median cumulative merge throttle time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
|       Max cumulative merge throttle time across primary shard |                                        |     78.3887      |     57.5648      |   -20.8239  |    min |  -26.56% |
|                     Cumulative refresh time of primary shards |                                        |     32.7415      |     33.2234      |     0.48188 |    min |   +1.47% |
|                    Cumulative refresh count of primary shards |                                        | 419699           | 420993           |  1294       |        |   +0.31% |
|              Min cumulative refresh time across primary shard |                                        |      0           |      0           |     0       |    min |    0.00% |
|           Median cumulative refresh time across primary shard |                                        |      6.66667e-05 |      8.33333e-05 |     2e-05   |    min |  +25.00% |
|              Max cumulative refresh time across primary shard |                                        |      9.42203     |      9.15885     |    -0.26318 |    min |   -2.79% |
|                       Cumulative flush time of primary shards |                                        |      6.75512     |      6.24327     |    -0.51185 |    min |   -7.58% |
|                      Cumulative flush count of primary shards |                                        |  50746           |  50754           |     8       |        |   +0.02% |
|                Min cumulative flush time across primary shard |                                        |      1.66667e-05 |      1.66667e-05 |     0       |    min |    0.00% |
|             Median cumulative flush time across primary shard |                                        |      5e-05       |      5e-05       |     0       |    min |    0.00% |
|                Max cumulative flush time across primary shard |                                        |      1.43217     |      1.43002     |    -0.00215 |    min |   -0.15% |
|                                       Total Young Gen GC time |                                        |    136.701       |    138.139       |     1.438   |      s |   +1.05% |
|                                      Total Young Gen GC count |                                        |   3538           |   3747           |   209       |        |   +5.91% |
|                                         Total Old Gen GC time |                                        |      0           |      0           |     0       |      s |    0.00% |
|                                        Total Old Gen GC count |                                        |      0           |      0           |     0       |        |    0.00% |
|                                                  Dataset size |                                        |     38.5072      |     36.2961      |    -2.21103 |     GB |   -5.74% |
|                                                    Store size |                                        |     38.5072      |     36.2961      |    -2.21103 |     GB |   -5.74% |
|                                                 Translog size |                                        |      0.00512289  |      0.0051232   |     0       |     GB |    0.01% |
|                                        Heap used for segments |                                        |      0           |      0           |     0       |     MB |    0.00% |
|                                      Heap used for doc values |                                        |      0           |      0           |     0       |     MB |    0.00% |
|                                           Heap used for terms |                                        |      0           |      0           |     0       |     MB |    0.00% |
|                                           Heap used for norms |                                        |      0           |      0           |     0       |     MB |    0.00% |
|                                          Heap used for points |                                        |      0           |      0           |     0       |     MB |    0.00% |
|                                   Heap used for stored fields |                                        |      0           |      0           |     0       |     MB |    0.00% |
|                                                 Segment count |                                        |  50361           |  50408           |    47       |        |   +0.09% |
|                                   Total Ingest Pipeline count |                                        |      2.10812e+08 |      2.10812e+08 |     0       |        |    0.00% |
|                                    Total Ingest Pipeline time |                                        |      1.3458e+07  |      1.34619e+07 |  3984       |     ms |   +0.03% |
|                                  Total Ingest Pipeline failed |                                        |      0           |      0           |     0       |        |    0.00% |
|                                                Min Throughput |                       insert-pipelines |      8.65798     |      8.19807     |    -0.45991 |  ops/s |   -5.31% |
|                                               Mean Throughput |                       insert-pipelines |      8.65798     |      8.19807     |    -0.45991 |  ops/s |   -5.31% |
|                                             Median Throughput |                       insert-pipelines |      8.65798     |      8.19807     |    -0.45991 |  ops/s |   -5.31% |
|                                                Max Throughput |                       insert-pipelines |      8.65798     |      8.19807     |    -0.45991 |  ops/s |   -5.31% |
|                                      100th percentile latency |                       insert-pipelines |   1729.14        |   1826.18        |    97.0411  |     ms |   +5.61% |
|                                 100th percentile service time |                       insert-pipelines |   1729.14        |   1826.18        |    97.0411  |     ms |   +5.61% |
|                                                    error rate |                       insert-pipelines |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                             insert-ilm |     20.0165      |     18.7904      |    -1.22611 |  ops/s |   -6.13% |
|                                               Mean Throughput |                             insert-ilm |     20.0165      |     18.7904      |    -1.22611 |  ops/s |   -6.13% |
|                                             Median Throughput |                             insert-ilm |     20.0165      |     18.7904      |    -1.22611 |  ops/s |   -6.13% |
|                                                Max Throughput |                             insert-ilm |     20.0165      |     18.7904      |    -1.22611 |  ops/s |   -6.13% |
|                                      100th percentile latency |                             insert-ilm |     48.6596      |     52.2647      |     3.60506 |     ms |   +7.41% |
|                                 100th percentile service time |                             insert-ilm |     48.6596      |     52.2647      |     3.60506 |     ms |   +7.41% |
|                                                    error rate |                             insert-ilm |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput | validate-package-template-installation |     35.3818      |     33.2881      |    -2.0937  |  ops/s |   -5.92% |
|                                               Mean Throughput | validate-package-template-installation |     35.3818      |     33.2881      |    -2.0937  |  ops/s |   -5.92% |
|                                             Median Throughput | validate-package-template-installation |     35.3818      |     33.2881      |    -2.0937  |  ops/s |   -5.92% |
|                                                Max Throughput | validate-package-template-installation |     35.3818      |     33.2881      |    -2.0937  |  ops/s |   -5.92% |
|                                      100th percentile latency | validate-package-template-installation |     27.8724      |     29.6411      |     1.76866 |     ms |   +6.35% |
|                                 100th percentile service time | validate-package-template-installation |     27.8724      |     29.6411      |     1.76866 |     ms |   +6.35% |
|                                                    error rate | validate-package-template-installation |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |        update-custom-package-templates |     23.383       |     23.151       |    -0.23208 |  ops/s |   -0.99% |
|                                               Mean Throughput |        update-custom-package-templates |     23.383       |     23.151       |    -0.23208 |  ops/s |   -0.99% |
|                                             Median Throughput |        update-custom-package-templates |     23.383       |     23.151       |    -0.23208 |  ops/s |   -0.99% |
|                                                Max Throughput |        update-custom-package-templates |     23.383       |     23.151       |    -0.23208 |  ops/s |   -0.99% |
|                                      100th percentile latency |        update-custom-package-templates |    512.8         |    517.973       |     5.1734  |     ms |   +1.01% |
|                                 100th percentile service time |        update-custom-package-templates |    512.8         |    517.973       |     5.1734  |     ms |   +1.01% |
|                                                    error rate |        update-custom-package-templates |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                    add-many-shards-ilm |     22.7746      |     24.0445      |     1.26995 |  ops/s |   +5.58% |
|                                               Mean Throughput |                    add-many-shards-ilm |     22.7746      |     24.0445      |     1.26995 |  ops/s |   +5.58% |
|                                             Median Throughput |                    add-many-shards-ilm |     22.7746      |     24.0445      |     1.26995 |  ops/s |   +5.58% |
|                                                Max Throughput |                    add-many-shards-ilm |     22.7746      |     24.0445      |     1.26995 |  ops/s |   +5.58% |
|                                      100th percentile latency |                    add-many-shards-ilm |     40.6528      |     38.0951      |    -2.55768 |     ms |   -6.29% |
|                                 100th percentile service time |                    add-many-shards-ilm |     40.6528      |     38.0951      |    -2.55768 |     ms |   -6.29% |
|                                                    error rate |                    add-many-shards-ilm |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                        initial-indices |     80.4095      |    193.747       |   113.337   | docs/s | +140.95% |
|                                               Mean Throughput |                        initial-indices |   1330.25        |   1464.08        |   133.828   | docs/s |  +10.06% |
|                                             Median Throughput |                        initial-indices |   1190.46        |   1253.05        |    62.5963  | docs/s |   +5.26% |
|                                                Max Throughput |                        initial-indices |   2368.7         |   2970.9         |   602.2     | docs/s |  +25.42% |
|                                       50th percentile latency |                        initial-indices |   3346.26        |   3237.45        |  -108.817   |     ms |   -3.25% |
|                                       90th percentile latency |                        initial-indices |   5895.69        |   5856.14        |   -39.5453  |     ms |   -0.67% |
|                                       99th percentile latency |                        initial-indices |   6435.26        |   6946.69        |   511.428   |     ms |   +7.95% |
|                                     99.9th percentile latency |                        initial-indices |   7391.12        |   7359.31        |   -31.803   |     ms |   -0.43% |
|                                    99.99th percentile latency |                        initial-indices |   7879.23        |   7396.52        |  -482.71    |     ms |   -6.13% |
|                                      100th percentile latency |                        initial-indices |   7883.95        |   7400.07        |  -483.878   |     ms |   -6.14% |
|                                  50th percentile service time |                        initial-indices |   3348.81        |   3254.16        |   -94.6597  |     ms |   -2.83% |
|                                  90th percentile service time |                        initial-indices |   5897.76        |   5861.6         |   -36.1654  |     ms |   -0.61% |
|                                  99th percentile service time |                        initial-indices |   6439.89        |   6935.02        |   495.129   |     ms |   +7.69% |
|                                99.9th percentile service time |                        initial-indices |   7397.52        |   7359.17        |   -38.3486  |     ms |   -0.52% |
|                               99.99th percentile service time |                        initial-indices |   7879.23        |   7396.31        |  -482.913   |     ms |   -6.13% |
|                                 100th percentile service time |                        initial-indices |   7883.95        |   7400.07        |  -483.878   |     ms |   -6.14% |
|                                                    error rate |                        initial-indices |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                        discover/search |      0.00228704  |      0.0022892   |     0       |  ops/s |   +0.09% |
|                                               Mean Throughput |                        discover/search |      0.00759078  |      0.0075417   |    -5e-05   |  ops/s |   -0.65% |
|                                             Median Throughput |                        discover/search |      0.00804437  |      0.00805151  |     1e-05   |  ops/s |   +0.09% |
|                                                Max Throughput |                        discover/search |      0.009853    |      0.00985411  |     0       |  ops/s |   +0.01% |
|                                       50th percentile latency |                        discover/search |    501.631       |    461.239       |   -40.392   |     ms |   -8.05% |
|                                       90th percentile latency |                        discover/search |   2164.05        |   1526.74        |  -637.31    |     ms |  -29.45% |
|                                      100th percentile latency |                        discover/search |   5054.01        |   2835.38        | -2218.63    |     ms |  -43.90% |
|                                  50th percentile service time |                        discover/search |    424.471       |    422.045       |    -2.42682 |     ms |   -0.57% |
|                                  90th percentile service time |                        discover/search |    835.619       |    610.23        |  -225.389   |     ms |  -26.97% |
|                                 100th percentile service time |                        discover/search |   5053.24        |   2834.44        | -2218.8     |     ms |  -43.91% |
|                                                    error rate |                        discover/search |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                     discover/visualize |      0.00231463  |      0.00231566  |     0       |  ops/s |   +0.04% |
|                                               Mean Throughput |                     discover/visualize |      0.00829714  |      0.00828498  |    -1e-05   |  ops/s |   -0.15% |
|                                             Median Throughput |                     discover/visualize |      0.0079041   |      0.00791961  |     2e-05   |  ops/s |   +0.20% |
|                                                Max Throughput |                     discover/visualize |      0.014204    |      0.0142083   |     0       |  ops/s |   +0.03% |
|                                       50th percentile latency |                     discover/visualize |    407.759       |    406.416       |    -1.3436  |     ms |   -0.33% |
|                                       90th percentile latency |                     discover/visualize |    698.916       |   1425.95        |   727.032   |     ms | +104.02% |
|                                      100th percentile latency |                     discover/visualize |   2994.83        |   4964.49        |  1969.66    |     ms |  +65.77% |
|                                  50th percentile service time |                     discover/visualize |    405.559       |    404.619       |    -0.9395  |     ms |   -0.23% |
|                                  90th percentile service time |                     discover/visualize |    477.164       |    431.831       |   -45.3327  |     ms |   -9.50% |
|                                 100th percentile service time |                     discover/visualize |   2994.52        |   4963.8         |  1969.28    |     ms |  +65.76% |
|                                                    error rate |                     discover/visualize |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                                  kafka |      0.00502374  |      0.0050233   |    -0       |  ops/s |   -0.01% |
|                                               Mean Throughput |                                  kafka |      0.0129604   |      0.0129603   |    -0       |  ops/s |   -0.00% |
|                                             Median Throughput |                                  kafka |      0.0134311   |      0.0134308   |    -0       |  ops/s |   -0.00% |
|                                                Max Throughput |                                  kafka |      0.0187264   |      0.0187249   |    -0       |  ops/s |   -0.01% |
|                                       50th percentile latency |                                  kafka |    408.485       |    409.231       |     0.74591 |     ms |   +0.18% |
|                                       90th percentile latency |                                  kafka |    563.111       |    503.329       |   -59.7819  |     ms |  -10.62% |
|                                      100th percentile latency |                                  kafka |   3529.2         |   2589.28        |  -939.919   |     ms |  -26.63% |
|                                  50th percentile service time |                                  kafka |    406.878       |    405.843       |    -1.03513 |     ms |   -0.25% |
|                                  90th percentile service time |                                  kafka |    516.008       |    444.948       |   -71.0593  |     ms |  -13.77% |
|                                 100th percentile service time |                                  kafka |   3528.47        |   2588.53        |  -939.94    |     ms |  -26.64% |
|                                                    error rate |                                  kafka |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                                  nginx |      0.00627563  |      0.00640298  |     0.00013 |  ops/s |   +2.03% |
|                                               Mean Throughput |                                  nginx |      0.0218074   |      0.021858    |     5e-05   |  ops/s |   +0.23% |
|                                             Median Throughput |                                  nginx |      0.0211443   |      0.0211092   |    -4e-05   |  ops/s |   -0.17% |
|                                                Max Throughput |                                  nginx |      0.0345796   |      0.0346968   |     0.00012 |  ops/s |   +0.34% |
|                                       50th percentile latency |                                  nginx |   1822.27        |   1809.96        |   -12.307   |     ms |   -0.68% |
|                                       90th percentile latency |                                  nginx |   3817.53        |   3603.74        |  -213.796   |     ms |   -5.60% |
|                                      100th percentile latency |                                  nginx |   6164.35        |   6433.94        |   269.591   |     ms |   +4.37% |
|                                  50th percentile service time |                                  nginx |   1797.21        |   1792.62        |    -4.5863  |     ms |   -0.26% |
|                                  90th percentile service time |                                  nginx |   1971.45        |   1925.03        |   -46.4196  |     ms |   -2.35% |
|                                 100th percentile service time |                                  nginx |   4425.57        |   6269.32        |  1843.75    |     ms |  +41.66% |
|                                                    error rate |                                  nginx |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                                 apache |      0.0233558   |      0.0233904   |     3e-05   |  ops/s |   +0.15% |
|                                               Mean Throughput |                                 apache |      0.0529463   |      0.053147    |     0.0002  |  ops/s |   +0.38% |
|                                             Median Throughput |                                 apache |      0.029454    |      0.0294394   |    -1e-05   |  ops/s |   -0.05% |
|                                                Max Throughput |                                 apache |      0.153354    |      0.158261    |     0.00491 |  ops/s |   +3.20% |
|                                       50th percentile latency |                                 apache |    402.697       |    403.761       |     1.064   |     ms |   +0.26% |
|                                       90th percentile latency |                                 apache |    527.587       |    565.939       |    38.3529  |     ms |   +7.27% |
|                                       99th percentile latency |                                 apache |   1063.42        |   2234.87        |  1171.44    |     ms | +110.16% |
|                                      100th percentile latency |                                 apache |   1197.8         |   3002.72        |  1804.92    |     ms | +150.69% |
|                                  50th percentile service time |                                 apache |    399.323       |    399.308       |    -0.01587 |     ms |   -0.00% |
|                                  90th percentile service time |                                 apache |    478.489       |    507.146       |    28.6571  |     ms |   +5.99% |
|                                  99th percentile service time |                                 apache |   1062.43        |    990.018       |   -72.4101  |     ms |   -6.82% |
|                                 100th percentile service time |                                 apache |   1072.94        |   2285.63        |  1212.69    |     ms | +113.02% |
|                                                    error rate |                                 apache |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                            system/auth |      0.0427313   |      0.0427309   |    -0       |  ops/s |   -0.00% |
|                                               Mean Throughput |                            system/auth |      0.0845312   |      0.0841283   |    -0.0004  |  ops/s |   -0.48% |
|                                             Median Throughput |                            system/auth |      0.0742458   |      0.0726546   |    -0.00159 |  ops/s |   -2.14% |
|                                                Max Throughput |                            system/auth |      0.152622    |      0.152598    |    -2e-05   |  ops/s |   -0.02% |
|                                       50th percentile latency |                            system/auth |    425.621       |    420.478       |    -5.14297 |     ms |   -1.21% |
|                                       90th percentile latency |                            system/auth |    564.255       |    585.691       |    21.4362  |     ms |   +3.80% |
|                                       99th percentile latency |                            system/auth |    745.554       |    748.368       |     2.81351 |     ms |   +0.38% |
|                                      100th percentile latency |                            system/auth |   2069.43        |   2052.91        |   -16.5122  |     ms |   -0.80% |
|                                  50th percentile service time |                            system/auth |    420.992       |    417.734       |    -3.25795 |     ms |   -0.77% |
|                                  90th percentile service time |                            system/auth |    515.911       |    519.907       |     3.99606 |     ms |   +0.77% |
|                                  99th percentile service time |                            system/auth |    635.143       |    630.145       |    -4.99761 |     ms |   -0.79% |
|                                 100th percentile service time |                            system/auth |   2067.88        |   2051.77        |   -16.1043  |     ms |   -0.78% |
|                                                    error rate |                            system/auth |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                system/syslog/dashboard |      0.0242005   |      0.0242006   |     0       |  ops/s |    0.00% |
|                                               Mean Throughput |                system/syslog/dashboard |      0.0382994   |      0.0382988   |    -0       |  ops/s |   -0.00% |
|                                             Median Throughput |                system/syslog/dashboard |      0.0316978   |      0.0316979   |     0       |  ops/s |    0.00% |
|                                                Max Throughput |                system/syslog/dashboard |      0.0694738   |      0.0694743   |     0       |  ops/s |    0.00% |
|                                       50th percentile latency |                system/syslog/dashboard |    414.382       |    413.18        |    -1.20157 |     ms |   -0.29% |
|                                       90th percentile latency |                system/syslog/dashboard |    710.115       |    730.094       |    19.9796  |     ms |   +2.81% |
|                                      100th percentile latency |                system/syslog/dashboard |   2363.52        |   4358.08        |  1994.56    |     ms |  +84.39% |
|                                  50th percentile service time |                system/syslog/dashboard |    406.338       |    407.164       |     0.82582 |     ms |   +0.20% |
|                                  90th percentile service time |                system/syslog/dashboard |    466.415       |    453.558       |   -12.8566  |     ms |   -2.76% |
|                                 100th percentile service time |                system/syslog/dashboard |   2362.42        |   4355.89        |  1993.47    |     ms |  +84.38% |
|                                                    error rate |                system/syslog/dashboard |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                     system/syslog/lens |      0.0113572   |      0.0113578   |     0       |  ops/s |    0.00% |
|                                               Mean Throughput |                     system/syslog/lens |      0.0240039   |      0.0240045   |     0       |  ops/s |    0.00% |
|                                             Median Throughput |                     system/syslog/lens |      0.0234542   |      0.0234489   |    -1e-05   |  ops/s |   -0.02% |
|                                                Max Throughput |                     system/syslog/lens |      0.0583249   |      0.0583405   |     2e-05   |  ops/s |   +0.03% |
|                                       50th percentile latency |                     system/syslog/lens |   1207.61        |    752.114       |  -455.496   |     ms |  -37.72% |
|                                       90th percentile latency |                     system/syslog/lens |   6158.28        |   6372.18        |   213.898   |     ms |   +3.47% |
|                                      100th percentile latency |                     system/syslog/lens |  10520.8         |  10251.2         |  -269.656   |     ms |   -2.56% |
|                                  50th percentile service time |                     system/syslog/lens |    433.978       |    429.784       |    -4.19458 |     ms |   -0.97% |
|                                  90th percentile service time |                     system/syslog/lens |   3987.15        |   4311.3         |   324.15    |     ms |   +8.13% |
|                                 100th percentile service time |                     system/syslog/lens |   8202.82        |   7139.54        | -1063.28    |     ms |  -12.96% |
|                                                    error rate |                     system/syslog/lens |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                        mysql/dashboard |      0.00224979  |      0.00224995  |     0       |  ops/s |    0.01% |
|                                               Mean Throughput |                        mysql/dashboard |      0.0163243   |      0.016256    |    -7e-05   |  ops/s |   -0.42% |
|                                             Median Throughput |                        mysql/dashboard |      0.0176932   |      0.0176348   |    -6e-05   |  ops/s |   -0.33% |
|                                                Max Throughput |                        mysql/dashboard |      0.0245867   |      0.024587    |     0       |  ops/s |    0.00% |
|                                       50th percentile latency |                        mysql/dashboard |    401.275       |    397.085       |    -4.18967 |     ms |   -1.04% |
|                                       90th percentile latency |                        mysql/dashboard |    461.174       |    458.566       |    -2.60863 |     ms |   -0.57% |
|                                      100th percentile latency |                        mysql/dashboard |   4248.56        |   5149.3         |   900.734   |     ms |  +21.20% |
|                                  50th percentile service time |                        mysql/dashboard |    398.951       |    394.793       |    -4.15854 |     ms |   -1.04% |
|                                  90th percentile service time |                        mysql/dashboard |    454.797       |    434.747       |   -20.0505  |     ms |   -4.41% |
|                                 100th percentile service time |                        mysql/dashboard |   3782.91        |   4773.67        |   990.755   |     ms |  +26.19% |
|                                                    error rate |                        mysql/dashboard |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                                  redis |      0.00958383  |      0.00958404  |     0       |  ops/s |    0.00% |
|                                               Mean Throughput |                                  redis |      0.0275553   |      0.027551    |    -0       |  ops/s |   -0.02% |
|                                             Median Throughput |                                  redis |      0.0264381   |      0.0264381   |     0       |  ops/s |    0.00% |
|                                                Max Throughput |                                  redis |      0.0725947   |      0.0725957   |     0       |  ops/s |    0.00% |
|                                       50th percentile latency |                                  redis |    417.702       |    422.458       |     4.75635 |     ms |   +1.14% |
|                                       90th percentile latency |                                  redis |   1723.59        |   1885.1         |   161.508   |     ms |   +9.37% |
|                                       99th percentile latency |                                  redis |   6220.81        |   6433.67        |   212.861   |     ms |   +3.42% |
|                                      100th percentile latency |                                  redis |   7394.47        |   7611.63        |   217.154   |     ms |   +2.94% |
|                                  50th percentile service time |                                  redis |    412.661       |    415.541       |     2.87961 |     ms |   +0.70% |
|                                  90th percentile service time |                                  redis |    455.106       |    454.817       |    -0.28988 |     ms |   -0.06% |
|                                  99th percentile service time |                                  redis |   3889.22        |   4336.51        |   447.285   |     ms |  +11.50% |
|                                 100th percentile service time |                                  redis |   7392.12        |   7610.52        |   218.406   |     ms |   +2.95% |
|                                                    error rate |                                  redis |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                             mysql/lens |      0.00680615  |      0.00680621  |     0       |  ops/s |    0.00% |
|                                               Mean Throughput |                             mysql/lens |      0.0164084   |      0.0163666   |    -4e-05   |  ops/s |   -0.25% |
|                                             Median Throughput |                             mysql/lens |      0.0127808   |      0.0127902   |     1e-05   |  ops/s |   +0.07% |
|                                                Max Throughput |                             mysql/lens |      0.0469888   |      0.0469951   |     1e-05   |  ops/s |   +0.01% |
|                                       50th percentile latency |                             mysql/lens |    404.016       |    402.003       |    -2.01257 |     ms |   -0.50% |
|                                       90th percentile latency |                             mysql/lens |   4300.69        |   3737.38        |  -563.313   |     ms |  -13.10% |
|                                      100th percentile latency |                             mysql/lens |   5224.48        |   5203.7         |   -20.7812  |     ms |   -0.40% |
|                                  50th percentile service time |                             mysql/lens |    400.352       |    399.291       |    -1.06073 |     ms |   -0.26% |
|                                  90th percentile service time |                             mysql/lens |   2829           |   2744.07        |   -84.9239  |     ms |   -3.00% |
|                                 100th percentile service time |                             mysql/lens |   5222.35        |   5201.7         |   -20.6519  |     ms |   -0.40% |
|                                                    error rate |                             mysql/lens |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                    postgresql/overview |      0.00513509  |      0.0051353   |     0       |  ops/s |    0.00% |
|                                               Mean Throughput |                    postgresql/overview |      0.061792    |      0.0618321   |     4e-05   |  ops/s |   +0.06% |
|                                             Median Throughput |                    postgresql/overview |      0.0631799   |      0.0631955   |     2e-05   |  ops/s |   +0.02% |
|                                                Max Throughput |                    postgresql/overview |      0.0893931   |      0.0893947   |     0       |  ops/s |    0.00% |
|                                       50th percentile latency |                    postgresql/overview |   1230.84        |   1159.25        |   -71.5958  |     ms |   -5.82% |
|                                       90th percentile latency |                    postgresql/overview |   2272.46        |   1904.9         |  -367.56    |     ms |  -16.17% |
|                                       99th percentile latency |                    postgresql/overview |   6530.01        |   3554.92        | -2975.08    |     ms |  -45.56% |
|                                      100th percentile latency |                    postgresql/overview |   7323.11        |   4446.62        | -2876.5     |     ms |  -39.28% |
|                                  50th percentile service time |                    postgresql/overview |   1153.96        |   1127.72        |   -26.2367  |     ms |   -2.27% |
|                                  90th percentile service time |                    postgresql/overview |   1547.05        |   1543.46        |    -3.58295 |     ms |   -0.23% |
|                                  99th percentile service time |                    postgresql/overview |   4738.04        |   2405.06        | -2332.97    |     ms |  -49.24% |
|                                 100th percentile service time |                    postgresql/overview |   7163.95        |   3250.68        | -3913.26    |     ms |  -54.62% |
|                                                    error rate |                    postgresql/overview |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                    postgresql/duration |      0.0224523   |      0.0224591   |     1e-05   |  ops/s |   +0.03% |
|                                               Mean Throughput |                    postgresql/duration |      0.0365889   |      0.0360483   |    -0.00054 |  ops/s |   -1.48% |
|                                             Median Throughput |                    postgresql/duration |      0.0302364   |      0.0300863   |    -0.00015 |  ops/s |   -0.50% |
|                                                Max Throughput |                    postgresql/duration |      0.0801545   |      0.0802444   |     9e-05   |  ops/s |   +0.11% |
|                                       50th percentile latency |                    postgresql/duration |    416.413       |    413.75        |    -2.66229 |     ms |   -0.64% |
|                                       90th percentile latency |                    postgresql/duration |    801.768       |    726.736       |   -75.0322  |     ms |   -9.36% |
|                                      100th percentile latency |                    postgresql/duration |   3985.51        |   2733.46        | -1252.06    |     ms |  -31.42% |
|                                  50th percentile service time |                    postgresql/duration |    410.434       |    409.814       |    -0.61966 |     ms |   -0.15% |
|                                  90th percentile service time |                    postgresql/duration |    495.168       |    492.75        |    -2.41761 |     ms |   -0.49% |
|                                 100th percentile service time |                    postgresql/duration |   3984.41        |   2658.64        | -1325.77    |     ms |  -33.27% |
|                                                    error rate |                    postgresql/duration |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                             bulk-index |    207.92        |     30.5598      |  -177.36    | docs/s |  -85.30% |
|                                               Mean Throughput |                             bulk-index |  58833.2         |  54130           | -4703.28    | docs/s |   -7.99% |
|                                             Median Throughput |                             bulk-index |  59610.4         |  56925.2         | -2685.21    | docs/s |   -4.50% |
|                                                Max Throughput |                             bulk-index |  60014.6         |  58411.2         | -1603.49    | docs/s |   -2.67% |
|                                       50th percentile latency |                             bulk-index |    323.108       |    330.28        |     7.17122 |     ms |   +2.22% |
|                                       90th percentile latency |                             bulk-index |    559.409       |    566.228       |     6.81899 |     ms |   +1.22% |
|                                       99th percentile latency |                             bulk-index |    947.029       |    898.712       |   -48.3165  |     ms |   -5.10% |
|                                     99.9th percentile latency |                             bulk-index |   3474.96        |   3168.35        |  -306.608   |     ms |   -8.82% |
|                                    99.99th percentile latency |                             bulk-index |   5629.83        |  27283.1         | 21653.3     |     ms | +384.62% |
|                                      100th percentile latency |                             bulk-index |   8841.69        |  32663.5         | 23821.8     |     ms | +269.43% |
|                                  50th percentile service time |                             bulk-index |    323.951       |    329.281       |     5.33038 |     ms |   +1.65% |
|                                  90th percentile service time |                             bulk-index |    566.043       |    564.474       |    -1.56934 |     ms |   -0.28% |
|                                  99th percentile service time |                             bulk-index |    945.919       |    899.869       |   -46.0503  |     ms |   -4.87% |
|                                99.9th percentile service time |                             bulk-index |   3477.48        |   3162.67        |  -314.811   |     ms |   -9.05% |
|                               99.99th percentile service time |                             bulk-index |   5605.52        |  26956.8         | 21351.3     |     ms | +380.90% |
|                                 100th percentile service time |                             bulk-index |   8841.69        |  32663.5         | 23821.8     |     ms | +269.43% |
|                                                    error rate |                             bulk-index |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                             field-caps |      2.42492     |      2.21015     |    -0.21477 |  ops/s |   -8.86% |
|                                               Mean Throughput |                             field-caps |      3.02836     |      2.82335     |    -0.20501 |  ops/s |   -6.77% |
|                                             Median Throughput |                             field-caps |      3.08178     |      2.89666     |    -0.18512 |  ops/s |   -6.01% |
|                                                Max Throughput |                             field-caps |      3.14591     |      2.95005     |    -0.19586 |  ops/s |   -6.23% |
|                                       50th percentile latency |                             field-caps |    606.82        |    647.399       |    40.5797  |     ms |   +6.69% |
|                                       90th percentile latency |                             field-caps |    672.35        |    685.512       |    13.1619  |     ms |   +1.96% |
|                                       99th percentile latency |                             field-caps |    694.742       |    770.939       |    76.1967  |     ms |  +10.97% |
|                                      100th percentile latency |                             field-caps |    763.887       |    853.836       |    89.9483  |     ms |  +11.78% |
|                                  50th percentile service time |                             field-caps |    606.82        |    647.399       |    40.5797  |     ms |   +6.69% |
|                                  90th percentile service time |                             field-caps |    672.35        |    685.512       |    13.1619  |     ms |   +1.96% |
|                                  99th percentile service time |                             field-caps |    694.742       |    770.939       |    76.1967  |     ms |  +10.97% |
|                                 100th percentile service time |                             field-caps |    763.887       |    853.836       |    89.9483  |     ms |  +11.78% |
|                                                    error rate |                             field-caps |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |        field-caps-exclude-empty-fields |      0.354446    |      0.372199    |     0.01775 |  ops/s |   +5.01% |
|                                               Mean Throughput |        field-caps-exclude-empty-fields |      0.37615     |      0.393143    |     0.01699 |  ops/s |   +4.52% |
|                                             Median Throughput |        field-caps-exclude-empty-fields |      0.377413    |      0.394415    |     0.017   |  ops/s |   +4.50% |
|                                                Max Throughput |        field-caps-exclude-empty-fields |      0.378146    |      0.395655    |     0.01751 |  ops/s |   +4.63% |
|                                       50th percentile latency |        field-caps-exclude-empty-fields |   5292.47        |   5042.15        |  -250.313   |     ms |   -4.73% |
|                                       90th percentile latency |        field-caps-exclude-empty-fields |   5363.24        |   5100.19        |  -263.051   |     ms |   -4.90% |
|                                       99th percentile latency |        field-caps-exclude-empty-fields |   5404.22        |   5153.63        |  -250.591   |     ms |   -4.64% |
|                                      100th percentile latency |        field-caps-exclude-empty-fields |   5427.63        |   5235.53        |  -192.098   |     ms |   -3.54% |
|                                  50th percentile service time |        field-caps-exclude-empty-fields |   5292.47        |   5042.15        |  -250.313   |     ms |   -4.73% |
|                                  90th percentile service time |        field-caps-exclude-empty-fields |   5363.24        |   5100.19        |  -263.051   |     ms |   -4.90% |
|                                  99th percentile service time |        field-caps-exclude-empty-fields |   5404.22        |   5153.63        |  -250.591   |     ms |   -4.64% |
|                                 100th percentile service time |        field-caps-exclude-empty-fields |   5427.63        |   5235.53        |  -192.098   |     ms |   -3.54% |
|                                                    error rate |        field-caps-exclude-empty-fields |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                field-caps-index-filter |      2.09805     |      1.93966     |    -0.15839 |  ops/s |   -7.55% |
|                                               Mean Throughput |                field-caps-index-filter |      2.20143     |      2.06766     |    -0.13377 |  ops/s |   -6.08% |
|                                             Median Throughput |                field-caps-index-filter |      2.20826     |      2.07898     |    -0.12927 |  ops/s |   -5.85% |
|                                                Max Throughput |                field-caps-index-filter |      2.22311     |      2.0895      |    -0.13361 |  ops/s |   -6.01% |
|                                       50th percentile latency |                field-caps-index-filter |    890.331       |    935.82        |    45.4898  |     ms |   +5.11% |
|                                       90th percentile latency |                field-caps-index-filter |    961.78        |   1038.12        |    76.3429  |     ms |   +7.94% |
|                                       99th percentile latency |                field-caps-index-filter |   1098.11        |   1154.67        |    56.5621  |     ms |   +5.15% |
|                                      100th percentile latency |                field-caps-index-filter |   1171.06        |   1274.29        |   103.224   |     ms |   +8.81% |
|                                  50th percentile service time |                field-caps-index-filter |    890.331       |    935.82        |    45.4898  |     ms |   +5.11% |
|                                  90th percentile service time |                field-caps-index-filter |    961.78        |   1038.12        |    76.3429  |     ms |   +7.94% |
|                                  99th percentile service time |                field-caps-index-filter |   1098.11        |   1154.67        |    56.5621  |     ms |   +5.15% |
|                                 100th percentile service time |                field-caps-index-filter |   1171.06        |   1274.29        |   103.224   |     ms |   +8.81% |
|                                                    error rate |                field-caps-index-filter |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |   field-caps-index-filter-empty-fields |      0.315071    |      0.33431     |     0.01924 |  ops/s |   +6.11% |
|                                               Mean Throughput |   field-caps-index-filter-empty-fields |      0.328892    |      0.345836    |     0.01694 |  ops/s |   +5.15% |
|                                             Median Throughput |   field-caps-index-filter-empty-fields |      0.329896    |      0.346685    |     0.01679 |  ops/s |   +5.09% |
|                                                Max Throughput |   field-caps-index-filter-empty-fields |      0.330953    |      0.348813    |     0.01786 |  ops/s |   +5.40% |
|                                       50th percentile latency |   field-caps-index-filter-empty-fields |   6043.94        |   5737.52        |  -306.421   |     ms |   -5.07% |
|                                       90th percentile latency |   field-caps-index-filter-empty-fields |   6130.38        |   5845.22        |  -285.157   |     ms |   -4.65% |
|                                       99th percentile latency |   field-caps-index-filter-empty-fields |   6232.07        |   5993.56        |  -238.514   |     ms |   -3.83% |
|                                      100th percentile latency |   field-caps-index-filter-empty-fields |   6289.37        |   6087.26        |  -202.112   |     ms |   -3.21% |
|                                  50th percentile service time |   field-caps-index-filter-empty-fields |   6043.94        |   5737.52        |  -306.421   |     ms |   -5.07% |
|                                  90th percentile service time |   field-caps-index-filter-empty-fields |   6130.38        |   5845.22        |  -285.157   |     ms |   -4.65% |
|                                  99th percentile service time |   field-caps-index-filter-empty-fields |   6232.07        |   5993.56        |  -238.514   |     ms |   -3.83% |
|                                 100th percentile service time |   field-caps-index-filter-empty-fields |   6289.37        |   6087.26        |  -202.112   |     ms |   -3.21% |
|                                                    error rate |   field-caps-index-filter-empty-fields |      0           |      0           |     0       |      % |    0.00% |
|                                                Min Throughput |                 esql_from_all_limit_50 |      0.388699    |      0.356429    |    -0.03227 |  ops/s |   -8.30% |
|                                               Mean Throughput |                 esql_from_all_limit_50 |      0.45163     |      0.413915    |    -0.03772 |  ops/s |   -8.35% |
|                                             Median Throughput |                 esql_from_all_limit_50 |      0.461227    |      0.421679    |    -0.03955 |  ops/s |   -8.57% |
|                                                Max Throughput |                 esql_from_all_limit_50 |      0.484858    |      0.44865     |    -0.03621 |  ops/s |   -7.47% |
|                                       50th percentile latency |                 esql_from_all_limit_50 |   1913.89        |   1994.26        |    80.3652  |     ms |   +4.20% |
|                                       90th percentile latency |                 esql_from_all_limit_50 |   2073.65        |   2233.39        |   159.743   |     ms |   +7.70% |
|                                      100th percentile latency |                 esql_from_all_limit_50 |   2182.14        |   2409.2         |   227.058   |     ms |  +10.41% |
|                                  50th percentile service time |                 esql_from_all_limit_50 |   1913.89        |   1994.26        |    80.3652  |     ms |   +4.20% |
|                                  90th percentile service time |                 esql_from_all_limit_50 |   2073.65        |   2233.39        |   159.743   |     ms |   +7.70% |
|                                 100th percentile service time |                 esql_from_all_limit_50 |   2182.14        |   2409.2         |   227.058   |     ms |  +10.41% |
|                                                    error rate |                 esql_from_all_limit_50 |      0           |      0           |     0       |      % |    0.00% |