linkerd2 Monitoring Mixin for Grafana

This mixin is a jsonnet package which can be used standalone or as part of your own config.

It generates Grafana dashboards for linkerd2 monitoring and can work in standalone (default) or in multi cluster setups.

How to use

This mixin is designed to be vendored into the repo with your infrastructure config.

Generating config files

To manually generate the dashboards, you can - but it requires some additional tooling:

Get the jsonnet-bundler and jsonnet itself:

go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb
brew install jsonnet

Then grab the mixin and any dependencies:

git clone https://github.com/andrew-waters/linkerd2-mixin
cd linkerd2-mixin
jb install

Then you can build the mixin:

make dashboards

Multi-cluster support

This mixin can support a set of dashboards that can used for multiple clusters (if your metrics are aggregated and cluster labeled).

If you're using kube-prometheus, you can label your cluster by adding the following:

{
  _config+:: {
    clusterName: 'my-cluster-name',
  },
  prometheus+:: {
    prometheus+: {
      spec+: {
        externalLabels: {
          cluster: $._config.clusterName
        },
      },
    },
  },
}

Once your metrics are labelled by cluster, they will be available for filtering within PromQL.

You can now take advantage of these labels within the mixin by using the following (replace <clusterLabel> with the key of the cluster label, usually cluster):

{
  linkerd+:: {
    multiCluster: {
      enabled: true,
      label: '<clusterLabel>,
    },
  },
}

Customising the mixin

The linkerd2-mixin also allows you to customise the name and tags for your dashbaords, which is very valuable when you are monitoring more than just linkerd2.

linkerd2 {
  _config+:: {
    multiCluster: {
      enabled: true,
      label: 'cluster',
    },
    dashboard: {
      namePrefix: 'Linkerd2 / ',
      tags: ['linkerd', 'infrastucture'],
    }
  },
}

Using in an existing project

If you have an existing project that you want to use this mixin with, you can use it like the following:

local c =
  (import 'linkerd2-mixin/mixin.libsonnet') +
  {
    linkerd+:: {
      datasource: 'prometheus',
      // ... all other config options
    },
  };

{ 'linkerd-dashboards': c.linkerd.dashboards }

Adds

• Multi cluster support for a single pane of glass
• Interval variable for custom scrape frequencies
• Loki logs

Intervals and scrape frequencies

Generally speaking, if you're running a multi cluster setup, you're going to be scraping a bit slower than Linkerd's Prometheus will be. This breaks some graphs which depend on Range Vector selectors with 30s intervals.

To counter this, there is an additional variable on each dashboard if you don't want to update your scrape freqency called interval. In most cases, setting this to 1m (one minute) will be enough to visualise requests over time, but your mileage will vary.

Todo

Javascript in common.header to check most recent linkerd version.

Add Loki logs