Alertmanager: Initialize skipped Grafana Alertmanagers receiving requests

[santihernandezc]

Description

If the grafana_alertmanager_conditionally_skip_tenant_suffix option is configured and a Grafana Alertmanager tenant doesn't have a promoted, non-default, non-empty configuration, we skip initializing it.

The problem is that clients making requests against an uninitialized Alertmanager get 406s. Unless a Grafana Alertmanager has a "usable" configuration, users won't be able to test templates, test and get receivers, create silences, etc.

This PR makes the multi-tenant Alertmanager start per-tenant Grafana Alertmanagers on incoming requests. This way, requests can be handled even if tenants were initially skipped.

It also adds a grace period for idle Alertmanagers. Whenever a skipped Alertmanager gets a request, we start the Alertmanager and keep a timestamp indicating when this request was received. After the grace period elapses, we shut down the Alertmanager.

Testing

I tested this PR by spinning up two Alertmanagers (read-write mode) with:

• multitenancy_enabled: true
• grafana_alertmanager_compatibility_enabled: true
• grafana_alertmanager_conditionally_skip_tenant_suffix: -grafana

I then created 200 Alertmanager tenants with empty configurations:

• 100 of them not matching the configured suffix and using an empty configuration
• 99 of them matching the suffix and using an empty configurations
• 1 of them matching the suffix and using a promoted, non-default, non-empty config

The 99 tenants with empty configuration and matching the suffix were initially skipped. I then sent test alerts for each tenant matching the suffix. Alertmanagers for each of them were started.

After the default grace period passed, all Alertmanagers for tenants matching the configured suffix were stopped, except the one using a promoted, non-default, non-empty configuration.

Query for tenants skipped per instance

Query for active Alertmanagers by type (Mimir/Grafana)

[github-actions]

💻 Deploy preview available: https://deploy-preview-mimir-10691-zb444pucvq-vp.a.run.app/docs/mimir/latest/

[santihernandezc]

Unrelated fix, this was not being used here.

[tacole02]

Docs look great! Thank you!

[JacobsonMT]

Concern: This method is doing a lot and appears to have code that doesn't match our current -grafana tenant strategy. For example, it's not clear to me what the following is meant to be doing anymore and why we don't need it in the new startAlertmanager code:

	if cfgs.Mimir.RawConfig == am.fallbackConfig || cfgs.Mimir.RawConfig == "" {
		level.Debug(am.logger).Log("msg", "using grafana config with the default globals", "user", cfgs.Mimir.User)
		cfg, err := am.createUsableGrafanaConfig(cfgs.Grafana, am.fallbackConfig)
		return cfg, true, err
	}

If it is in fact, not needed in startAlertmanager we might want to start cleaning up some of the redundant code that doesn't fit the current strategy. Or at least extract it somewhere where it is clear it's not part of the functional flow.

[santihernandezc]

That bit of code gets executed after we've checked for usable (promoted, non-default, non-empty) Grafana configuration. If we reach this far, the tenant has a Grafana configuration we can use to start their Alertmanager.

It does indeed not match our current approach. It's part of the original one, where Grafana and non-Grafana tenants were the same, and they couldn't be distinguished by a suffix.

It's not necessary to add computeConfig() to the startAlertmanager() function as it will only be called for skipped Grafana Alertmanagers. If a Grafana Alertmanager was skipped, it had no usable configuration, so we can just use a default config.

[JacobsonMT]

I think this method can be combined with alertmanagerFromFallbackConfig, they are effectively doing the same thing. Probably keep the startAlertmanager name and make sure the correct errors are returned from the combined method.

[santihernandezc]

I didn't want to change the behavior of Mimir Alertmanager tenants, at least not yet. I was planning on doing that in a future PR, when applying strict initialization to all tenants.

[JacobsonMT]

This seems like a more interesting metric to collect: When a skipped config was initialized because of a request. This will give us a feel for flappiness and how effectively our idle timeout value is.