Support option to mirror requests to a secondary Cortex

[locmai]

Propose for an option for mirroring requests to a secondary Cortex cluster following the guide at operations/requests-mirroring-to-secondary-cluster.

The new setup will look like:

• Both the primary and secondary clusters will be released with the Cortex chart (2 different Helm releases).
• On the primary, we will add an option to enable the mirroring with envoy proxy by specifying the primary's distributor service and the secondary ingress/service.
• On the secondary, we will add the option to expose distributor with ingress.

This feature request targets very specific use cases, such as setting up disaster recovery for Cortex or performing cluster migrations. Therefore, I would understand if we decide not to include it here or if further discussion is necessary.

Proposed new resources/changes:

• Deployment for a standalone envoy proxy
• Configuration for the proxy to dual-write to primary and secondary clusters
• Optional ingress for cortex-distributor
• Configurable for routing requests to /api/prom/push and /api/v1/push to the envoy proxy instead

[kd7lxl]

This seems like a valuable feature, but could it be done as its own chart? Maybe even genetically?

[locmai]

I agree on having it on it's own chart.

We are doing this in our environment:

• Then customize the push endpoints in this chart
• Have standalone envoy proxy Helm chart

umbrella chart
|_ cortex (this)
|_ envoy

The lines we added to wrap around these lines

cortex-helm-chart/templates/nginx/nginx-config.yaml

Lines 91 to 98 in 6ecdcfd

	location = /api/prom/push {
	proxy_pass {{ .Values.nginx.config.upstream_protocol }}://{{ template "cortex.fullname" . }}-distributor.{{ $rootDomain }}$request_uri;
	}
	## New Remote write API. Ref: https://cortexmetrics.io/docs/api/#remote-write
	location = /api/v1/push {
	proxy_pass {{ .Values.nginx.config.upstream_protocol }}://{{ template "cortex.fullname" . }}-distributor.{{ $rootDomain }}$request_uri;
	}

        {{- if .Values.nginx.config.push_endpoint.override }}
        location = /api/prom/push {
          proxy_pass      {{ .Values.nginx.config.upstream_protocol }}://{{ .Values.nginx.config.push_endpoint.url }}$request_uri;
        }
        location = /api/v1/push {
          proxy_pass      {{ .Values.nginx.config.upstream_protocol }}://{{ .Values.nginx.config.push_endpoint.url }}$request_uri;
        }
        {{- else }}
        ## The https://github.com/cortexproject/cortex-helm-chart/blob/6ecdcfd095fc95e320a4c404f149ff3f9d0b238d/templates/nginx/nginx-config.yaml#L91-L98
        {{- end }}