Expose per-resource reconciliation status as a Prometheus metric #4808

Describe the feature you are requesting

A per-resource reconciliation-status metric that includes the resource's namespace and name, so Prometheus can tell which resource is failing to reconcile — not just that some reconcile in a controller is failing.

Today the reconcile-error metrics are labeled only by controller/error type:

• controller_runtime_reconcile_errors_total{controller}
• awslbc_controller_reconcile_errors_total{controller, error_category}

So you can see that controller="ingress" reconciles are erroring, but not which Ingress. The resource identity currently only exists in the controller logs and in the FailedBuildModel / FailedUpdateStatus Kubernetes Events.

Proposed metric:

awslbc_reconcile_condition{controller, namespace, name, status}   # 0 | 1

Motivation

We alert on reconcile failures, but because the current metrics don't carry the resource identity, the alerts can't say which Ingress/Service is affected or be routed to the owning team. A per-resource status metric lets us build alerts keyed on namespace/name (and route them based on those labels), instead of relying on log scraping or event exporters.

Describe the proposed solution you'd like

A bounded per-resource gauge — one series per reconciled object:

awslbc_reconcile_condition{controller, namespace, name, status} = 0 | 1

• controller: ingress | service | targetGroupBinding | ...
• namespace, name: the reconciled resource
• status: the reconcile condition (e.g. True/False), value 1 for the current state

The existing awslbc_controller_reconcile_errors_total{controller, error_category} would stay as-is for the aggregate "how many / what category" view; this is an additional, complementary metric.

This follows the same per-resource status-condition shape already used across the ecosystem:

• kube-state-metrics — kube_deployment_status_condition{deployment, namespace, condition, status}
• Flux — gotk_reconcile_condition{kind, name, namespace, type, status}
• cert-manager — certmanager_certificate_ready_status{name, namespace, condition, ...}

and is consistent with this project's own per-resource awslbc_top_talkers{controller, namespace, name}.

Simpler alternative shape (no status label)

The status-label shape above matches the kube-state-metrics convention, but it emits two series per resource and requires zeroing out the opposite status on every transition. A simpler, lower-cardinality option is a single gauge without the status label:

awslbc_reconcile_condition{controller, namespace, name} = 1 (reconciled) | 0 (failing)

• Half the series (one per resource instead of two), and no need to clear the opposite status value on each flip.
• Alerting is a plain == 0 instead of filtering on status="False".
• Trade-off: the KSM status-label shape is what many off-the-shelf alerting/mixin libraries expect, so the single-gauge form is slightly less "standard".

Either shape works and shares the same implementation surface; noting it here since it may be simpler to implement and reason about.

Regardless of shape, the series for a resource must be cleared (DeleteLabelValues / DeletePartialMatch) when the resource is deleted, so the metric doesn't report a frozen state forever and cardinality stays bounded to live resources.

Describe alternatives you've considered

• Parsing the controller logs — the resource and reason are there, but it isn't a metric.
• kubernetes-event-exporter on the FailedBuildModel/FailedUpdateStatus Events — works, but adds a component and Events are ephemeral.

Contribution Intention (Optional)

• Yes, I am willing to contribute a PR to implement this feature
• No, I cannot work on a PR at this time