Enabling option to ignore namespaces

[enxebre]

What would you like to be added (User Story)?

As a cluster service provider managing a multi-tenant environment I would like to setup my capi controllers to ignore some namespaces

Detailed Description

In a multi-tenant environment where multiple capi controllers are running I'd like to ignore some well known namespaces that might be watched by different tenant.
This should be possible by exposing a new option for "excluded-namespaces" and let the manager cache to filter out by field

			DefaultFieldSelector: fields.AndSelectors(
				fields.OneTermNotEqualSelector("metadata.namespace", "foo"),
				fields.OneTermNotEqualSelector("metadata.namespace", "bar")),

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPI contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[nrb]

Would you like to see this in addition to #11192?

The options appear mutually exclusive in use. Otherwise, there would need to be logic to handle/error when a namespace appears in both sets.

[sbueringer]

As of today we have --watch-filter and --namespace. --watch-filter is used for event filters, while --namespace is used for the cache configuration.

I think overall it comes down to figuring out what exactly we want to configure on the controller-runtime level (cache configuration, event filters & on which objects) and then based on that model it as command-line flags.

It's also important to figure out what to configure in the cache configuration vs what we are filtering out later, because if the cache is watching on a namespace, it also needs permissions for that namespace.

We also have to make sure that the options we provide via command-line flags don't restrict us in how we can internally optimize the cache configuration for good performance at scale.

fyi we had a similar previous request here: #7775
Also related #11192

[JoelSpeed]

Looking at @enxebre's suggestion, isn't this an optimisation of the caches by itself? His suggested implementation puts the onus on the API server to filter requests, and not our controllers.

If you aren't using the --namespace flag today, you need RBAC to watch on all namespaces, so reducing the watched namespaces provides an opportunity to reduce the RBAC scope, it wouldn't need to increase it.

[fabriziopandini]

Following up to the discussion came up at the office hours, e.g #11395 is another PR/feature that might be impacted by the whole idea of restrict controllers to work on a subset of namespaces.

Overall, before thinking at flags, cache etc. I think we should think about tenancy and re-validate the current project stance in https://cluster-api.sigs.k8s.io/developer/architecture/controllers/multi-tenancy and in https://cluster-api.sigs.k8s.io/developer/architecture/controllers/support-multiple-instances (which have been used all across the codebase in the last 3/4 years)