Namespace deletion is stuck when using namespace selector startup mode

[nicolasadeo]

Long story short

When running a kopf operator with namespace selector AND deleting this namespace that contains a resource with a finalizer: kopf may not call the delete handler method.

Deleting the namespace stops the watch-stream for this namespace before the resources contained in that namespace get the chance have its delete method to be called.

Therefore, the finalizer is never removed and the namespace deletion is stuck waiting all resources to be deleted (but they won't since the watch-stream is stopped).

We manage to find a workaround using the clusterwide setting.

Earlier, we say "may not call"; in some cases the delete handler is called but it is quickly canceled by a CancelledError coming from kopf. We suppose it may be relatee to the event queue size. In the simple example provided, we never manage to get the delete handler to be called.

Kopf version

1.36.2

Kubernetes version

1.27.4

Python version

3.11.1

Code

import kopf
import asyncio
import threading
import logging
from kopf._core.actions import loggers


@kopf.on.create('kopfexamples')
def create_fn(**kwargs):
    print('CREATED!!!')


@kopf.on.delete('kopfexamples')
async def delete_fn(**kwargs):
    await asyncio.sleep(2)
    print(f'DELETED!!!')


@kopf.on.startup()  # type: ignore
def configure(settings: kopf.OperatorSettings, **_) -> None:
    settings.posting.level = logging.DEBUG
    loggers.configure(
        debug=True,
        verbose=False,
        quiet=False,
        log_format=loggers.LogFormat.PLAIN,  # loggers.LogFormat.JSON,
        log_refkey="k8s_obj",
        log_prefix=False,
    )

def kopf_thread():
    asyncio.run(kopf.operator(namespaces=["example"]))


def main():
    logging.getLogger("kopf").setLevel(logging.DEBUG)
    thread = threading.Thread(target=kopf_thread)
    thread.start()
    thread.join()


if __name__ == "__main__":
    main()

Logs

Adding the finalizer, thus preventing the actual deletion.
Patching with: {'metadata': {'finalizers': ['kopf.zalando.org/KopfFinalizerMarker']}}
Creation is in progress: {'apiVersion': 'kopf.dev/v1', 'kind': 'KopfExample', 'metadata': {'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"kopf.dev/v1","kind":"KopfExample","metadata":{"annotations":{"someannotation":"somevalue"},"labels":{"somelabel":"somevalue"},"name":"kopf-example-1","namespace":"example"},"spec":{"duration":"1m","field":"value","items":["item1","item2"]}}\n', 'someannotation': 'somevalue'}, 'creationTimestamp': '2024-01-10T15:07:37Z', 'finalizers': ['kopf.zalando.org/KopfFinalizerMarker'], 'generation': 1, 'labels': {'somelabel': 'somevalue'}, 'managedFields': [{'apiVersion': 'kopf.dev/v1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:finalizers': {'.': {}, 'v:"kopf.zalando.org/KopfFinalizerMarker"': {}}}}, 'manager': 'kopf', 'operation': 'Update', 'time': '2024-01-10T15:07:37Z'}, {'apiVersion': 'kopf.dev/v1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:kubectl.kubernetes.io/last-applied-configuration': {}, 'f:someannotation': {}}, 'f:labels': {'.': {}, 'f:somelabel': {}}}, 'f:spec': {'.': {}, 'f:duration': {}, 'f:field': {}, 'f:items': {}}}, 'manager': 'kubectl-client-side-apply', 'operation': 'Update', 'time': '2024-01-10T15:07:37Z'}], 'name': 'kopf-example-1', 'namespace': 'example', 'resourceVersion': '334932', 'uid': '829ca9ed-c7a5-4426-baab-7b3762f9b7fb'}, 'spec': {'duration': '1m', 'field': 'value', 'items': ['item1', 'item2']}}
Handler 'create_fn' is invoked.
Handler 'create_fn' succeeded.
Creation is processed: 1 succeeded; 0 failed.
Patching with: {'metadata': {'annotations': {'kopf.zalando.org/last-handled-configuration': '{"spec":{"duration":"1m","field":"value","items":["item1","item2"]},"metadata":{"labels":{"somelabel":"somevalue"},"annotations":{"someannotation":"somevalue"}}}\n'}}}
CREATED!!!
Something has changed, but we are not interested (the essence is the same).
Handling cycle is finished, waiting for new changes.
Stopping the watch-stream for kopfexamples.v1.kopf.dev in 'example'.

Additional information

After some code digging trying to understand what's going on, we think the problem is here and the namespace is removed from the watching list without any check:

kopf/kopf/_core/reactor/observation.py

Line 202 in 91e8fd6

if deleted:

[nicolasadeo]

more code snippets.

The CRD we use:

# A demo CRD for the Kopf example operators.
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: kopfexamples.kopf.dev
spec:
  scope: Namespaced
  group: kopf.dev
  names:
    kind: KopfExample
    plural: kopfexamples
    singular: kopfexample
    shortNames:
      - kopfexes
      - kopfex
      - kexes
      - kex
  versions:
    - name: v1
      served: true
      storage: true
      subresources: { status: { } }  # comment/uncomment for experiments
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              x-kubernetes-preserve-unknown-fields: true
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
        - name: Duration
          type: string
          priority: 0
          jsonPath: .spec.duration
          description: For how long the pod should sleep.
        - name: Children
          type: string
          priority: 0
          jsonPath: .status.create_fn.children
          description: The children pods created.
        - name: Message
          type: string
          priority: 0
          jsonPath: .status.create_fn.message
          description: As returned from the handler (sometimes).

and the resource manifest:

apiVersion: kopf.dev/v1
kind: KopfExample
metadata:
  name: kopf-example-1
  namespace: example
  labels:
    somelabel: somevalue
  annotations:
    someannotation: somevalue
spec:
  duration: 1m
  field: value
  items:
  - item1
  - item2