Unable to delete DBInstance in AWS using deletion policy delete on my DBInstance CRD

[ferozemohideen]

Describe the bug
I am installing my RDS controller using https://github.com/aws-controllers-k8s/ack-chart (image: public.ecr.aws/aws-controllers-k8s/rds-controller:1.2.1) with the following values:

rds:
  enabled: true
  deployment:
    replicas: 1
  aws:
    region: us-east-1 
  deletionPolicy: retain
  resources:
    requests:
      memory: "64Mi"
      cpu: "50m"
    limits:
      memory: "128Mi"
      cpu: "100m"

I include deletionPolicy: retain for general safety. I provision a DBInstance using the following file:

apiVersion: rds.services.k8s.aws/v1alpha1
kind: DBInstance
metadata:
  name: "test-db"
  namespace: "<aws-account-id>"
  annotations:
    services.k8s.aws/region: "<aws-region>"
spec:
  dbInstanceClass: db.t4g.medium
  dbInstanceIdentifier: "test-db"
  engine: postgres
  engineVersion: "15.3"
  allocatedStorage: 30
  masterUsername: postgres
  masterUserPassword:
    namespace: default
    name: "test-secret"
    key: password

I would like to override the deletion policy on the controller, such that when I delete this DBInstance, it deletes the resource in AWS. Based on these docs, I tried adding a services.k8s.aws/deletion-policy: delete annotation on by DBInstance CRD first, then deleting it, but doing so still does not clean up the DBInstance in AWS.

I was able to achieve the behavior I desired by restarting the RDS controller after I add the annotation, and then deleting, after which I saw the resource delete in AWS. I believe the bug here is that the CRD does not immediately reconcile after I add this annotation.

Steps to reproduce
Install the chart
Expected outcome
A concise description of what you expected to happen.

Environment

• Kubernetes version
  1.24
• Using EKS (yes/no), if so version?
  yes, 1.24
• AWS service targeted (S3, RDS, etc.)
  RDS

[ack-bot]

Issues go stale after 180d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 60d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle stale