Controllers Interrupted During Reconcile Create Duplicate Resources

[nromriell]

Edited: Removed ec2 specification, Upon further investigation this would be true for all controllers

Describe the bug
During the reconcile of a VPC if the controller is interrupted before the reconcile completes, on restart the VPC is created a second time.

This is because the status on the VPC is empty and not updated to track the remote resource until after reconciliation is completed. This leads to orphaned duplicate resources

Steps to reproduce

• Create a new VPC object

apiVersion: ec2.services.k8s.aws/v1alpha1
kind: VPC
metadata:
  name: example
spec:
  ...

• As soon as the creation for the vpc is trigger on the remote, stop or restart the controller
  • alternatively place an interrupt inside the code between create and object update https://github.com/aws-controllers-k8s/ec2-controller/blob/main/pkg/resource/vpc/sdk.go#L266

       resp, err = rm.sdkapi.CreateVpcWithContext(ctx, input)
       rm.metrics.RecordAPICall("CREATE", "CreateVpc", err)
       if err != nil {
           return nil, err
       }
       panic("interrupt to orphan")
    

• Check the the status on the vpc, if the condition has triggered the status will still be empty with no reference to the remote
• When the controller runs the reconcile again a new object with the same information will be created

This was found because of a panic on the controller causing a restart during the reconcile of the VPC causing the interrupt during creation

{"level":"info","ts":"2024-04-24T18:00:29.176Z","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"elasticipaddress",...}

Expected outcome
VPC status is tracked as pending before final status update on crd after creation is complete. The controller would attempt to then reconcile with remote before creating a new object

Environment
EKS version: eks.15
Kubernetes Version: 1.27
Service Targeted: EC2
EC2 ACK Controller Version: 1.2.6

[nromriell]

A potential option:

• Update status to pending before running create api call
• Automatically add initial tag to AWS resource matching uid of crd object manifest as part of api create call
• During reconciles, if the status of an object is pending attempt to reconcile from remote first before just reverting to creation of new item

[nromriell]

Found during crashloop caused by #2063

[a-hilaly]

@nromriell Thank you for reporting this! I completely agree with everything you said above, and I really like the idea to use tags to track resources :) - with this said we will bring this during a community meeting today and discuss next steps.

[nromriell]

Thanks @a-hilaly for the quick triage and getting the discussion started during the community meeting today!

I've updated the title and added a note on the description that this would impact all controllers to reflect what I've found on further investigation. I'd love to see this get a resolution despite the initial cause being related to the crashloop. Let me know if you'd like me to take getting a proposal together for this