Cache workspace BLOBs in TF Controller in case the source gets deleted before Terraform CR

[chanwit]

User Story

As a Terraform Controller user,
I'd like to have the workspace blobs cached in the TF Controller,
so that the deletion of Source won't cause issues in processing Terraform CRs.

Acceptance Criteria

• The workspace blobs are cached in the TF Controller before processing the Terraform CRs.
• Even when the source is deleted before the Terraform CR, the processing won't fail due to the cached blobs.
• Adequate tests are added to ensure the caching mechanism works properly, and there is no regression in other functionalities.

Sub tasks

• Implement BLOB encryption using service account tokens as keys within the tf-runner. #1121
• Implement the CreateWorkspaceBlob gRPC function to create, compress, encrypt, and retrieve Workspace BLOBs from the tf-runner. #1120
• Cache workspace blob into tf-controller file system #1157
• Implement a new deletion routine in the finalizer that uses the Workspace BLOB to create the destroy plan and destroy resources.
• Develop a mechanism to handle and rotate encryption keys using service account tokens and Secrets in Kubernetes.
• Ensure service account tokens have the least privileges to safeguard against Elevation of Privilege.
• Implement checksum validation for handling Tampering. #1158
• Ensure local storage has write-protect (0600) mode for handling Tampering.
• Integrate logging and auditing functionalities to address Repudiation.
• Develop mechanisms to monitor storage usage and perform automatic cleanup in the finalizer after the Terraform resouces are completely destroyed to prevent Denial of Service.
• Adopt the Source Controller's persistence mechanism for the TF-Controller's MVP.

[LappleApple]

Are we aware of users/customers currently having issues because we don't have this enhancement in place?

[chanwit]

It's the problem we categorized as the single resource deletion problem reported by #454 and #666

[chanwit]

Terraform BLOB caching design doc:
#911

[LappleApple]

Sorry, I transferred a question over from notes that I had made before reading those issues later on. 🤦 Thanks for the prompt response!