TF-Controller BLOB Caching Design Document

[chanwit]

Workspace BLOB Caching

This document outlines the design and architecture of the BLOB caching mechanism in TF-Controller. The primary goal of this mechanism is to efficiently address the resource deletion problem using the contents of generated Workspace BLOBs, ensuring a streamlined process for Terraform finalization.

Definitions

• Source BLOB: A BLOB downloaded from an artifact URL advertised by the Source Controller.
• Workspace BLOB: A tar.gz file created by the TF-Controller, which contains the Workspace file system. This file system includes files from the Source BLOB, a variable file generated by the runner (containing information from Secrets and ConfigMaps specified in the Terraform CR), and files from File Mapping specified in the Terraform CR.

Workflow

Italics are already implemented steps. Bolds are steps proposed to implement.

• Downloading Source BLOB: The TF-Controller fetches the Source BLOB from the Source Controller.
• Pushing to Runner: The TF-Controller pushes the Source BLOB to a runner.
• Generating Workspace File System: The runner processes the Source BLOB to generate a Workspace file system. This system contains:
  • Files from the Source BLOB.
  • A variable file generated by the runner.
  • Files from File Mapping, as specified in the Terraform CR.
• Creating Workspace BLOB: The TF-Controller invokes a new gRPC function named "CreateWorkspaceBlob".
  The content of the Workspace file system is compressed using tar and gzip to produce the Workspace BLOB.
  The TF-Controller retrieves this BLOB as a byte array via the return value of the CreateWorkspaceBlob gRPC function.
• Caching Mechanism: The creation of the WorkspaceBlob is placed right before the Terraform Initialization step. This ensures that the relevant data is used. (Note: the break-the-glass logic is also implemented at this juncture). The TF-Controller caches each Workspace Blob on its local disk, naming it by the convention "$namespace-$name.tar.gz".
• Persistence Mechanism: The persistence mechanism employed by the Source Controller will be used for the TF-Controller's persistence volume.

BLOB encryption
Encryption/decryption tasks to the runner, while the controller only manages the storage of encrypted BLOBs.

For each namespace in this setup, a service account (usually named tf-runner) is essential. The most appropriate encryption key in this case would be the token of this service account. This is natively supported by Kubernetes, as detailed in their official documentation: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#manually-create-a-long-lived-api-token-for-a-serviceaccount

STRIDE Analysis

The STRIDE model is a security framework designed to identify threats in a system, focusing on six categories: Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, and Elevation of Privilege.

Here's an analysis for the Workload BLOB caching system using the model.

Spoofing:

Threats:

• Unauthorized access to the TF-Controller, leading to fetching of Workload BLOBs.
• Impersonation of the gRPC function "CreateWorkspaceBlob".

Mitigation:

• Use Kubernetes RBAC to restrict access to pods and controllers.
• Implement mutual authentication for the gRPC communication.

Tampering:

Threats:

• Tampering with a Workspace BLOB before it's stored on the local disk.
• Altering the contents of a Workspace BLOB in the local disk cache.

Mitigation:

• Implement integrity checks using checksums to verify the contents of the Workspace BLOB.
• Ensure that local disk storage is write-protected from other users or services using 0600 permissions.

Repudiation:

Threats:

• A user or system denies performing an action, like creating a BLOB or initiating a Terraform operation.

Mitigation:

• Implement strong logging and auditing mechanisms. This will help in tracking the activities and maintaining non-repudiation.

Information Disclosure:

Threats:

• Exposure of encrypted BLOBs leading to potential decryption attacks.
• Exposure of encryption keys, especially if they're namespace-based, which might be guessable or discoverable.
• Accidental leakage of service account tokens.

Mitigation:

• Use strong encryption algorithms for encrypting Workspace BLOBs.
• Periodically rotate the encryption keys.
• Avoid using predictable or easily discoverable values, like namespace's names, for encryption keys. Namespace UUIDs and ServiceAccount's Token Secrets are good candidates.
• Limit access to service account tokens using Kubernetes secrets and RBAC.

Denial of Service:

Threats:

• Filling up the local volume storage with BLOBs.

Mitigation:

• Monitor the local volume for available space and implement automatic cleanup procedures if nearing capacity.

Elevation of Privilege:

Threats:

• Using service account tokens to gain unauthorized access.

Mitigation:

• Secure the service accounts and their tokens, ensuring they have minimal necessary permissions.

Note: Each namespace in the Helm Chart will require an additional secret.

For the first MVP of this BLOB caching, the default pod local volume will suffice. However, it's crucial to note that a Controller restart will clear the BLOB cache. Given that a controller restart clears the BLOB cache, we would need to ensure data integrity and availability by backing up essential data elsewhere. We should consider implementing persistent volumes in a future version.

[squaremo]

If I understand accurately, this will end up putting secrets in plain text on the host disk. What mitigations are there for this extra "attack surface"?

[chanwit]

Yep. That's a concern.
Should we do any encryption?

If so, the encryption would be done by a runner for example.

[chanwit]

My proposed solution is to assign the encryption/decryption tasks to the runner, while the controller only manages the storage of encrypted BLOBs.

For each namespace in this setup, a service account (usually named tf-runner) is essential. The most appropriate encryption key in this case would be the token of this service account. This is natively supported by Kubernetes, as detailed in their official documentation: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#manually-create-a-long-lived-api-token-for-a-serviceaccount

Note: Each namespace in the Helm Chart will require an additional secret.

[chanwit]

The STRIDE model is a security framework designed to identify threats in a system, focusing on six categories: Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, and Elevation of Privilege.

Here's an analysis for the Workload BLOB caching system using the model.

Spoofing:

Threats:

• Unauthorized access to the TF-Controller, leading to fetching of Workload BLOBs.
• Impersonation of the gRPC function "CreateWorkspaceBlob".

Mitigation:

• Use Kubernetes RBAC to restrict access to pods and controllers.
• Implement mutual authentication for the gRPC communication.

Tampering:

Threats:

• Tampering with a Workspace BLOB before it's stored on the local disk.
• Altering the contents of a Workspace BLOB in the local disk cache.

Mitigation:

• Implement integrity checks using checksums to verify the contents of the Workspace BLOB.
• Ensure that local disk storage is write-protected from other users or services using 0600 permissions.

Repudiation:

Threats:

• A user or system denies performing an action, like creating a BLOB or initiating a Terraform operation.

Mitigation:

• Implement strong logging and auditing mechanisms. This will help in tracking the activities and maintaining non-repudiation.

Information Disclosure:

Threats:

• Exposure of encrypted BLOBs leading to potential decryption attacks.
• Exposure of encryption keys, especially if they're namespace-based, which might be guessable or discoverable.
• Accidental leakage of service account tokens.

Mitigation:

• Use strong encryption algorithms for encrypting Workspace BLOBs.
• Periodically rotate the encryption keys.
• Avoid using predictable or easily discoverable values, like namespace's names, for encryption keys. Namespace UUIDs and ServiceAccount's Token Secrets are good candidates.
• Limit access to service account tokens using Kubernetes secrets and RBAC.

Denial of Service:

Threats:

• Filling up the local volume storage with BLOBs.

Mitigation:

• Monitor the local volume for available space and implement automatic cleanup procedures if nearing capacity.

Elevation of Privilege:

Threats:

• Using service account tokens to gain unauthorized access.

Mitigation:

• Secure the service accounts and their tokens, ensuring they have minimal necessary permissions.

Another consideration: given that a controller restart clears the BLOB cache, we would need to ensure data integrity and availability by backing up essential data elsewhere. We should consider implementing persistent volumes in a future version.

[yitsushi]

Spoofing
[Addition] I don't know how viable this scenario is, but....

• Threat: After a resource was deleted, recreating the same resource to access the same file. I don't think it's possible to exploit tho.
• Mitigation: Delete cache file when we handle terraform resource deletions.

Repudiation

Implement strong logging and auditing mechanisms. This will help in tracking the activities and maintaining non-repudiation.

Potentially a managed transaction identifier for the whole process. From Reconciliation through the runner to saving BLOB content, that way a manual or automatic auditing tool can easily see where it started and what steps in made across components (runner log, tf controller log).

Information Disclosure

Limit access to service account tokens using Kubernetes secrets and RBAC.

While it's valid, I don't think it's our job to define and ensure that, only on Chart level. It's a general Kubernetes principle. It's like saying "store your password in a safe place or don't store at all".

Avoid using predictable or easily discoverable values, like namespace's names, for encryption keys. Namespace UUIDs and ServiceAccount's Token Secrets are good candidates.

I think that contradicts this:

Exposure of encryption keys, especially if they're namespace-based, which might be guessable or discoverable.

Information Disclosure + Spoofing

If mTLS keys leaked, they can make API calls to the runner.

Denial of Service

[Addition]

• Threat: bombing requests to the runner. Even if the request will be discarded because of missing auth, the service will still receive it, or is it handled already in other ways?
• Mitigation: Throttling.

[yitsushi]

Questions for the future:

Periodically rotate the encryption keys.

If the runner encrypts the BLOB, what happens if the encryption key was rotated? We'll discard the BLOB and handle it as it was not there (at the end it's only a cache)? Or we try to re-encrypt somehow?

we would need to ensure data integrity and availability by backing up essential data elsewhere

I would say Object Storage (S3, AZ Blob, Minio, etc) sounds a good option to store them. It can be secured with and access from a scalable environment.