SANS CloudSecNext Summit 2024 - Security Configuration Management in the Cloud: Policy as Code for CI/CD Gating

... Welcome to our hands-on workshop, we are excited to have you join us for an engaging and informative session where you'll learn how to integrate Open Policy Agent (OPA) with GitHub Actions to enforce policy-as-code using Terraform as an example.

As organizations increasingly adopt Infrastructure as Code (IaC) practices, ensuring that infrastructure configurations are compliant with security and governance policies becomes crucial. Open Policy Agent (OPA) is a powerful, open-source policy engine that allows you to define and enforce policies across your infrastructure.

During this workshop we will complete the Setup, Working with code locally, Review the Pipeline, Review an OPA Policy and Exercise 1 as a group. The remainder will be done independently.

Table of Contents

1. Prerequisites
2. Reference Materials
3. Setup
   1. Development with GitHub Actions
   2. Development with Rego Playground
   3. Local Development - Optional
4. Exercises
   1. Review the pipeline
   2. Review an OPA Policy
   3. Exercise 1 - Github Actions and a Failing Pipeline
   4. Exercise 2 - Enforce Encryption of RDS
   5. Exercise 3 - Enforce Tagging Resource
   6. Exercise 4 - Encryption Exception Case for Databases with Public Data
   7. Bonus Exercise
5. Next Steps
6. Questions you may have
7. Appendix - Local Development - Optional
   1. Working with code locally
   2. Creating terraform output files
   3. Options for formatting the json output
   4. Evaluating our terraform plan output

Prerequisites

Required

• Github.com Account

Optional

• Installed locally: git
• IDE / Code Editor
• Installed locally: Docker/Docker Compose OR Podman/Podman Compose
• Installed locally: Conftest
• Installed locally: Terraform

Reference Materials

• OPA Overview
• OPA Language Reference
• OPA Cheat Sheet

Setup

Fork the repository and first Github Action

1. Fork the repo into your own account, allowing you to use the github runners. This can be done using the fork button at the top of the repository page.
   
2. Click "Create a new fork"
   
3. On the "Create a new fork" page, at the bottom click "Create fork"
   
4. Open up the browser to your forked version of the code (https://github.com/<YOUR_GITHUB_ID>/SANS-Workshop-2024).
5. Select the "Actions" Tab
   
6. Click "Enable Action on this repository"
   
7. Click back to the "Code" tab.
   
8. Create a branch by clicking the branch button that will say "main", then enter the branch name "workshop", and click "Create branch workshop from main"
   
9. Where the README is being display click the pencil button to edit the read me.
   
10. Add a minor change to the file (like a period at the end on the title). Then click the "Commit changes..." button.
    
11. On the "Commit Changes" dialog, click the "Commit changes" button (you can also update the commit message).
    
12. Click "Pull Requests" tab.
    
13. There should be a message at the top indicating your new branch has changed, click the "Compare & pull request" button".
    
14. On the Open a pull request page, select the destination to be your version of the repository and the main branch, and your version of the repository and your new branch.
    
15. Click the "Create pull request" button.
    
16. Give the actions about 30 seconds to start and you should see them begin to run.
    
17. Clicking one of the "Detail links will take you to the action (alternatively you can click the Actions tab at the top and select the running action).
    

🎆 Congrats, you have successfully run your first pipeline action with Policy as Code gating. 🎆

Development with GitHub Actions

In our simulated scenario, GitHub actions is our CI tool and also provides the gating for our OPA rules. It runs a containerized environment that executes the terraform plans and OPA validations. Using this we can both gate our work flow as well as test our changes without the need for specific tooling on our local machine.

We walked through initiating an action in the Setup section. We will further want through it in Exercise 1 - Github Actions and a Failing Pipeline

Working with code in github.dev

With the project open in GitHub, pressing the . key on your keyboard will open a web based IDE.

From the Explorer Menu () on the left side, you can view the files in the project.

From the Source Control Menu () on the left side, you can commit and push changes to the project.

From the Source Control Dialog you can add a comment for your commit and push the changes to your project.

Development with Rego Playground

We can use the Rego Playground to do virtual development.

• The terraform plan outputs go into the input section
• The policy we are working on goes into the coding section.
• The rego output is displayed in the outputs.
• Any print statements are displayed in the browser's developer console.

Local Development - Optional

You can also do development on your local machine. Setting that up is covered in the Appendix - Local Development - Optional section. We will walk through it after we have gone through Exercise 1 - Github Actions and a Failing Pipeline, for those who are interested.

Exercises

Review the pipeline

GitHub Actions defines a workflow pipeline in .github/workflows/ci.yml. Lets take a look at the steps it performs to do validation of the planned Terraform.

name: OPA Terraform Validation
on:  pull_request # only run on PR

jobs:
  get-working-directories: # This section reads in the terraform directories and creates a pipeline for each one
    ...
      
  run-opa-tests: # This is the primary pipeline that runs for each terraform folder
    name: 'Run OPA Tests - ${{ matrix.terraform_dirs }}'
    ...
    steps:
      - uses: actions/checkout@v4 # Checks out the code
      - name: Install Conftest # Installs the policy as code tool -> conftest
        ...
      - name: Install terraform-local # Installs the terraform local tool - used for our simulated AWS environment 
        ...
      - name: Start LocalStack # Starts up our simulated AWS environment
        ...
      - name: Terraform Init # Initializes terraform
        ...
      - name: Terraform Plan # Performs terraform plan and creates the output - this output is what is evaluated by conftest (policy as code)
        ...
      - name: Print Terraform Plan # Formats the terraform plan so it is readable
        ...
      - name: Store Terraform Plan Output # Store the terraform output in case we want to dowload it
       ...
      - name: Validate OPA # Does the OPA (policy as code) validation of the terraform outputs with rego
        ...
      - name: Print OPA Std Out # Print the OPA/Rego output for debugging
        ...
      - name: Print OPA Trace # Print the OPA/Rego trace statememnts for debugging
        ...
      - name: Store OPA Trace # Store the OPA/rego trace output in case we want to dowload it
        ...

When running the pipeline there are 3 outputs that are particularly useful for debugging:

• Print Terraform Plan - Displays the terraform output that OPA is evaluating
• Validate OPA - Displays the response of the OPA evaluation
• Print OPA Std Out - Displays the standard output from the evaluation. This will include any print statements that have been added to the rego code
• Print OPA Trace - The debug output from rego. It details out the evaluations it is doing to get to its result

Review an OPA Policy

There are already two OPA Policy created in:

• policies/pass.rego - this is a stripped down policy that always passes.
• policies/rds/password.rego - this policy ensures a plaintext password is not set for the database.

Exercise 1 - Github Actions and a Failing Pipeline

In this exercise we want to trigger Github Actions to test our OPA checks, and observe both a pass and fail output. For the later scenario, we will then to correct our code to get the pipeline to pass.

1. Add a line to the bottom of this README.md
2. Commit and Push the change.
3. Create a pull request for the branch
4. Open the repo in a browser and browse to your PR
5. Open the link to the failing action Run OPA Tests - rds
6. Under the failing Validate OPA step expand the Testing 'tfplan.json' against 2 policies in namespace 'aws.validation' section. What is the error?
7. Open the terraform for the RDS in terraform/rds/main.tf and fix the problem
8. Commit and push the code
9. The OPA checks should succeed now.

Hint

The fix is commented out in terraform/rds/main.tf.

Exercise 2 - Enforce Encryption of RDS

In this exercise we want to ensure our databases are encrypted.

1. Open the terraform/rds/main.tf files. You should see that storage encryption (storage_encrypted) is disabled.
2. If your local environment setup you can run the steps to generate the plan outputs with storage_encrypted set to true and false. Otherwise the plan files are in testfiles/aws/rds/encryption.
3. Evaluate the difference in the plan outputs.
4. Write a policy in the policies/rds folder.

Bonus - In terraform, if a parameter is not required it will have a default value assignment if it is not specified. In the case of storage_encrypted, the default value is false. Remove the storage_encrypted parameter and redo the plan (or look in testfiles/aws/rds/encryption/not-set). How does this compare to the other plan outputs? Will your policy need to be updated to cover this case?

Exercise 3 - Enforce Tagging Resources

In this exercise we want to ensure all our resources have proper tags.

1. Open the terraform/rds/main.tf and the terraform/s3/main.tf files. You should see the lines commented out to enable tagging.
2. If your local environment setup you can run the steps to generate the plan outputs with and without tagging enabled. Otherwise the plan files are in testfiles/aws/s3/tagging and testfiles/aws/rds/tagging
3. Evaluate the difference in the plan outputs.
4. Write a policy in the policies folder that requires a data_classification and owner_email tag.

Hint

Undefined or null references will cause the expression block to halt and exit immediately. If you have checks for null, break them up into multiple expressions, wrap in functions, or use a combination of is_object(resource) and contains_element(object.keys(resource), "KEY")

Bonus - Have you validated the tagging values? Making sure people are providing good data is important for resource tagging. Ensure that the values are constrained to:

• data_classification is either public or private.
• owner_email is a valid email address.

Hint

Rego supports regex expressions to do the email validation: https://docs.styra.com/opa/rego-by-example/builtins/regex

Bonus - Some AWS resources do not support tags, for example Security Hub. What will happen when your policy evaluates the terraform plan for Security Hub? You can uncomment out the terraform for Security Hub in terraform/securityhub and run an plan (or look in testfiles/aws/securityhub ). Will your policy need to be updated to cover this case?

Exercise 4 - Encryption Exception Case for Databases with Public Data

In the exercise 2 we ensured our RDS instances were encrypted, but that may not always be required, for example if we are storing public data. We will put in an exception to the encryption requirement if someone has tagged their RDS instance as public.

1. Open the terraform/rds/main.tf files.
2. Locate the data_classification tag.
3. If your local environment setup you can run the steps to generate the plan outputs with data_classification set to public and private. Otherwise the plan files are in testfiles/aws/rds/exception_for_encryption.
4. Evaluate the difference in the plan outputs.
5. Update the policy in the policies/rds folder that you wrote in Exercise 3

Hint

You can chain multiple checks together on multiple lines (in a rule body), and they evaluate with and.

Hint 2

There is no intrinsic or in rego, but if you have 2 boolean expressions that are assigned to the same value they will use an or for the assignment. How to express OR in Rego

Hint 3

If you need to nest an and within an or, try putting use a function to wrap the and statement Functions

Hint 4

If you want a function to have a default value if the evaluation fails, you can add a default value, like funcName(args) if { LOGIC} else := false

Bonus Exercise

By default new buckets don't allow public access. We can use the terraform resource s3_bucket_public_access_block to allow a bucket to be public (also plan is in testfiles/aws/s3/allow-public).

Can you add a restriction that enforces that a bucket that is made public and is also tagged as public? Update the terraform, run some plans and see.

Next Steps

Repository Setup:

• This is just a sample project. Typically the OPA Policies and the terraform would live in separate repositories so the policies can be shared across multiple terraform repos.
• In this workshop we have built policies and tested them against a failing and passing test case manually. This isn't scalable or automated. With a library of policies we can automate policy validation/testing. This is explained in the OPA documentation.

Things to think about:

• When to use Warn vs Deny vs Violation - You can prefix your policy names based on how you want conftest to handle failures. Think about the behavior you want when you are writing the policies. Read more here
• Better messaging for errors - Make your error messages as descriptive as possible to help developers understand why the policy is failing.

Questions you may have

• Why do we do we evaluate the plan vs the terraform directly? Although our examples are very simple, terraform can get complex with levels of indirection through the use of multiple files and modules. The plan is an output of all that combined and gives us a single file for evaluation.
• Why are my trace statements are not printing. Make sure you are not specifying an output, like --output json, as this will suppress printed messages or trace statements.
• My tests are passing when they should fail, why? OPA evaluation will silently fail if it gets a null reference. Try using a print statement (print(<VARIABLE>) of what you are evaluating to see what is being checked.

Appendix - Local Development - Optional

If you want to debug on your local machine, instead of just leveraging github actions, there are two options:

• (Recommended) Use the simulated containerized environment
• Setup the each of the tools on your machine and connect to an AWS account

Working with code locally

There are many ways to pull the source code to your local machine. We will detail using the command line, but you can choose your preferred method.

Command Line - Download the code to your local machine

Clone the repo to your local machine using your preferred method. From a command prompt run: git clone https://github.com/<YOUR_GITHUB_ID>/SANS-Workshop-2024.git

Upload the code back to the repository

When you are ready to commit code back to the repository:

• cd into the directory you want the code checked out.
• Run: git add . stage the code for committing
• Run: git commit -m'updates to workshop' to commit the code the change
• Run git push to push the code up to the remote repository

Here is what the output will look like:

SANS-Workshop-2024$ git add .
SANS-Workshop-2024$ git commit -m'updates to workshop'
[main 41a7ade] updates to workshop
 1 file changed, 65 insertions(+), 20 deletions(-)
SANS-Workshop-2024$ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 1.69 KiB | 1.69 MiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To github.com:SANS-Workshop/SANS-Workshop-2024.git
   1109cbc..41a7ade  main -> main

If you committed to a branch with an active PR, you can navigate back to your repository in github.com and see the action executing.

Now you can open the code you just checked out in your preferred IDE.

Creating terraform output files

Note: If you are running locally against a real AWS account replace tflocal with terraform

1. Enter the terraform directory containing the terraform with which you are working. For example run: cd /workspace/terraform/rds or cd /workspace/terraform/s3
2. Run: tflocal init to initialize terraform. If you are familiar with terraform you will notice we are using tflocal instead of terraform. Which is intended to leverage localstack to simulate our AWS account.
3. Run: tflocal plan --out tfplan.binary to create a plan for the terraform file and output it as a binary.
4. Run: tflocal show -json tfplan.binary > tfplan.json to generate the json terraform output the we will evaluate with OPA.

Options for formatting the json output

The output of the tflocal/terraform show will not have whitespace formatting, making it hard to read, consider using one of the following options to format the code:

• Use IDEs like VSCode, for example to formant a document right click and select Format and the document will be formatted
• With jq installed, run: jq . tfplan.json > tfplan-pretty.json
• Paste the JSON into the following website and have it formatted for you (Note this is not recommend if the terraform output could have sensitive data in it): https://jsonformatter.org/

Evaluating our terraform plan output

We use conftest to evaluate the terraform output against our policies.

From the folder where the terraform output was created, run:

• For JSON output: conftest test --all-namespaces -p /workspace/policies tfplan.json --output json

The output will state whether the policies written were successful or failed.

Example:

[
        {
                "filename": "tfplan.json",
                "namespace": "aws.validation",
                "successes": 1,
                "failures": [
                        {
                                "msg": "RDS should not specify passwords",
                                "metadata": {
                                        "details": {
                                                "rds_with_password": [
                                                        "my_db"
                                                ]
                                        }
                                }
                        }
                ]
        }
]