Source code and documentation for the BSides Orlando 2022 (11/18 - 11/19) talk: DevSecOps on a $ store budget
Security tooling can be expensive, very expensive. Never fear though, there are a multitude of cheap and open source options out there. You too can build a robust DevSecOps pipeline on a dollar store budget.
Covering everything from open source SAST tools to free secret scanning and Infrastructure-as-Code audits, this talk walks you through options to build out CI/CD pipelines that help to secure your code base without taking out a second mortgage.
Some basic knowledge of DevOps and CI/CD would be useful but is not mandatory.
This talk will walk the audience through how to build a variety of simple CI/CD pipelines with Jenkins and GitHub Actions. The CI/CD pipeline will demonstrate how to incorporate a number of security and code analysis tools including:
-
Cloc - What’s in the repository?
-
Checkov - Bridgecrews free IaC scanner
-
PHPMetrics - considering Cyclomatic Complexity from a security perspective
-
Tartufo - GoDaddy’s free Secrets Scanning tool
-
Git-secrets - Prevent your AWS secrets from hitting the repository
-
CodeQL and Dependabot - free scanning of open source repos in GitHub
The talk will then wrap up by looking at Horusec an open source SAST platform that incorporates a variety of security and linting tools in a container based environment.
An example PHP application using Terraform Infrastructure-as-Code targeting an AWS environment will be used to demo the DevSecOps process to the audience. This code can be downloaded during the talk to allow participants to follow along in a hands on fashion.
-
Introduction
-
What is DevSecOps
-
Commercial tools aren't the only option
-
Jenkins overview
-
A walk through our example repository
-
Scanning a repository with cloc
-
Reviewing Terraform with Checkov
-
Auditing PHP with PHPMetrics
-
Scan for secrets using Tortufo
-
Pre-commit hooks with git-secrets
-
Open source Shift Left with GitHub
-
Wrapping up with Horusec
-
Closing statement
-
QA
The following guide provides instructions for setting up the Jenkins CI/CD pipeline for this DevSecOps project. Depending on the environment you chose to use for running Jenkins, you will need to install/setup a number of prequisite services prior to building your pipeline. For example when setting up the cloud infrastructure you will need to ensure you have Terraform installed locally, and if you wish to map a domain to Jenkins, ensure this is registered in Route 53.
Each section will provide a guide on the pre-requisites and the steps to get up and running.
To kick the the tires on the application and the pipeline you can install Jenkins locally.
Jenkins install instructions can be found at: https://www.jenkins.io/doc/book/installing/
The guide below covers local setup on personal devices such as Windows and Mac laptops.
If you want to follow along locally you will need:
-
Jenkins setup locally
-
A forked copy of the course code (this repository)
-
A bridgecrew account - you can set this up with your GitHub user: https://www.bridgecrew.cloud/
In addition to the above, there will be some local OS specific requirements, which are covered under the relevant section below.
Make sure you have brew installed on your Mac and that you have updated it recently.
With brew in place we can then beging by adding Jenkins.
Install the LTS version:
brew install jenkins-lts
Once installed, start it up e.g.
brew services start jenkins-lts
Navigate to:
http://localhost:8080
You will now be prompted to enter the default password. This will be stored in a location shared with you on the webpage and will be in a format similar to:
/Users/<user>/.jenkins/secrets/initialAdminPassword
Using this password you should now be logged in, and can configure Jenkins, change the password and get started.
The default plugin installation option presented initially should be sufficent for this talk, bar one missing plugin you will need. The default plugins installation wizard will also allow you to setup a new admin user.
Once this is complete, you will need to add the afore mentioned missing plugin, which is the Docker pipeline one.
Navigate to your Jenkins management view:
Dashboard > Manage Jenkins
From here select Plugin Manager.
From the list of Available plugins select Docker Pipeline and install this.
Dashboard > Manage Jenkins > Manage Plugins > Available (tab) > docker-workflow.
You should now be all set to start building CI/CD pipelines in Jenkins.
If you hit issues with Docker not being found when you run the pipeline, edit:
/usr/local/opt/jenkins-lts/homebrew.mxcl.jenkins-lts.plist
Add:
<key>EnvironmentVariables</key>
<dict>
<key>PATH</key>
<string>/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
</dict>
Then restart:
brew services restart jenkins-lts
Instructions here are based upon the Jenkins documentation at: https://www.jenkins.io/doc/book/installing/windows/
-
Start by grabbing the MSI installer from the downlaods page: https://www.jenkins.io/download/#downloading-jenkins
-
Once downloaded, run the installer and select a suitable location for it to install Jenkins into e.g.
c:\ program Files\Jenkins -
Run the Jenkins service using a
localordomainuser. Specify the user name and password for running Jenkins then click onTest Credentialsto test them -
Select the port as
8080- make sure you have nothing else runnong on this port locally -
Select the Java home directory. This is the location of
JDKorJRE. If you don't have it installed already, follow the installation prompts. -
Configure any other services tha need to be installed, such as Firewall Exceptions to run on port
8080 -
Click
Installto kick off the process -
Once installed you will need to obtain the default password from
<path\to\Jenkins>\secrets\iniitalAdminPassword -
Enter this into the
Administrator passwordtext field -
Select
Install suggested plugins
You are now ready to setup your new admin user and get started.
Jenkins can be installed on Linux and the current distros are supported:
-
Debian/Ubuntu - https://www.jenkins.io/doc/book/installing/linux/#debianubuntu
-
Fedora - https://www.jenkins.io/doc/book/installing/linux/#fedora
-
Red Hat/CentOS - https://www.jenkins.io/doc/book/installing/linux/#red-hat-centos
For the cloud infrastructure option later in this README, Amazon Linux is used.
The official Jenkins installation instructions for supported versions of Linux are available at:
https://www.jenkins.io/doc/book/installing/linux/
For users who wish to build clloud based infrastructure that can be shared among team members, these instructions guide you through deploying the Terraform code for setting up the environment, using Packer to create a pre-baked AMI and finally integrating your Jenkins cloud hosted environment with GitHub.
Currently AWS is the only cloud vendor supported, but the code could easy by expanded to support Microsoft Azure and GCP.
The AWS installation instructions walk you through how to setup an environment in AWS to hosted Jenkins. This includes:
-
Building a Jenkins AMI with baked in Docker support, which can be deployed on an EC2 instance
-
Terraform code that builds out:
a. The VPC and S3 buckets for storing state
b. Subnets (Private and Public)
c. Bastion host and security group
d. Jenkins servers and security group
e. ELB and security group
-
Manually adding in your domain name and certificate (and instructions on disabling this feature if you don't want to use it)
At the completion of the infrastructure setup you will have a fully built Jenkins CI/CD environment that can be easily added to or destroyed via Terraform.
To follow along in the cloud, you will need:
-
An AWS account
-
A forked copy of the source code
-
A bridgecrew account - you can set this up with your GitHub user
-
A domain name/subdomain configured as a Hosted zone in your AWS account. A number of services offer free domain registration or domains for as little as $0.01. Note: if you do not want to setup a domain and SSL cert, and map these to the load balancer, instructions will be provided on disabling this portion of the Terraform code.
Follow the steps here to manually create a subdomain:
https://aws.amazon.com/premiumsupport/knowledge-center/create-subdomain-route-53/
Then add the SSL cert like this:
-
Request a certificate
-
Add the reference to DNS
-
Copy the Cert ARN. You wll need to add this to your .tfsecrets file in the next stage
-
Cert can now be hooked up to the ELB via Terraform
A separate README is provided with a detailed description of the Terraform code and what it is does.
From a high-level the Terraform files are responsible for building:
-
The S3 bucket for storing state in AWS
terraform > tf-state -
The Jenkins server and associated infrastructure
terraform > network-infra -
The Jenkins AMI
packer -
The PHP server for running our vulnerable webapp.
terraform > app-server
In order to run the Terraform scripts make sure you have Terraform installed on the device you will be executing them from.
Instructions for Mac, Windows and Linux are located here: https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli
You will also need Packer installed to create the AMI. Instructions for this are located at: https://developer.hashicorp.com/packer/tutorials/docker-get-started/get-started-install-cli
Once installation is complete you are ready to begin.
The Terraform scripts will reference the AMI that you have Jenkins installed on. Included in this repository is a simple Packer configuration that will build the AMI and store it in your AWS account.
In order for this to work you will need to have a VPC already in place. This creates something of a chicken and egg problem, as in order to create the VPC for Jenkins, you need the AMI reference.
The easiest way to do this is to create a VPC manually through the web console, and then point Packer to it. For further information on this, please refer to: $
AWS Marketplace > Discover products
Use the Search AWS Marketplace products field to search for Jenkins.
There are multiple options, many of which would be very cheap or practically free for testing these instructions out.
Alternatively, you could create an AMI manually by installing Jenkins on an EC2 instance and taking an AMI snapshot.
For further instructions on this, refer to: https://www.jenkins.io/doc/tutorials/tutorial-for-installing-jenkins-on-AWS/
Search results
Follow the instructions at: https://github.com/moduslabs/dollarstore-devsecops-bsides-orlando-2022/blob/master/terraform/README.md then return back here.
(Optional) Next add the load balancer to the domain:
https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-elb-load-balancer.html
Once you have Jenkins up and running either locally or in the Cloud, you can begin to experiment with adding pipelines and injecting secrtes into your pipelines, so you can cmmunicate with third party services.
As a pre-requisite to this section, if you wish to integrate the Checkov IaC scanner into your CI/CD pipeline, please create an account as you will need to generate an API key later in the instructions.
You can read more on account creation at: https://www.checkov.io/
If you have further questions on Jenkins, you can always refer to: https://www.jenkins.io/doc/
The first thing we intend to do is setup a pipeline for running our CI/CD processes. This pipeline will point to our code repository and yse the Jenkinsfiles located in it, based upon the environment you are working in.
Get started by following these steps:
-
Make sure you used a forked version of this repository and cloned it (you'll need it again shortly)
-
Login to Jenkins
-
Create a new pipeline and call your pipeline
bsidesorlando2022 -
Description- add a description of your choosing e.g.BSides Orlando Talk -
Under pipeline select
Pipeline script from SCM -
Select
Gitfrom SCM and add the repository location. Note, if you have a private fork you will need to enter login credentials -
Add the path to the Jenkinsfile in your repository you plan to use e.g.
Jenkinsfile.aws -
Save your pipeline
-
Next select
Build Now -
Expect to see a few errors, as we need to generate an API key for Checkov.
We are going to start with adding in an API key. This can be generated from the Bridgecrew website and is used by Checkov to communicate with the bridgecrew API to generate findings information. For more information/instructions on generating the API Key in yor account, refer to: https://docs.bridgecrew.io/docs/get-api-token
Once you have your API key we are now going to add this to Jenkins.
Start by selecting the pipeline you just created and select the edit option.
Update as follows:
-
Choose
This project is parameterized -
Next select:
Credentials parameter -
We are now going to use the create a Jenkins credential feature:
Jenkins Credentials Provider: Jenkins
From the options add:
i. Domain, leave default
ii. Kind - secret text
iii. Scope: Global
iv. Secret - paste your Checkov API key
v. ID: checkov-api-key
vi. Description - say what it is e.g. Bsides Orlando API Key
- Save these changes. When the pop up closes, set the
Default Valueto the BSides Orlando API Key you just created.
With the refer to the API key now available in the pipeline, the final step is going to be tell our Jenkinsfile where it can find the API key when it executes a stage of the build pipeline.
Edit the Jenkinsfile for the environment you are working in e.g. Jenkinsfile.aws and update the Checkov call to include your API key environment var name. Save the file and commit and push back to the repository.
Once your file is present back in GitHub we can now go back to Jenkins.
Select your pipeline and then execute the option:
Run build with Parameters.
You should now see you can select your API key to be included when the pipeline is run.
Our CI/CD process is now executing.
View the Console log for your pipeline, should see some errors. We need to fix these!
Our CI/CD pipeline located in the Jenkinsfile is made up of multiple stages. Each stage is responsible for executing a specific task, such as running a security tool.
Cloc can be used to ascertain what files existing in a rpeository. This can be a great and quick way of looking for things that shouldn't be in the repostiroy, and could contain security breaches. Examples include:
-
Word docs with passwords
-
Spreadsheets with passwords
-
Binary files with hard coded secrets
When this stage executes, you can see in the Jenkins console the findings.
Will see errors in Terraform files.
These are:
.
They are fixed by doing the following:
https://phpmetrics.org/index.html
If you installed PHPMetrics locally, we also installed composer alongside PHPMetrics, if you are using the AWS environment option, we have a container in place that executes PHPMetrics.
The key difference between the local and cloud installation is the Installation step, which isn't required in Jenkinsfile.aws since the container is leveraged.
Execute via Pipeline against target code base.
Tartufo is a project run by GoDaddy that acts as a wrapper for TurrfleHog, and allows configuration of secrets scanning via a .toml file.
You can find background on the tool at: https://tartufo.readthedocs.io/en/stable/
For this project, configuration including exlusions for false positives are added to the following file:
tartufo.toml
The git-secdrets library built by AWS labs, allows for the detection of AWS API keys. It can be run in a pipeline or locally as pre-commit hook.
Further documentation on the tool plus the source code can be foudn at: https://github.com/awslabs/git-secrets
A containerized version of thistool running on Alpine is available at: https://registry.hub.docker.com/r/moduscreate/alpine-git-secrets
We are going to run the basic version without the supplementary container tools from inside Jenkins.
The instructions for this are located at: https://docs.horusec.io/docs/pt-br/tutorials/how-to-use-horusec-without-docker/
The supported scans are: https://docs.horusec.io/docs/pt-br/cli/analysis-tools/open-source-horusec-engine/
Based off of a simple tutorial here: https://code.tutsplus.com/tutorials/how-to-build-a-simple-rest-api-in-php--cms-37000
To add to this project please fork the repository and create a pull request.
Modus Create is a digital product consultancy. We use a distributed team of the best talent in the world to offer a full suite of digital product design-build services; ranging from consumer facing apps, to digital migration, to agile development training, and business transformation.
This project is part of Modus Labs.
This project is MIT licensed.