Skip to content

How we set up continuous integration on Jenkins

Jump to bottom
Adam Hooper edited this page Aug 7, 2017 · 23 revisions
  1. At https://cloud-images.ubuntu.com/locator/ec2/, pick the latest hvm:ebs-ssd instance for the region (us-east-1 in our case)
    1. Choose the dedicated CI VPC, with IAM role Jenkins-CI.
    2. Tag it Environment: ci
    3. Give it the jenkins-ci security group (inbound port 443)
    4. Launch it!
  2. Name it Jenkins-CI
  3. Give yourself temporary SSH access (via the security group) and SSH in.
  4. Install Jenkins:
    1. sudo apt-get update && sudo apt-get dist-upgrade
    2. Follow the instructions at https://pkg.jenkins.io/debian-stable/ to install: 
      wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -
sudo apt-add-repository 'deb https://pkg.jenkins.io/debian-stable binary/'
sudo apt-get update
sudo apt-get install jenkins
  5. Set up GitHub for authentication: at https://jenkins-ci.overviewdocs.com/securityRealm/finishLogin set up overview-jenkins-ci with a URL of https://jenkins-ci.overviewdocs.com.
  6. Setup Jenkins:
    1. From your computer, ssh -L 8080:localhost:8080 ubuntu@[JenkinsIP] and browse to http://localhost:8080
    2. Copy/paste the administrator password in (as prompted by Jenkins)
    3. "Select plugins to install" => choose defaults, ...
      • plus: embeddable-build-status, JUnit Plugin, GitHub Plugin, GitHub Authentication Plugin, Amazon EC2 Plugin
      • minus: Ant Plugin, Gradle Plugin
    4. Skip creating the administrator user. Click "Start Using Jenkins"
    5. "Manage Jenkins" => "Configure System":
      • # of executors: 0
      • Jenkins URL: https://jenkins-ci.overviewdocs.com
      • System Admin e-mail address: [email protected]
      • SMTP Server: email-smtp.us-east-1.amazonaws.com
      • Check User SMTP Authentication and enter Amazon's SMTP settings
    6. "Manage Jenkins" => "Configure Global Security"
      • Security realm => Github Authentication Plugin
      • Enter Client ID and Client Secret from the overview-jenkins-ci app page on GitHub
      • Authorization => GitHub Committer Authorization Strategy
      • Enter comma-separated Admin User Names
      • Check Use GitHub repository permissions, and don't fill in Participant in Organization
      • Check Grant READ permissions for Anonymous Users
      • Check Grant ViewStatus permissions for Anonymous Users
      • Save. You'll be locked out.
  7. Adjust DNS to point to your new server.
  8. Set up the HTTPS proxy:
    • Install programs 
      sudo apt-get install haproxy
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install certbot
sudo certbot certonly --standalone -d jenkins-ci.overviewdocs.com -m [email protected] --agree-tos -n
    • Fill in /etc/haproxy/haproxy.cfg: 
      global
  log /dev/log  local0
  log /dev/log  local1 notice
  chroot /var/lib/haproxy
  user haproxy
  group haproxy
  daemon
  # https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy-1.5.14&openssl=1.0.1e&hsts=yes&profile=modern
  ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
  ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
  ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
  ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

defaults
  log global
  mode http
  option httplog
  option dontlognull
  option forwardfor
  option http-server-close
  timeout connect 5000
  timeout client 50000
  timeout server 50000
  errorfile 502 /etc/haproxy/errors/502.http

frontend jenkins-ci.overviewdocs.com
  bind :80
  bind :443 ssl crt /etc/haproxy/ssl.pem
  redirect scheme https if !{ ssl_fc }
  reqadd X-Forwarded-Proto:\ https
  use_backend jenkins

backend jenkins
  server jenkins01 127.0.0.1:8080
    • Fill in /etc/letsencrypt/post-renew.sh: 
      #!/bin/sh

cat \
    /etc/letsencrypt/live/jenkins-ci.overviewdocs.com/privkey.pem \
    /etc/letsencrypt/live/jenkins-ci.overviewdocs.com/fullchain.pem \
    > /etc/haproxy/ssl.pem
chown haproxy:haproxy /etc/haproxy/ssl.pem
chmod 0600 /etc/haproxy/ssl.pem
systemctl restart haproxy
    • Fill in /etc/letsencrypt/pre-renew.sh: 
      #!/bin/sh
systemctl stop haproxy
    • chmod +x /etc/letsencrypt/post-renew.sh /etc/letsencrypt/pre-renew.sh
    • /etc/letsencrypt/post-renew.sh (will actually start haproxy)
    • Fill in /etc/cron.daily/letsencrypt: 
      #!/bin/sh

exec /usr/bin/certbot renew \
    --pre-hook /etc/letsencrypt/pre-renew.sh \
    --post-hook /etc/letsencrypt/post-renew.sh \
    --quiet
    • chmod +x /etc/cron.daily/letsencrypt
  9. Set up EC2 spot-instance nodes
    1. Browse to Jenkins, "Manage Jenkins" -> "Configure System"
    2. "Add New Cloud" => "Amazon EC2"
    3. Check Use EC2 instance profile to obtain credentials
    4. Choose region us-east-1
    5. Enter a private key (I forget how I generated this....)
    6. Advanced -> "Add" an AMI....
    7. Description: Jenkins-CI
    8. AMI ID: Build using Jenkins-CI Test Slave AMI instructions and set the ID here
    9. Instance Type: something big, Availability Zone: whatever's best. At time of writing, C48xlarge and us-east-1d have a good price. (Use the AWS console to find a combination that seems reliable.)
    10. Spot Max Bid Price: whatever -- 0.5 means 50 cents per hour. (Expect 3 builds per hour. Jenkins won't build every commit, if you commit lots of commits at the same time.)
    11. Choose Bid Type: persistent
    12. Security group names: test-slave (which grants SSH access to Jenkins-CI security group)
    13. Remote FS root: /home/ubuntu
    14. Remove user: ubuntu
    15. Labels: test-slave
    16. Idle termination time: 30
    17. Advanced => Number of Executors: 1
    18. Check Stop/Disconnect on Idle Timeout
    19. Tags: role:jenkins-test-slave, Name:jenkins-test-slave, Environment:ci
    20. IAM Profile: The description of your IAM "instance-profile" Role. This starts with arn: and ends with instance-profile/jenkins-test-slave -- a role with permission to write to s3://overview-builds.
  10. Add overview-server project
    1. Browse to Jenkins and click Create new jobs
    2. Name overview-server, Pipeline, OK
    3. Check GitHub project and enter URL https://github.com/overview/overview-server
    4. Check GitHub hook trigger for GITScm polling
    5. Check Quiet period and enter 120 seconds (builds cost time and money; this can save us a few)
    6. Make the Pipeline script Pipeline script from SCM. Select Git as a source, and enter https://github.com/overview/overview-server
    7. Make */master and */feature-* are the branch specifiers
  11. Hook up to GitHub
    1. Add [![Build Status](http://jenkins-ci.overviewdocs.com/job/overview-server/badge/icon)](http://jenkins-ci.overviewdocs.com/job/overview-server/) to README.md
    2. In overview-server project settings on GitHub, add "Integrations and Services" -> "Jenkins (GitHub plugin)" -> https://jenkins-ci.overviewdocs.com/github-webhook/
  12. Hook up to Slack
    1. On Jenkins, add "Slack" plugin in Manage Plugins
    2. On Slack, add "Jenkins" plugin
    3. Copy/paste the Team Domain and Integration Token from Slack into Jenkins
    4. (That's it -- the Jenkinsfile will handle the rest.)
Clone this wiki locally