Release 07/24/2020

Headline News

• Big update on the Wiki which now includes MOADSD-NG-SERVER and the CONFIGURATOR.
• The CONFIGURATOR now supports GCP, AWS and the ESX variant.
• The kubernetes configuration within Jenkins is now persistent and survives a pause-resume cycle.
• OPA does include some test scenarios which can be deployed with deploy_opa_testcases.
• Refactored Smart Check Deployment which is now idempotent.
• You can now "deploy" a scheduled shutdown for instances to lower costs in case you forget to pause the environment after a busy session :-). It defaults to 240 minutes when deployed. To schedule the shutdown run the playbook pause_scheduled. Cancelling a scheduled shutdown is done by pause_schedule_cancel. To change the delay, overwrite the variable instance_shutdown_delay in your configuration.yml. The auto-pause is automatically activated after a resume.
• Kubernetes Dashboard upgraded to v2
• It is no longer required to set the variables aws_region, aws_zone, aws_access_key and aws_secret_key manually within the configuration, since they are now read from the aws configuration of the awscli. The aws_zone is set to the value of aws_region with an a appended.
• It is no longer required to set the variables gcp_project_id, gcp_region and gcp_zone manually within the configuration, since they are now read from the gcloud configuration.
• Bumped Jenkins to version 2.235.1
• Multiple instances of MOADSD-NG are now possible within one AWS region. Add the flag moadsd_ng_user: <YOUR_NAME> to your configuration.yml

The "one more thing" thing

• After the deployment of the software stack or a resume some ssh commands are now created within ./moadsd-ng to ease access to the jumphost and the Kubernetes master (only available for cloud environments).
  • ./ssh_jumphost - login to the jumphost as user ubuntu
  • ./ssh_master - login to the Kubernetes master as user ubuntu
  • ./ssh_master_pods - watch pods on the cluster
  • ./ssh_master_services - watch services on the cluster
  • ./ssh_master_deployments - watch deployments on the cluster
• Moved the task configuration within the menu to the target environment specific second level. The configuration is then automatically executed after the update of the configuration.
• Feature Testing: MOADSD-NG now reports the it's usage and high level configuration. This is done by a REST call whenever main playbooks of MOADSD-NG are called. For transparency, the following data is transmitted via https:

"datetime": {"S": "{{ lookup('pipe','date +%Y-%m-%d-%H-%M-%S') }}" },
"admin_email": {"S": "{{ admin_email | hash('sha256') }}" },
"type": {"S": "{{ type }}" },
"action": {"S": "{{ run_pb }}" },
"site_deploy_kubernetes": {"S": "{{ site_deploy_kubernetes }}" },
"site_deploy_openshift": {"S": "{{ site_deploy_openshift }}" },
"site_deploy_deepsecurity": {"S": "{{ site_deploy_deepsecurity }}" },
"site_deploy_smartcheck": {"S": "{{ site_deploy_smartcheck }}" },
"site_deploy_jenkins": {"S": "{{ site_deploy_jenkins }}" },
"site_deploy_gitlab": {"S": "{{ site_deploy_gitlab }}" },
"site_deploy_endpoints": {"S": "{{ site_deploy_endpoints }}" },
"site_deploy_linkerd": {"S": "{{ site_deploy_linkerd }}" },
"site_deploy_prometheus": {"S": "{{ site_deploy_prometheus }}" },
"site_deploy_grafana": {"S": "{{ site_deploy_grafana }}" },
"site_deploy_opa": {"S": "{{ site_deploy_opa }}" },
"site_deploy_jumphost": {"S": "{{ site_deploy_jumphost }}" }

Please note, that the admin_email is hashed. Resulting data collected:

"action": "deploy",
"admin_email": "ca8b7b30a5d9aa8782bd0f1926e388a631d075cbab0fac1138f3628ab7c8873c",
"datetime": "2020-06-09-16-52-59",
"site_deploy_deepsecurity": "False",
"site_deploy_endpoints": "False",
"site_deploy_gitlab": "False",
"site_deploy_grafana": "True",
"site_deploy_jenkins": "True",
"site_deploy_jumphost": "True",
"site_deploy_kubernetes": "True",
"site_deploy_linkerd": "False",
"site_deploy_opa": "True",
"site_deploy_openshift": "False",
"site_deploy_prometheus": "True",
"site_deploy_smartcheck": "True",
"type": "gcp"

• Two new commands on the server:
  • copy_master <filename> copies the file to the master
  • fetch_master <filename> fetches the file from the master

Current Default Configuration Overview

Component	Deployed
Kubernetes	yes
Openshift 4	no (1)
Deep Security	no
Deep Security Smart Check	yes
Jenkins	yes
GitLab	no
Endpoints	no
LinkerD	no
Prometheus	yes
Grafana	yes
OPA	no

Core Functionality Availability	GCP	AWS	ESXi
Site Creation	yes	yes	yes (2)
Jumphost Support	yes	yes	no (3)
Site Deployment	yes	yes	yes
Deployment of Endpoints	yes	yes	no
Credential Creation in Jenkins	yes	yes	yes
Pause Environment	yes	yes	no
Schedule Pause Environment	yes	yes	no
Resume Environment	yes	yes	no
Site Termination	yes	yes	no
Configurator	yes	yes	yes

Kubernetes Default Configuration	GCP	AWS	ESXi
Container Runtime	Docker	Docker	Docker
Worker Nodes	3	3	3
Pod Network	Flannel	Flannel	Flannel
Services	URL	URL	NodePort

(1) - Upcoming feature on GCP only

(2) - Pre-existing VMs required, MOADSD-NG will take control of them

(3) - Directly access the services by IP and (Node-)port

Release 04/15/2020

Top News - MOADSD-NG-SERVER and CONFIGURATOR

MOADSD-NG is now easier to setup and use! There's no need for a dedicated Ansible server or patching multiple yaml files anymore to setup and manage the environment. Therefore, proudly announcing the new MOADSD-NG-SERVER and the CONFIGURATOR.

• The MOADSD-NG-SERVER creates a Docker container with everything already built in to run and manage MOADSD-NG. It is designed to work on Mac OS X with Docker for Mac and Linux. Windows support is pending but will come, likely. Please follow the Link to the server and work quickly through the instructions.
• With the help of the CONFIGURATOR, there is no need to edit multiple yaml files anymore. Only one single file needs to be maintained by you, containing only the differences to the default values. Everything else will get generated, which includes some checks for a potential misconfiguration. The CONFIGURATOR is built in to MOADSD-NG. An example configuration can be reviewed here: Link

Headline News

• Updated the wiki to describe the setup and orchestration with Ansible using Python3. This updates includes especially changes on setting up the Ansible server itself.

• Added a simple demo app Troopers for Jenkins & Smart Check demos.

  • https://github.com/mawinkler/troopers
    • Troppers can be admired as demoapp-<0-0-0-0>.nip.io.
    • Be sure to change 0-0-0-0 to the public IP of your jumphost but written with dashes instead of dots.

• Integrated Cloud One Application Security demo capabilities. For this two additional apps were added.

  • https://github.com/mawinkler/c1-app-sec-djangonv
    • Django will be accessible as demoapp1-<0-0-0-0>.nip.io
  • https://github.com/mawinkler/c1-app-sec-moneyx
    • MoneyX will be accessible as demoapp2-<0-0-0-0>.nip.io
  • Ensure to have the following variables defined in your configuration.yml (site_secrets.yml):
    • application_security_key: < GROUP CREDENTIALS KEY >
    • application_security_secret: < GROUP CREDENTIALS SECRET >

• Bumped Jenkins to version 2.204.2 LTS with chart 1.9.19

• Our Jenkins is now able to manage GitHub WebHooks. To enable this you need to define the Personal Access Token from GitHub as a secret within Jenkins. This is done automatically by MOADSD-NG, but you need to define the token within your configuration.yml (site_secrets.yml) before running jenkins_create_credentials.yml.

  • jenkins_github_access_token: < GITHUB ACCESS TOKEN FOR JENKINS >

  Additionally, as of now, go to Manage Jenkins -> Configure System and down to 'GitHub Servers'. Hit 'Add GitHub Server'

  • Name - doesn't matter
  • API URL - https://api.github.com
  • Credentials - choose github-access-token
  • Manage Hooks - checked

  Hit 'Test Connection' to make sure everything is kosher.

• GitLab on Kubernetes is now functional, finally! GitLab does require a load balancer being available for the cluster, which in case of MOADSD-NG needs to be cloud independent. For that reason, the cluster now provides one :). GitLab is currently only deployed and not preconfigured.

  • Ensure to have the following variables defined in your configuration.yml (site_secrets.yml):
    • gitlab_runner_registration_token: <GITLAB RUNNER REGISTRATION TOKEN - YOU GET IT WITHIN GITLAB>
    • When first logging in to GitLab get the runners registration token and paste it to your site_secrets.yml. Then run deploy_gitlab_runnners.yml as usual.

• The cluster registry and load balancer became an intrinsic component of the MOADSD-NG cluster. Therefore both are now being deployed always when the kubernetes cluster itself is deployed.

• Introducing Prometheus and Grafana to MOADSD-NG! Prometheus as the default datasource is already preconfigured. Just add some dashboards from the Grafana library or use the ones provided in the dashboard-directory of MOADSD-NG.

• Bumped Kubernetes to 1.17

  • Please re-fork the demo apps you're using since the kubernetes api has changed in 1.17. Alternatively manually update your app.yml of the respective app.

• Bumped Rook-Ceph to 1.2 and integrated the Ceph Dashboard

• Created dedicated roles for Rook-Ceph and Helm.

• MOADSD-NG is now defaulting to DSaaS.

  • Ensure to have the following variables defined in your configuration.yml (site_secrets.yml):
    • deepsecurity_tenant_id: <DSaaS TENANT ID>
    • deepsecurity_token: <DSaaS TOKEN>

• Integrated a new role for Open Policy Agent.

  • If you choose to deploy opa, test cases to play with the rego policies are automatically created within the directory opa-tests on the kubernetes master.
  • To enable OPA add site_deploy_opa: yes to your configuration.yml.
  • OPA by default is configured to allow all but you can deploy any rego policy you want.

The "one more thing" thing

• patch_docker is not required anymore. After resuming the environment just run jenkins_create_credentials to renew the credentials and kubernetes config within Jenkins.
• terminate now just terminates everything besides the jumphost. This comes in handy if you are using publicly trusted certificates. When recreating the environtment just execute site as usual. The already existing jumphost will be reused. If you want to completely terminate the environment, execute terminate_site.
• Streamlined the menu.
• Cleanup for disassiocated EIPs on AWS now implemented.
• Upgraded the instance types on AWS from t2 to t3 which improves site setup drastically.
• Hopefully fixed the permission problems within Jenkins.
• Jenkins is now using a better, self made Jenkins JNLP Slave Docker Image which is currently provided on my Docker Hub. Build playbook is within roles/jenkins/tasks.
• Fixed the unnecessary and breaking space within the Jenkins slave pod label.
• GitLab on Docker is now removed.

Release 01/31/2020

Release 01/31/2020

Headline news

• It's now possible to let Ansible populate the Jenkins Credentials. You can now directly start over within Jenkins and create pipelines using the cluster registry and Smart Check. See the documentation of Jenkins and the demo video.
• Calico now working for Google GCP and ESXi.
• Major change - GCP and AWS only:
  • Instances which are technically not required to be directly addressable from the internet (e.g. the kubernetes cluster) have been moved to the private subnet. For that reason, Ansible does now use the jumphost instance to do it's orchestration. For the same reason, nat gateways got introduced.
  • If you are running multiple environments at the same time in different clouds, before rerunning the deployment playbook please switch to the target environment by the menu.sh script.
• Major change - GCP and AWS only:
  • All instances besides the jumphost, Deep Security and the Windows endpoints are now only accessible by ssh from the internet. Access to the services is now granted through a reverse proxy with certificates and by DNS names.
  • The direct links are constructed automatically with the following syntax: Given the public IP of your jumphost is 18.197.164.37, the resulting DNS name of a service would be servicename-18-197-164-37.nip.io. So the dots within the IP address are replaced by dashes. See your site_XXX-directory for details.

The "one more thing" thing

• Upgrade from Kubernetes 1.14.4 to 1.14.9
• Switch from cgroupfs to systemd
• Calico firewall fix to allow ip-in-ip encapsulation in Google GCP. Still problematic on AWS.
• Upgrade to Helm 3.0.1 - no Tiller anymore
• After resuming the environment, Kubernetes is now reenabled within Jenkins
• Certificate web-server pod now independent from Docker Hub
• Miau demo app now independent from Docker Hub
• New demo app troopers available
• Smart Check now connects to the pre-registry in addition to the cluster registry
• Fixed username handling for Smart Check

Breaking Changes and Adaptions to take out

When upgrading to the latest version, ensure to have the following variables set.

File:

/etc/ansible/ansible.cfg

force_valid_group_names = ignore

File:

vars/site_secrets.yml

add:

# Your valid e-Mail address, required for Letsencrypt
# Mandatory
admin_email: <YOUR VALID EMAIL ADDRESS>

Files:

vars/environment_gcp_vars.yml
vars/environment_aws_vars.yml

add:

jumphost_tld: nip.io

Initial public release

Initial public release of MOADSD-NG.

The MOADSD-NG project does provide a simple way to setup a hybrid cloud security demo, playground and learning environment within the clouds or alternatively on a local ESXi (no vCenter required). Core technologies used (besides of Trend Micro solutions) are the cloud native virtualization functionalities, a full-blown Kubernetes cluster with cluster storage and release management tools (Jenkins / GitLab). Ansible, the de facto most used orchestration tool, is used for the whole life-cycle of your MOADSD-NG environment.

See release notes in the wiki.