From f63839453506420064afb7350971c65be47d39a6 Mon Sep 17 00:00:00 2001 From: Irving Popovetsky Date: Thu, 20 Jun 2019 16:19:28 -0700 Subject: [PATCH] V5 and the road to a Marketplace-like experience (#50) * Add a marketplace-style solution template. Also a number of bug fixes and enhancements: * Add template version-number support, so that multiple releases can be published to S3 * Fix an issue in automate.yaml where the PublicIp wasn't available in the Outputs, so it bombed * Fix an issue where the dots appended to the domain name caused issues Signed-off-by: Irving Popovetsky * Marketplace v5 - initial working version Signed-off-by: Irving Popovetsky * move final_stage to its own configset, to resolve issues with purging Automate files too early Signed-off-by: Irving Popovetsky * Cost savings part 1: enable gp2 storage for RDS, shrink default instance sizes in marketplace.yaml Signed-off-by: Irving Popovetsky * Cost savings pt 2: Make the number of DB backups tunable and lower the default value Signed-off-by: Irving Popovetsky * Elasticsearch cost reduction, allow 2 instead of 3 Signed-off-by: Irving Popovetsky * Update images * Major cost reductions for the marketplace stack Signed-off-by: Irving Popovetsky * ES6 is not supported with chef-server today Signed-off-by: Irving Popovetsky * Improve browser and CLI launch experience Signed-off-by: Irving Popovetsky * Fix one stray template version Signed-off-by: Irving Popovetsky * fixes #51, fixes #52, fixes #53, bump version to 5.0.2 Signed-off-by: Irving Popovetsky * Increase Automate setup timeout to give smaller instances time to finish. Update AMIs for TCP SACK vulnerability Signed-off-by: Irving Popovetsky * Fix the redirect_uri for Supermarket Signed-off-by: Irving Popovetsky * majorly revamp the Pre-reqs doc to be much clearer Signed-off-by: Irving Popovetsky * New feature: Provide an Automate admin password instead of the file reading shenanigans Signed-off-by: Irving Popovetsky --- PREREQS.md | 49 +- README.md | 37 +- automate.yaml | 110 ++--- chef_elasticsearch.yaml | 24 +- chef_rds.yaml | 36 +- chef_server_ha.yaml | 182 +++----- chef_server_ops_dashboard.yaml | 2 +- files/after.sh | 5 - generate_os_mappings.sh | 18 +- main.yaml | 275 +++++------ marketplace.yaml | 543 ++++++++++++++++++++++ stack_parameters_main.json.example | 50 ++ stack_parameters_marketplace.json.example | 54 +++ supermarket.yaml | 84 +--- 14 files changed, 967 insertions(+), 502 deletions(-) delete mode 100644 files/after.sh create mode 100644 marketplace.yaml create mode 100644 stack_parameters_main.json.example create mode 100644 stack_parameters_marketplace.json.example diff --git a/PREREQS.md b/PREREQS.md index 5eea2f8..bfa8e64 100644 --- a/PREREQS.md +++ b/PREREQS.md @@ -1,46 +1,18 @@ ## Prerequisites -### AMI Acceptance +### Create an external Route53 zone that is resolvable -Before you start, make sure the AMI that is listed in the `backendless_chef.yaml` for your region has been accepted/approved for use in the AWS Marketplace. We are using native Amazon Linux AMIs, the most up to date list is [always here](https://aws.amazon.com/amazon-linux-ami/). +In order to use one of the "full stack" templates such as `main.yaml` or `marketplace.yaml`, you must have a domain that is hosted on Route53. If you're not ready to take the plunge for your entire domain, you can create a subdomain and route all traffic to that (for example chef.mycompany.com). AWS provides [instructions](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/CreatingNewSubdomain.html) and [a video](https://aws.amazon.com/premiumsupport/knowledge-center/create-subdomain-route-53/) on how to do that. -### For SSL -1. You must have a DNS Record Name and Zone Name planned for this cluster to match the SSL certificate. -1. You must have already created and uploaded an SSL cert to AWS. Once you've uploaded the cert, you will need to follow these steps to update the `backendless_chef.yaml` template with the correct values: - * Update the `SSLCertificateARN` parameter with the `ARN` for the SSL Cert. _For info on how to get the ARN, please follow the [AWS Certificate Manger](https://aws.amazon.com/certificate-manager/) docs._ - * **If you're using AWS Route 53**, fill in the `Route53RecordName` and `Route53HostedZone` parameters for the template - * **If you're not using AWS Route 53**, create the DNS record in your own nameservers to be a CNAME record that points at the stack's Application Load Balancer (e.g. my-chef-stack-1942464223.us-west-2.elb.amazonaws.com) - -### Using with Automate - -At this time, this template does not setup Automate as part of it's deployment. However, you can configure it to point to an Automate server that you will setup afterwards, or that is already setup. -1. You should have a token already generated, follow the [instructions here](https://automate.chef.io/docs/data-collection/) on how to generate an Automate token. -1. The Chef Automate URL, for more information on how this URL is formatted, [read this](https://automate.chef.io/docs/data-collection/). +In the parameters, fill in the `Route53HostedZone` parameter to match your Route53 zone, as well as the `AutomateDnsRecordName`, `ChefServerDnsRecordName` and `SupermarketDnsRecordName` values. -### Package Versions -It's recommended to have the packages downloaded and hosted locally before proceeding, an S3 bucket, Artifactory/Nexus or YUM/APT repository cache works well for this purpose. Once you've downloaded the correct EL7 packages and have them hosted, adjust the following variables accordingly to point to the proper URLs. - -* `ChefServerPackage`, `ChefManagePackage`, `PushJobsPackage` - -_Here's an example of setting up an S3 cache:_ +### For SSL -1. Install `mixlib-install` Ruby Gem from the [mixlib-install repo](https://github.com/chef/mixlib-install) on an EC2 instance that has access to the S3 bucket being used. -1. Install the `aws` cli tool on the same instance from [aws cli](https://aws.amazon.com/cli/). -1. Run the following commands: - ``` - mixlib-install download chef-server --platform el --platform-version 7.5 --architecture x86_64 - # Starting download https://packages.chef.io/files/stable/chef-server/12.17.33/el/7/chef-server-core-12.17.33-1.el7.x86_64.rpm - # Download saved to /Users/myname/chef-server-core-12.17.33-1.el7.x86_64.rpm +You must create or upload at least one SSL certficate to AWS Certificate Manager (ACM). If you wish, ACM can provide free SSL certificates for you and automatically manages renewals of those certificates. You may create 3 separate certificates, or a single wildcard certificate (ex: `*.chef.mycompany.com`) that is used in all 3 places. Since your Route53 zone is now working, use the `DNS Validation` option as it is far faster and more convenient. - aws s3 cp /Users/myname/chef-server-core-12.17.33-1.el7.x86_64.rpm s3://mybucket/package-cache/ --acl public-read - # upload: ./chef-server-core-12.17.33-1.el7.x86_64.rpm to s3://mybucket/package-cache/chef-server-core-12.17.33-1.el7.x86_64.rpm +In the parameters, fill in the `ChefSSLCertificateARN`, `AutomateSSLCertificateARN` and `SupermarketSSLCertificateARN` values with the ARNs for the certificates. The ARNs are always viewable in the ACM console, an example ACM ARN looks like: `arn:aws:acm:us-west-2:446539779517:certificate/82d30a13-b420-4f43-80de-9e7872f70b96` - aws s3 presign s3://mybucket/package-cache/chef-server-core-12.17.33-1.el7.x86_64.rpm | cut -d '?' -f 1 - # https://mybucket.s3.amazonaws.com/package-cache/chef-server-core-12.17.33-1.el7.x86_64.rpm - ``` -1. Set the last output (e.g. `https://mybucket.s3.amazonaws.com/package-cache/chef-server-core-12.17.33-1.el7.x86_64.rpm` as shown above) as your `ChefServerPackage` value in `backendless_chef.yaml`. - ## Network @@ -48,14 +20,15 @@ You must already have a VPC setup properly before continuing setting up the stac * Have enough IP's available to assign to nodes * Be split up into 3 subnets, each in different Availability Zones (AZ's) +* Provide the VPC ID and associated subnets to the `VPC` and `ServerSubnets` parameters ## Security -* You should already have created/uploaded an SSH key to AWS and have the ARN available. -* You should already have an Admin SG created for inbound SSH connections. The Security Group ID should be provided for the `InboundAdminSecurityGroupId` parameter in `backendless_chef.yaml`, otherwise you won't have any ssh connectivity to your cluster. +* You should already have created/uploaded an SSH key to AWS. Provide the keypair name to the `KeyName` parameter +* If you're using the `main.yaml` stack, you must also create a security group in the referenced VPC to define your administrative access. Provide sg ID to the `InboundAdminSecurityGroupId` parameter -## Amazon ElasticSearch and Service Linked Role (SLR) +## Amazon Elasticsearch and Service Linked Role (SLR) -Amazon ElasticSearch requires a specific SLR to be created prior to running this CloudFormation template, specifically one called `AWSServiceRoleForAmazonElasticsearchService`. This role cannot be created programmatically as it is created automatically when setting up a VPC access domain in the AWS console. For more information on this [please see this doc from AWS](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/slr-es.html), at this time, even though the doc points to a way to create this manually via the CLI, it still only works via console setup, however AWS adds new features all the time, and by the time you do this, it may work programmatically, please follow their documentation. Once you've created the VPC access domain for AWS ElasticSearch, you can then delete this domain, the role will still be there and you should be able to continue. +Amazon Elasticsearch requires a specific SLR to be created prior to running this CloudFormation template, specifically one called `AWSServiceRoleForAmazonElasticsearchService`. This role cannot be created programmatically as it is created automatically when setting up a VPC access domain in the AWS console. For more information on this [please see this doc from AWS](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/slr-es.html), at this time, even though the doc points to a way to create this manually via the CLI, it still only works via console setup, however AWS adds new features all the time, and by the time you do this, it may work programmatically, please follow their documentation. Once you've created the VPC access domain for AWS ElasticSearch, you can then delete this domain, the role will still be there and you should be able to continue. _Note: You will need to do this for each region you plan on setting up Chef in_ diff --git a/README.md b/README.md index 697cc8d..5f553e1 100644 --- a/README.md +++ b/README.md @@ -33,26 +33,35 @@ Before you fire it up, there are a few things you should make sure you have prep * Please review the [Prerequisites doc before proceeding](./PREREQS.md) -## Fire up the Chef Server stack +## Browser launch instructions (quickstart) You can launch this stack with the push of a button: -

Launch Stack

-However, the most repeatable and least error-prone way to launch this stack is to use the `aws` command-line. First copy file `stack_parameters.json.example` to `stack_parameters.json`, make the necessary changes, then run this command: +| Edition | Version | Est Monthly cost | Launch button | +| ------- | ------- | ---------------- | ------------- | +| High Performance | 5.0.2 | [$1500*](https://calculator.s3.amazonaws.com/index.html#r=IAD&s=EC2&key=calc-E9251374-D80C-45B7-BED7-C9C9778B1D6C) |

Launch Stack

| +| Lowest Cost | 5.0.2 | [$320*](https://calculator.s3.amazonaws.com/index.html#r=IAD&s=EC2&key=files/calc-165638db8e331664846c0d6654d743377bd3eac6&v=ver20190604sQ) |

Launch Stack

| + +_*NOTE: Estimated costs are just that, your mileage will vary. Use reserved instances to greatly reduce costs for long-running clusters._ + +## CLI Launch instructions +However, the most repeatable and least error-prone way to launch this stack is to use the `aws` command-line. First copy file `stack_parameters_main.json.example` to `stack_parameters_main.json`, make the necessary changes, then run this command: ```bash MYBUCKET=aws-native-chef-server MYID=mycompany +VERSION=5.0.2 +EDITION=main # main or marketplace # Configure the automate_stack_parameters.json and then launch the cloudformation stack: # If you're using your own bucket, uncomment the next line: -# aws s3 sync . s3://$MYBUCKET/ --exclude "*" --include "*.yaml" --include "files/*" && \ -aws cloudformation validate-template --template-url https://s3.amazonaws.com/$MYBUCKET/main.yaml && \ +# aws s3 sync . s3://$MYBUCKET/$VERSION/ --exclude "*" --include "*.yaml" --include "files/*" && \ +aws cloudformation validate-template --template-url https://s3.amazonaws.com/$MYBUCKET/$VERSION/$EDITION.yaml && \ aws cloudformation create-stack \ --stack-name ${MYID}-chef-stack \ - --template-url https://s3.amazonaws.com/$MYBUCKET/main.yaml \ + --template-url https://s3.amazonaws.com/$MYBUCKET/$VERSION/$EDITION.yaml \ --capabilities CAPABILITY_IAM \ --on-failure DO_NOTHING \ - --parameters file://stack_parameters.json + --parameters file://stack_parameters_$EDITION.json ``` ## Updating the stack @@ -62,14 +71,16 @@ If you've made changes to the template content or parameters and you wish to upd ```bash MYBUCKET=aws-native-chef-server MYID=mycompany +VERSION=5.0.2 +EDITION=main # main or marketplace # If you're using your own bucket, uncomment the next line: -# aws s3 sync . s3://$MYBUCKET/ --exclude "*" --include "*.yaml" --include "files/*" && \ -aws cloudformation validate-template --template-url https://s3.amazonaws.com/$MYBUCKET/main.yaml && \ +# aws s3 sync . s3://$MYBUCKET/$VERSION/ --exclude "*" --include "*.yaml" --include "files/*" && \ +aws cloudformation validate-template --template-url https://s3.amazonaws.com/$MYBUCKET/$VERSION/$EDITION.yaml && \ aws cloudformation update-stack \ --stack-name ${MYID}-chef-stack \ - --template-url https://s3.amazonaws.com/$MYBUCKET/main.yaml \ + --template-url https://s3.amazonaws.com/$MYBUCKET/$VERSION/$EDITION.yaml \ --capabilities CAPABILITY_IAM \ - --parameters file://stack_parameters.json + --parameters file://stack_parameters_$EDITION.json ``` Note: For production instances it is recommended to use the CloudFormation console so that you can get a report of all changes before executing them. Particularly pay attention to any resources that are being replaced. @@ -79,10 +90,8 @@ Note: For production instances it is recommended to use the CloudFormation conso If you're using a bastion host and need to SSH from the outside: ```bash -ssh -o ProxyCommand="ssh -W %h:%p -q user@bastion" -l user +ssh -o ProxyCommand="ssh -W %h:%p -q centos@bastion" -l centos ``` -Where "user" is `ec2-user` on the RHEL AMI, and `centos` on the CentOS AMI - otherwise just SSH directly to the public IPs of the chef servers ## Upgrading the Chef Server diff --git a/automate.yaml b/automate.yaml index c924202..de7cd43 100644 --- a/automate.yaml +++ b/automate.yaml @@ -1,12 +1,8 @@ AWSTemplateFormatVersion: '2010-09-09' -Description: AWS Native Automate 2 (auto-recovering standalone) v4.1.0 +Description: AWS Native Automate 2 (auto-recovering standalone) v5.0.2 Parameters: # Required Parameters - AutomationBucket: - Default: 'aws-native-chef-server' - Type: String - Description: AWS S3 bucket name that contains all of the cloudformation templates and scripts for this stack VPC: Description: Choose VPC to use Type: AWS::EC2::VPC::Id @@ -52,10 +48,12 @@ Parameters: Default: 1000 ############################################################################### # Security Settings + AutomateAdminPassword: + Description: Provide a password for the Chef Automate admin user + Type: String AutomateDataCollectorToken: Description: The token used for ingesting Data Collector (Chef) events Type: String - Default: 93a49a4f2482c64126f7b6015e6b0f30284287ee4054ff8807fb63d9cbd1c506 LoadBalancerScheme: Description: Network Scheme for the ELB Type: String @@ -77,18 +75,11 @@ Parameters: Route53HostedZone: Type: String Default: '' - Description: Supply a Route 53 Hosted Zone name (eg. mydomain.com.) for auto-creating a DNS record. Must end in a dot. (Leave blank to disable) + Description: Supply a Route 53 Hosted Zone name (eg. mydomain.com) for auto-creating a DNS record. Must NOT end in a dot. (Leave blank to disable) Route53RecordName: Type: String Default: automate Description: Supply a DNS record name that will be prepended to the Route 53 Hosted Zone - AMIFlavor: - Description: Choose the AMI Flavor, the High-Perf CentOS image or the stock and paid official RHEL image - Type: String - Default: 'centos' - AllowedValues: - - 'centos' - - 'rhel' SystemLogsGroup: Description: The logical name of the Cloudwatch Logs log group for systemn logs Type: String @@ -105,8 +96,8 @@ Metadata: Conditions: CreateRoute53Record: !Not [!Equals [ !Ref Route53HostedZone, '' ] ] - RHELAmiFlavor: - !Equals [ !Ref AMIFlavor, 'rhel' ] + UsePublicIp: + !Equals [ !Ref LoadBalancerScheme, 'internet-facing'] Resources: AutomateInstanceProfile: @@ -175,7 +166,7 @@ Resources: BlockDeviceMappings: - DeviceName: /dev/sda1 Ebs: - VolumeSize: 100 + VolumeSize: 20 VolumeType: gp2 DeleteOnTermination: true - DeviceName: /dev/xvdb @@ -203,40 +194,19 @@ Resources: "Fn::Base64": !Sub | #!/bin/bash -ex trap '/opt/aws/bin/cfn-signal -e 1 --region ${AWS::Region} --stack ${AWS::StackName} --resource AutomateServer' ERR - # Update the system first - yum upgrade -y - # Run before script, make sure to install cfn-init if not installed - # attempt to fetch the script first as a signed request (private bucket), then unsigned, then fallback - aws s3 cp s3://${AutomationBucket}/files/before.sh /root/before.sh || \ - curl -s https://${AutomationBucket}.s3.amazonaws.com/files/before.sh -o /root/before.sh || \ - curl -s https://aws-native-chef-server.s3.amazonaws.com/files/before.sh -o /root/before.sh - bash -ex /root/before.sh # Execute AWS::CloudFormation::Init /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource AutomateServer --region ${AWS::Region} /opt/aws/bin/cfn-signal -e 0 --region ${AWS::Region} --stack ${AWS::StackName} --resource AutomateServer CreationPolicy: ResourceSignal: Count: 1 - Timeout: PT10M + Timeout: PT20M Metadata: AWS::CloudFormation::Init: configSets: - default: [awslogs, automate_setup, nightly_snapshots] + default: [awslogs, automate_setup, nightly_snapshots, final_stage] automate_setup: - packages: - yum: - lvm2: [] - xfsprogs: [] - perl: [] - perl-DateTime: [] - perl-Sys-Syslog: [] - perl-LWP-Protocol-https: [] - perl-Digest-SHA: [] - zip: [] - unzip: [] files: - /root/chef-automate.zip: - source: https://packages.chef.io/files/current/automate/latest/chef-automate_linux_amd64.zip /etc/automate-config.toml: content: !Sub - | @@ -330,62 +300,33 @@ Resources: [auth_n.v1.sys.service] a1_data_collector_token = "${AutomateDataCollectorToken}" - { FQDN: !If [CreateRoute53Record, !Ref AutomateDNSRecord, !GetAtt ChefALB.DNSName] } - /etc/sysctl.d/00-chef.conf: - content: | - vm.swappiness=10 - vm.max_map_count=262144 - vm.dirty_ratio=20 - vm.dirty_background_ratio=30 - vm.dirty_expire_centisecs=30000 - /etc/security/limits.d/20-nproc.conf: - content: | - * soft nproc 65536 - * hard nproc 65536 - * soft nofile 1048576 - * hard nofile 1048576 commands: 02_before_script: command: | - sysctl -p /etc/sysctl.d/00-chef.conf mkdir -p /hab pvs | grep nvme1n1 || pvcreate /dev/nvme1n1 vgs | grep data-vg || vgcreate data-vg /dev/nvme1n1 lvs | grep data-lv || (lvcreate -n data-lv -l 95%VG data-vg ; mkfs.xfs /dev/data-vg/data-lv) grep hab /proc/mounts || mount /dev/data-vg/data-lv /hab + grep 'data--vg-data--lv' /etc/fstab || echo '/dev/mapper/data--vg-data--lv /hab xfs defaults 0 0' >> /etc/fstab 03_configure_automate: - command: | - cd /root - unzip chef-automate.zip -d /usr/local/sbin - /usr/local/sbin/chef-automate deploy /etc/automate-config.toml --accept-terms-and-mlsa --skip-preflight - 04_after_script: command: !Sub | - # attempt to fetch the script first as a signed request (private bucket), then unsigned (public bucket) - aws s3 cp s3://${AutomationBucket}/files/after.sh /root/after.sh || \ - curl -s https://${AutomationBucket}.s3.amazonaws.com/files/after.sh -o /root/after.sh - bash -ex /root/after.sh + cd /root + /var/cache/marketplace/chef-automate deploy /etc/automate-config.toml --accept-terms-and-mlsa --skip-preflight --airgap-bundle /var/cache/marketplace/chef-automate.bundle + sleep 10 + /bin/chef-automate iam admin-access restore '${AutomateAdminPassword}' 05_configure_cloudwatch_monitoring: command: | - mkdir /opt/cloudwatch_monitoring - cd /opt/cloudwatch_monitoring - curl -s -O http://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.1.zip - unzip CloudWatchMonitoringScripts-1.2.1.zip - rm -f CloudWatchMonitoringScripts-1.2.1.zip crontab -l | { cat; echo "*/5 * * * * /opt/cloudwatch_monitoring/aws-scripts-mon/mon-put-instance-data.pl --mem-util --disk-space-util --disk-path=/ --from-cron"; } | crontab - + crontab -l | { cat; echo "*/5 * * * * /opt/cloudwatch_monitoring/aws-scripts-mon/mon-put-instance-data.pl --disk-space-util --disk-path=/hab --from-cron"; } | crontab - nightly_snapshots: - files: - /usr/local/bin/ebs-snapshot.sh: - source: https://raw.githubusercontent.com/CaseyLabs/aws-ec2-ebs-automatic-snapshot-bash/master/ebs-snapshot.sh - mode: '000755' commands: create_nightly_backup_cronjob: command: | crontab -l | { cat; echo "55 23 * * * root /usr/local/bin/ebs-snapshot.sh"; } | crontab - awslogs: files: - /root/awslogs-agent-setup.py: - source: 'https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py' - mode: '000755' /etc/awslogs/awscli.conf: content: !Sub | [default] @@ -440,7 +381,16 @@ Resources: 01_setup: command: !Sub | mkdir -p /var/lib/awslogs - python /root/awslogs-agent-setup.py --region ${AWS::Region} --non-interactive -c /etc/awslogs/awslogs.conf + python /var/cache/marketplace/awslogs-agent-setup.py --region ${AWS::Region} --non-interactive -c /etc/awslogs/awslogs.conf + final_stage: + commands: + 09_cleanup: + command: | + # clean up the chef package cache, to maximize available disk space after install + if [ -d /var/cache/marketplace ]; then + rm -f /var/cache/marketplace/*.rpm + rm -f /var/cache/marketplace/*.aib + fi # EC2 System AutoRecovery # https://aws.amazon.com/blogs/aws/new-auto-recovery-for-amazon-ec2/ @@ -489,9 +439,9 @@ Resources: Type: AWS::Route53::RecordSet Condition: CreateRoute53Record Properties: - HostedZoneName: !Ref Route53HostedZone + HostedZoneName: !Sub ${Route53HostedZone}. + Name: !Sub ${Route53RecordName}.${Route53HostedZone}. Comment: !Sub Created by Cloudformation ${AWS::StackName} - Name: !Sub ${Route53RecordName}.${Route53HostedZone} Type: CNAME TTL: '900' ResourceRecords: @@ -536,7 +486,7 @@ Resources: - Name: MountPath Value: '/' - Name: Filesystem - Value: !If [RHELAmiFlavor, "/dev/nvme0n1p2", "/dev/nvme0n1p1"] + Value: "/dev/nvme0n1p1" DiskSpaceUtilizationDataDisk: Type: AWS::CloudWatch::Alarm @@ -603,4 +553,4 @@ Outputs: Name: !Sub "${AWS::StackName}-DNSName" ServerIP: Description: The IP Address of the Automate Server - Value: !GetAtt AutomateServer.PublicIp + Value: !If [UsePublicIp, !GetAtt AutomateServer.PublicIp, !GetAtt AutomateServer.PrivateIp] diff --git a/chef_elasticsearch.yaml b/chef_elasticsearch.yaml index 1ac6a3e..9b7be64 100644 --- a/chef_elasticsearch.yaml +++ b/chef_elasticsearch.yaml @@ -1,5 +1,5 @@ AWSTemplateFormatVersion: '2010-09-09' -Description: Chef Server Elasticsearch v4.1.0 +Description: Chef Server Elasticsearch 5.0.2 Parameters: # Required Parameters @@ -27,15 +27,16 @@ Parameters: 'm3.medium.elasticsearch', 'm3.large.elasticsearch', 'm3.xlarge.elasticsearch', 'm3.medium.elasticsearch', 'r3.large.elasticsearch', 'r3.xlarge.elasticsearch', 'r3.2xlarge.elasticsearch', 'r3.4xlarge.elasticsearch', 'r3.8xlarge.elasticsearch' ] ElasticSearchVersion: - Description: Version of ElasticSearch to use. Chef 12.16 supports v2.3 or 5.x. (5.5 recommended for new installs) + Description: Version of ElasticSearch to use Type: String - Default: '5.5' + Default: '5.6' AllowedValues: - '2.3' - '5.3' - '5.5' + - '5.6' ElasticSearchShardCount: - Description: Number of ElasticSearch hosts to provision at launch (3 recommended) + Description: Number of ElasticSearch hosts to provision at launch (3 recommended, 2 provides HA) Default: 3 Type: Number ############################################################################### @@ -52,6 +53,10 @@ Parameters: Description: SNS topic Type: String +Conditions: + 2ZoneES: + !Equals [!Ref ElasticSearchShardCount, 2] + Resources: # ElasticSearch ######################################################################################### @@ -72,7 +77,7 @@ Resources: ElasticsearchVersion: !Ref ElasticSearchVersion ElasticsearchClusterConfig: InstanceCount: !Sub ${ElasticSearchShardCount} - ZoneAwarenessEnabled: false + ZoneAwarenessEnabled: !If [2ZoneES, true, false] InstanceType: !Ref ElasticSearchInstanceType DedicatedMasterEnabled: false SnapshotOptions: @@ -86,8 +91,11 @@ Resources: Action: "es:*" Resource: !Sub "arn:aws:es:${AWS::Region}:${AWS::AccountId}:*" VPCOptions: - SubnetIds: - - !Select [ 0, !Ref ChefServerSubnets ] + SubnetIds: !If [ + 2ZoneES, + [ !Select [ 0, !Ref ChefServerSubnets ], !Select [ 1, !Ref ChefServerSubnets ]], + [ !Select [ 0, !Ref ChefServerSubnets ] ] + ] SecurityGroupIds: - !Ref ESSecurityGroup AdvancedOptions: @@ -155,7 +163,7 @@ Resources: Period: 60 # Reports when any node in the cluster falls below the threshold. Statistic: Minimum - Threshold: 5000 + Threshold: 2000 Dimensions: - Name: DomainName Value: !Ref ElasticsearchDomain diff --git a/chef_rds.yaml b/chef_rds.yaml index 9e1990c..ec774ba 100644 --- a/chef_rds.yaml +++ b/chef_rds.yaml @@ -1,5 +1,5 @@ AWSTemplateFormatVersion: '2010-09-09' -Description: Chef Server PostgreSQL v4.1.0 +Description: Chef Server PostgreSQL v5.0.2 Parameters: # Required Parameters @@ -22,12 +22,18 @@ Parameters: ############################################################################### # Performance Settings DBInstanceClass: - Description: EC2 Instance type for RDS DBs (EBS Optimized instances recommended) - Default: 'db.m4.large' + Description: EC2 Instance type for RDS DBs (M5 class recommended for performance) + Default: 'db.m5.large' Type: String - AllowedValues: [db.t2.medium, db.t2.large, db.t2.xlarge, db.t2.2xlarge, + AllowedValues: [db.t3.small, db.t3.medium, db.t3.large, db.t3.xlarge, db.t3.2xlarge, db.m4.large, db.m4.xlarge, db.m4.2xlarge, db.m4.4xlarge, db.m4.10xlarge, db.m4.16xlarge, - db.r4.large, db.r4.xlarge, db.r4.2xlarge, db.r4.4xlarge, db.r4.8xlarge, db.r4.16xlarge] + db.r4.large, db.r4.xlarge, db.r4.2xlarge, db.r4.4xlarge, db.r4.8xlarge, db.r4.16xlarge, + db.m5.large, db.m5.xlarge, db.m5.2xlarge, db.m5.4xlarge, db.m5.12xlarge, db.m5.24xlarge] + DBStorageType: + Description: Select gp2 or io1 DB storage. io1 is much higher performing, must select minimum 100GB storage and an IOPS value + Default: 'gp2' + Type: String + AllowedValues: [gp2, io1] DBAllocatedStorage: Description: Storage size allocated for the database Default: '100' @@ -42,11 +48,23 @@ Parameters: Description: Supply a security group for your chef frontends Type: AWS::EC2::SecurityGroup::Id ############################################################################### + # Backup settings + BackupDaystoKeep: + Description: Number of days of automatic backups to keep + Default: 7 + Type: Number + MinValue: 0 + MaxValue: 35 + ############################################################################### # Input from Parent Stack AlertNotificationTopic: Description: SNS topic Type: String +Conditions: + UsePiopsStorage: + !Equals [ !Ref DBStorageType, 'io1' ] + Resources: DBSecurityGroup: Type: AWS::EC2::SecurityGroup @@ -71,17 +89,17 @@ Resources: Properties: DBName: chef AllocatedStorage: !Ref DBAllocatedStorage - Iops: !Ref DBIops + Iops: !If [UsePiopsStorage, !Ref DBIops, 0] MasterUsername: chefadmin MasterUserPassword: !Ref DBPassword DBInstanceClass: !Ref DBInstanceClass - StorageType: io1 + StorageType: !Ref DBStorageType MultiAZ: true Engine: postgres - EngineVersion: 9.6.8 + EngineVersion: 9.6.12 AllowMajorVersionUpgrade: true AutoMinorVersionUpgrade: true - BackupRetentionPeriod: '35' + BackupRetentionPeriod: !Ref BackupDaystoKeep DBSubnetGroupName: !Ref DBSubnetGroup VPCSecurityGroups: - !Ref DBSecurityGroup diff --git a/chef_server_ha.yaml b/chef_server_ha.yaml index 5af2892..705e021 100644 --- a/chef_server_ha.yaml +++ b/chef_server_ha.yaml @@ -1,5 +1,5 @@ AWSTemplateFormatVersion: '2010-09-09' -Description: AWS Native Chef Server (auto-scaling HA) v4.1.0 +Description: AWS Native Chef Server (auto-scaling HA) v5.0.2 Parameters: # Required Parameters @@ -60,12 +60,18 @@ Parameters: Description: Enter how many licenses you have purchased Type: String DBInstanceClass: - Description: EC2 Instance type for RDS DBs (EBS Optimized instances recommended) - Default: 'db.m4.large' + Description: EC2 Instance type for RDS DBs (M5 class recommended for performance) + Default: 'db.m5.large' Type: String - AllowedValues: [db.t2.medium, db.t2.large, db.t2.xlarge, db.t2.2xlarge, + AllowedValues: [db.t3.small, db.t3.medium, db.t3.large, db.t3.xlarge, db.t3.2xlarge, db.m4.large, db.m4.xlarge, db.m4.2xlarge, db.m4.4xlarge, db.m4.10xlarge, db.m4.16xlarge, - db.r4.large, db.r4.xlarge, db.r4.2xlarge, db.r4.4xlarge, db.r4.8xlarge, db.r4.16xlarge] + db.r4.large, db.r4.xlarge, db.r4.2xlarge, db.r4.4xlarge, db.r4.8xlarge, db.r4.16xlarge, + db.m5.large, db.m5.xlarge, db.m5.2xlarge, db.m5.4xlarge, db.m5.12xlarge, db.m5.24xlarge] + DBStorageType: + Description: Select gp2 or io1 DB storage. io1 is much higher performing, must select minimum 100GB storage and an IOPS value + Default: 'gp2' + Type: String + AllowedValues: [gp2, io1] DBAllocatedStorage: Description: Storage size allocated for the database Default: '100' @@ -74,6 +80,12 @@ Parameters: Description: IOPS allocated to the storage (storage size * 10) Default: '1000' Type: String + DBBackupDaystoKeep: + Description: Number of days of automatic backups to keep + Default: 7 + Type: Number + MinValue: 0 + MaxValue: 35 ElasticSearchInstanceType: Description: The Instance type to use for ElasticSearch instances (Note, must have ephemeral storage, the instance type affects the total amount of elasticsearch storage. i3 strongly recommended) Type: String @@ -84,36 +96,23 @@ Parameters: 'm3.medium.elasticsearch', 'm3.large.elasticsearch', 'm3.xlarge.elasticsearch', 'm3.medium.elasticsearch', 'r3.large.elasticsearch', 'r3.xlarge.elasticsearch', 'r3.2xlarge.elasticsearch', 'r3.4xlarge.elasticsearch', 'r3.8xlarge.elasticsearch' ] ElasticSearchVersion: - Description: Version of ElasticSearch to use. Chef 12.16 supports v2.3 or 5.x. (5.5 recommended for new installs) + Description: Version of ElasticSearch to use Type: String - Default: '5.5' + Default: '5.6' AllowedValues: - '2.3' - '5.3' - '5.5' + - '5.6' ElasticSearchShardCount: Description: Number of ElasticSearch hosts to provision at launch (3 recommended) Default: 3 Type: Number ElasticSearchReplicaCount: - Description: Replication factor for ElasticSearch sha vrds (how many extra copies to keep) + Description: Replication factor for ElasticSearch shards (how many extra copies to keep) Default: 2 Type: Number ############################################################################### - # Package Versions & Locations - ChefServerPackage: - Description: The URL to the chef server EL7 (chef-server-core) package which will be downloaded - Type: String - Default: 'https://packages.chef.io/files/stable/chef-server/12.17.33/el/7/chef-server-core-12.17.33-1.el7.x86_64.rpm' - ChefManagePackage: - Description: The URL to the chef-manage EL7 package which will be downloaded - Type: String - Default: 'https://packages.chef.io/files/stable/chef-manage/2.5.16/el/7/chef-manage-2.5.16-1.el7.x86_64.rpm' - PushJobsPackage: - Description: The URL to the push jobs server package which will be downloaded - Type: String - Default: 'https://packages.chef.io/files/stable/opscode-push-jobs-server/2.2.8/el/7/opscode-push-jobs-server-2.2.8-1.el7.x86_64.rpm' - ############################################################################### # Security Settings LoadBalancerScheme: Description: Network Scheme for the ELB @@ -166,27 +165,27 @@ Parameters: Default: '' Type: String Description: The S3 location of a custom chef-server configuration snippet that will be injected into the chef-server.rb (optional) + SupermarketUrl: + Description: The URL of the Private Supermarket Server, for oc-id authentication. Leave blank to disable. + Type: String + Default: '' Route53HostedZone: Type: String Default: '' - Description: Supply a Route 53 Hosted Zone name (eg. mydomain.com.) for auto-creating a DNS record. Must end in a dot. (Leave blank to disable) + Description: Supply a Route 53 Hosted Zone name (eg. mydomain.com) for auto-creating a DNS record. Must NOT end in a dot. (Leave blank to disable) Route53RecordName: Type: String Default: 'chef' Description: Supply a DNS record name that will be prepended to the Route 53 Hosted Zone - AMIFlavor: - Description: Choose the AMI Flavor, the High-Perf CentOS image or the stock and paid official RHEL image - Type: String - Default: 'centos' - AllowedValues: - - 'centos' - - 'rhel' SystemLogsGroup: Description: The logical name of the Cloudwatch Logs log group for systemn logs Type: String AppLogsGroup: Description: The logical name of the Cloudwatch Logs log group for systemn logs Type: String + TemplateVersion: + Type: String + Default: "5.0.2" Metadata: cfn-lint: @@ -197,8 +196,6 @@ Metadata: Conditions: CreateRoute53Record: !Not [!Equals [ !Ref Route53HostedZone, '' ] ] - RHELAmiFlavor: - !Equals [ !Ref AMIFlavor, 'rhel' ] Resources: # Frontend Autoscale Groups @@ -426,19 +423,8 @@ Resources: UserData: "Fn::Base64": !Sub | #!/bin/bash -ex - trap "{ - /opt/aws/bin/cfn-signal -e 1 --region ${AWS::Region} --stack ${AWS::StackName} --resource ServerLaunchConfig - # Uncomment this next line before going to production - # aws autoscaling set-instance-health --health-status Unhealthy --region ${AWS::Region} --instance-id $(curl -s http://169.254.169.254/latest/meta-data/instance-id) - }" ERR - # Update the system first - yum upgrade -y - # Run before script, make sure to install cfn-init if not installed - # attempt to fetch the script first as a signed request (private bucket), then unsigned (public bucket) - aws s3 cp s3://${AutomationBucket}/files/before.sh /root/before.sh || \ - curl -s https://${AutomationBucket}.s3.amazonaws.com/files/before.sh -o /root/before.sh || \ - curl -s https://aws-native-chef-server.s3.amazonaws.com/files/before.sh -o /root/before.sh - bash -ex /root/before.sh + # Uncomment this next line before going to production, note: It can cause instances to cycle rapidly, watch Cloudwatch Logs for errors + # trap "aws autoscaling set-instance-health --health-status Unhealthy --region ${AWS::Region} --instance-id $(curl -s http://169.254.169.254/latest/meta-data/instance-id)" ERR # Execute AWS::CloudFormation::Init /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource ServerLaunchConfig --region ${AWS::Region} # All is well so signal success and let CF know wait function is complete @@ -446,12 +432,9 @@ Resources: Metadata: AWS::CloudFormation::Init: configSets: - default: [awslogs, es_proxy_setup, chef_server_setup, filebeat] + default: [awslogs, es_proxy_setup, chef_server_setup, filebeat, final_stage] es_proxy_setup: files: - /usr/local/bin/aws-signing-proxy: - source: https://github.com/chef-customers/aws-signing-proxy/releases/download/v0.5.0/aws-signing-proxy - mode: '000755' /etc/systemd/system/aws-signing-proxy.service: content: | [Unit] @@ -474,21 +457,9 @@ Resources: stdout-log: true commands: 01_start_es_proxy: - command: systemctl daemon-reload && systemctl start aws-signing-proxy + command: | + systemctl daemon-reload && systemctl start aws-signing-proxy chef_server_setup: - packages: - rpm: - chef-server-core: !Sub ${ChefServerPackage} - chef-manage: !Sub ${ChefManagePackage} - opscode-push-jobs-server: !Sub ${PushJobsPackage} - yum: - perl: [] - perl-DateTime: [] - perl-Sys-Syslog: [] - perl-LWP-Protocol-https: [] - perl-Digest-SHA: [] - zip: [] - unzip: [] files: /etc/opscode-push-jobs-server/opscode-push-jobs-server.rb: content: !Sub | @@ -538,34 +509,27 @@ Resources: rabbitmq['queue_length_monitor_enabled'] = false opscode_expander['enable'] = false dark_launch['actions'] = false - oc_id['applications'] ||= {} - oc_id['applications']['supermarket'] = { - 'redirect_uri' => 'https://supermarket.${Route53HostedZone}/auth/chef_oauth2/callback' - } + unless '${SupermarketUrl}'.empty? + oc_id['applications'] ||= {} + oc_id['applications']['supermarket'] = { + 'redirect_uri' => '${SupermarketUrl}/auth/chef_oauth2/callback' + } + end default_orgname '${ChefDefaultOrgName}'.empty? ? null : '${ChefDefaultOrgName}' # special handling for Chef Automate URL and Token which might be empty strings unless '${ChefAutomateServerUrl}'.empty? || '${ChefAutomateToken}'.empty? - data_collector['root_url'] = '${ChefAutomateServerUrl}' + data_collector['root_url'] = '${ChefAutomateServerUrl}/data-collector/v0/' data_collector['token'] = '${ChefAutomateToken}' + profiles['root_url'] = '${ChefAutomateServerUrl}' end - { DNSNAME: !If [CreateRoute53Record, !Sub "${Route53RecordName}.${Route53HostedZone}", !GetAtt ChefALB.DNSName] } - /etc/sysctl.d/00-chef.conf: - content: | - vm.swappiness=10 - vm.max_map_count=262144 - vm.dirty_ratio=20 - vm.dirty_background_ratio=30 - vm.dirty_expire_centisecs=30000 - /etc/security/limits.d/20-nproc.conf: - content: | - * soft nproc 65536 - * hard nproc 65536 - * soft nofile 1048576 - * hard nofile 1048576 commands: 02_before_script: command: !Sub | - sysctl -p /etc/sysctl.d/00-chef.conf + # If we're using a marketplace, install locally cached packages + pushd /var/cache/marketplace + rpm -ivh chef-server-core.rpm chef-manage.rpm push-jobs-server.rpm + popd if [ -n '${ChefServerCustomConfig}' ]; then aws s3 cp ${ChefServerCustomConfig} /etc/opscode/chef-server-custom.rb echo "# Appended from chef-server-custom.rb" >> /etc/opscode/chef-server.rb @@ -577,29 +541,13 @@ Resources: export BUCKET=${ChefSecretsBucket} export AWS_REGION=${AWS::Region} export WAITHANDLE="${WaitHandle}" - # attempt to fetch the script first as a signed request (private bucket), then unsigned (public bucket) - aws s3 cp s3://${AutomationBucket}/files/main.sh /root/main.sh || \ - aws s3 cp --no-sign-request s3://${AutomationBucket}/files/main.sh /root/main.sh - bash -x /root/main.sh - 04_after_script: - command: !Sub | - # attempt to fetch the script first as a signed request (private bucket), then unsigned (public bucket) - aws s3 cp s3://${AutomationBucket}/files/after.sh /root/after.sh || \ - curl -s https://${AutomationBucket}.s3.amazonaws.com/files/after.sh -o /root/after.sh - bash -ex /root/after.sh + + bash -x /var/cache/marketplace/main.sh 05_configure_cloudwatch_monitoring: command: | - mkdir /opt/cloudwatch_monitoring - cd /opt/cloudwatch_monitoring - curl http://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.1.zip -O - unzip CloudWatchMonitoringScripts-1.2.1.zip - rm -f CloudWatchMonitoringScripts-1.2.1.zip crontab -l | { cat; echo "*/5 * * * * /opt/cloudwatch_monitoring/aws-scripts-mon/mon-put-instance-data.pl --auto-scaling --mem-util --disk-space-util --disk-path=/ --from-cron"; } | crontab - awslogs: files: - /root/awslogs-agent-setup.py: - source: 'https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py' - mode: '000755' /etc/awslogs/awscli.conf: content: !Sub | [default] @@ -706,11 +654,8 @@ Resources: 01_setup: command: !Sub | mkdir -p /var/lib/awslogs - python /root/awslogs-agent-setup.py --region ${AWS::Region} --non-interactive -c /etc/awslogs/awslogs.conf + python /var/cache/marketplace/awslogs-agent-setup.py --region ${AWS::Region} --non-interactive -c /etc/awslogs/awslogs.conf filebeat: - packages: - rpm: - filebeat: https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.6-x86_64.rpm files: /etc/filebeat/filebeat.yml: content: | @@ -732,7 +677,18 @@ Resources: group: root commands: 01_systemd: - command: systemctl daemon-reload && systemctl start filebeat.service + command: | + rpm -ivh /var/cache/marketplace/filebeat-5.6.6-x86_64.rpm + systemctl daemon-reload && systemctl start filebeat.service + final_stage: + commands: + 09_cleanup: + command: | + # clean up the chef package cache, to maximize available disk space after install + if [ -d /var/cache/marketplace ]; then + rm -f /var/cache/marketplace/*.rpm + rm -f /var/cache/marketplace/*.aib + fi WaitCondition: Type: AWS::CloudFormation::WaitCondition @@ -749,7 +705,7 @@ Resources: PostgreSQLStack: Type: AWS::CloudFormation::Stack Properties: - TemplateURL: !Sub https://s3.amazonaws.com/${AutomationBucket}/chef_rds.yaml + TemplateURL: !Sub https://s3.amazonaws.com/${AutomationBucket}/${TemplateVersion}/chef_rds.yaml Parameters: VPC: !Sub ${VPC} ChefServerSubnets: !Join [ ",", !Ref ChefServerSubnets ] @@ -757,8 +713,10 @@ Resources: ContactEmail: !Ref ContactEmail ContactDept: !Ref ContactDept DBInstanceClass: !Ref DBInstanceClass + DBStorageType: !Ref DBStorageType DBAllocatedStorage: !Ref DBAllocatedStorage DBIops: !Ref DBIops + BackupDaystoKeep: !Ref DBBackupDaystoKeep FrontendSecurityGroupId: !Ref FrontendSecurityGroupId AlertNotificationTopic: !Ref AlertNotificationTopic @@ -767,7 +725,7 @@ Resources: ElasticsearchStack: Type: AWS::CloudFormation::Stack Properties: - TemplateURL: !Sub https://s3.amazonaws.com/${AutomationBucket}/chef_elasticsearch.yaml + TemplateURL: !Sub https://s3.amazonaws.com/${AutomationBucket}/${TemplateVersion}/chef_elasticsearch.yaml Parameters: VPC: !Sub ${VPC} ChefServerSubnets: !Join [ ",", !Ref ChefServerSubnets ] @@ -786,9 +744,9 @@ Resources: Type: AWS::Route53::RecordSet Condition: CreateRoute53Record Properties: - HostedZoneName: !Ref Route53HostedZone + HostedZoneName: !Sub ${Route53HostedZone}. + Name: !Sub ${Route53RecordName}.${Route53HostedZone}. Comment: !Sub Created by Cloudformation ${AWS::StackName} - Name: !Sub ${Route53RecordName}.${Route53HostedZone} Type: CNAME TTL: '900' ResourceRecords: @@ -800,7 +758,7 @@ Resources: OpsDashboardStack: Type: AWS::CloudFormation::Stack Properties: - TemplateURL: !Sub https://s3.amazonaws.com/${AutomationBucket}/chef_server_ops_dashboard.yaml + TemplateURL: !Sub https://s3.amazonaws.com/${AutomationBucket}/${TemplateVersion}/chef_server_ops_dashboard.yaml Parameters: BootstrapAutoScaleGroup: !Ref BootstrapAutoScaleGroup FrontendAutoScaleGroup: !Ref FrontendAutoScaleGroup @@ -831,7 +789,7 @@ Resources: - Name: MountPath Value: '/' - Name: Filesystem - Value: !If [RHELAmiFlavor, "/dev/nvme0n1p2", "/dev/nvme0n1p1"] + Value: '/dev/nvme0n1p1' BootstrapDiskSpaceUtilizationNvme: Type: AWS::CloudWatch::Alarm @@ -853,7 +811,7 @@ Resources: - Name: MountPath Value: '/' - Name: Filesystem - Value: !If [RHELAmiFlavor, "/dev/nvme0n1p2", "/dev/nvme0n1p1"] + Value: '/dev/nvme0n1p1' MemoryUtilization: Type: AWS::CloudWatch::Alarm diff --git a/chef_server_ops_dashboard.yaml b/chef_server_ops_dashboard.yaml index 84731c1..35dc58d 100644 --- a/chef_server_ops_dashboard.yaml +++ b/chef_server_ops_dashboard.yaml @@ -1,5 +1,5 @@ AWSTemplateFormatVersion: '2010-09-09' -Description: AWS Native OpsDashboard for Chef Server v4.1.0 +Description: AWS Native OpsDashboard for Chef Server v5.0.2 Parameters: # Required Parameters diff --git a/files/after.sh b/files/after.sh deleted file mode 100644 index 60c817c..0000000 --- a/files/after.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/bin/bash - -# place your customizations here, for things that run after main.sh - -exit 0 diff --git a/generate_os_mappings.sh b/generate_os_mappings.sh index 37b5a48..4e24a71 100755 --- a/generate_os_mappings.sh +++ b/generate_os_mappings.sh @@ -1,23 +1,17 @@ #!/bin/bash -# get the latest Name value from: -# RHEL: aws ec2 describe-images --owners 309956199498 --filters "Name=name,Values=RHEL-7.6*" --query "Images[*].Name" --output text -# CentOS highperf: `aws ec2 describe-images --owners 446539779517 --filters "Name=name,Values=chef-highperf-centos7*" --query "Images[*].Name" | sort` - -RHEL_RELEASE='RHEL-7.6_HVM_GA-20190128-x86_64-0-Hourly2-GP2' -CENTOS_RELEASE='chef-highperf-centos7-201902131539' +CENTOS_RELEASE='aws-native-chef-server-5.0.*' +IMAGE_OWNERID="406084061336" printf "Mappings:\n AMI:\n" regions=$(aws ec2 describe-regions --query "Regions[].RegionName" --output text) for region in $regions; do - rhel_ami=$(aws --region $region ec2 describe-images \ - --filters "Name=name,Values=${RHEL_RELEASE}" \ - --query "Images[0].ImageId" --output "text") - centos_ami=$(aws --region $region ec2 describe-images \ + --owners $IMAGE_OWNERID \ --filters "Name=name,Values=${CENTOS_RELEASE}" \ - --query "Images[0].ImageId" --output "text") + --query "Images[0].ImageId" \ + --output "text") - printf " $region:\n rhel: $rhel_ami\n centos: $centos_ami\n" + printf " $region:\n centos: $centos_ami\n" done diff --git a/main.yaml b/main.yaml index 3b55e46..44457bd 100644 --- a/main.yaml +++ b/main.yaml @@ -1,5 +1,5 @@ AWSTemplateFormatVersion: '2010-09-09' -Description: AWS Native Chef Stack v4.1.0 +Description: AWS Native Chef Stack v5.0.2 Parameters: # Required Parameters @@ -27,6 +27,21 @@ Parameters: Type: String ############################################################################### # Automate Stack Settings + ChefAutomateAdminPassword: + Description: Provide a password for the Chef Automate admin user + NoEcho: true + Type: String + MinLength: 8 + MaxLength: 64 + ConstraintDescription: Must be at least 8 characters + ChefAutomateToken: + Description: The secret token used to communicate with the Chef Automate Server, for forwarding Visibility data. Leave blank to disable. + Type: String + Default: 93a49a4f2482c64126f7b6015e6b0f30284287ee4054ff8807fb63d9cbd1c506 + AllowedPattern: ^[a-z0-9]*$ + MinLength: 64 + MaxLength: 64 + ConstraintDescription: Must be 64 alphanumeric characters AutomateSSLCertificateARN: Description: SSL Certficate ARN for SSL Certficate (must begin with arn:aws:iam::) Type: String @@ -45,6 +60,8 @@ Parameters: Description: Amount of storage space to allocate on a dedicated data volume (must be between 500 and 16000) Type: Number Default: 1000 + MinValue: 500 + MaxValue: 16000 ############################################################################### # Chef Server Stack Settings ChefSSLCertificateARN: @@ -55,33 +72,47 @@ Parameters: Type: String Default: 'chef' ChefInstanceType: - Description: EC2 Instance type for Chef Server Frontends (high-CPU recommended) + Description: EC2 Instance type for Chef Server Frontends (C5 recommended for performance) Default: c5.large Type: String AllowedValues: [t3.small, t3.medium, t3.large, t3.xlarge, t3.2xlarge, m5.large, m5.xlarge, m5.2xlarge, m5.4xlarge, m5.12xlarge, m5.24xlarge, c5.large, c5.xlarge, c5.2xlarge, c5.4xlarge, c5.9xlarge, c5.18xlarge] ChefDBInstanceClass: - Description: EC2 Instance type for RDS DBs (EBS Optimized instances recommended) - Default: 'db.m4.large' + Description: EC2 Instance type for RDS DBs (M5 class recommended for performance) + Default: 'db.m5.large' Type: String - AllowedValues: [db.t2.medium, db.t2.large, db.t2.xlarge, db.t2.2xlarge, + AllowedValues: [db.t3.medium, db.t3.large, db.t3.xlarge, db.t3.2xlarge, db.m4.large, db.m4.xlarge, db.m4.2xlarge, db.m4.4xlarge, db.m4.10xlarge, db.m4.16xlarge, - db.r4.large, db.r4.xlarge, db.r4.2xlarge, db.r4.4xlarge, db.r4.8xlarge, db.r4.16xlarge] + db.r4.large, db.r4.xlarge, db.r4.2xlarge, db.r4.4xlarge, db.r4.8xlarge, db.r4.16xlarge, + db.m5.large, db.m5.xlarge, db.m5.2xlarge, db.m5.4xlarge, db.m5.12xlarge, db.m5.24xlarge] + ChefDBStorageType: + Description: Select gp2 or io1 DB storage. io1 is much higher performing, must select minimum 100GB storage and an IOPS value + Default: 'io1' + Type: String + AllowedValues: [gp2, io1] ChefDBAllocatedStorage: - Description: Storage size allocated for the database + Description: Storage size allocated for the database (gp2 minimum 20, maximum 16000) (io1 minimum 100, maximum 32768) Default: '100' Type: String ChefDBIops: - Description: IOPS allocated to the storage (storage size * 10) + Description: IOPS reserved for DB storage, only applicable if ChefDBStorageType is io1 (ChefDBAllocatedStorage * 10) Default: '1000' Type: String + ChefDBBackupDaystoKeep: + Description: Number of days of automatic backups to keep + Default: 7 + Type: Number + MinValue: 0 + MaxValue: 35 ChefDBPassword: - Description: Enter DB Password + Description: Create a strong RDS admin password NoEcho: true Type: String + MinLength: 8 + MaxLength: 64 ChefElasticSearchInstanceType: - Description: The Instance type to use for ElasticSearch instances (Note, must have ephemeral storage, the instance type affects the total amount of elasticsearch storage. i3 strongly recommended) + Description: The Instance type to use for ElasticSearch instances (i3 recommended for performance) Type: String Default: 'i3.large.elasticsearch' AllowedValues: [ @@ -90,21 +121,19 @@ Parameters: 'm3.medium.elasticsearch', 'm3.large.elasticsearch', 'm3.xlarge.elasticsearch', 'm3.medium.elasticsearch', 'r3.large.elasticsearch', 'r3.xlarge.elasticsearch', 'r3.2xlarge.elasticsearch', 'r3.4xlarge.elasticsearch', 'r3.8xlarge.elasticsearch' ] ChefElasticSearchVersion: - Description: Version of ElasticSearch to use. Chef 12.16 supports v2.3 or 5.x. (5.5 recommended for new installs) + Description: Version of ElasticSearch to use. (5.6 recommended for new installs) Type: String - Default: '5.5' + Default: '5.6' AllowedValues: - - '2.3' - '5.3' - '5.5' + - '5.6' ChefElasticSearchShardCount: - Description: Number of ElasticSearch hosts to provision at launch (3 recommended) + Description: Number of ElasticSearch hosts to provision at launch (3 recommended, 2 provides HA) Default: 3 Type: Number - ChefElasticSearchReplicaCount: - Description: Replication factor for ElasticSearch sha vrds (how many extra copies to keep) - Default: 2 - Type: Number + MinValue: 2 + MaxValue: 3 ############################################################################### # Automate Stack Settings SupermarketSSLCertificateARN: @@ -124,7 +153,7 @@ Parameters: SupermarketDataVolumeSize: Description: Amount of storage space to allocate for the Supermarket single OS volume Type: Number - Default: 100 + Default: 20 ############################################################################### # Security Settings (these apply to all stacks) LoadBalancerScheme: @@ -159,21 +188,14 @@ Parameters: Default: '' ############################################################################### # Other Settings - ChefAutomateToken: - Description: The secret token used to communicate with the Chef Automate Server, for forwarding Visibility data. Leave blank to disable. + TemplateVersion: + Description: The version of this template to use (do not change this unless directed by support) Type: String - Default: 93a49a4f2482c64126f7b6015e6b0f30284287ee4054ff8807fb63d9cbd1c506 + Default: "5.0.2" Route53HostedZone: Type: String Default: '' - Description: Supply a Route 53 Hosted Zone name (eg. mydomain.com.) for auto-creating a DNS record. Must end in a dot. (Leave blank to disable) - AMIFlavor: - Description: Choose the AMI Flavor, the High-Perf CentOS image or the stock and paid official RHEL image - Type: String - Default: 'centos' - AllowedValues: - - 'centos' - - 'rhel' + Description: Supply a Route 53 Hosted Zone name (eg. mydomain.com) for auto-creating a DNS record. Do not append a dot at the end. (Leave blank to disable) LogsRetentionInDays: Description: Specifies the number of days you want to retain cloudwatch log events. Type: Number @@ -185,51 +207,54 @@ Metadata: config: ignore_checks: - W2507 - # AWS::CloudFormation::Interface: - # ParameterGroups: - # - Label: - # default: "Required Parameters" - # Parameters: - # - VPC - # - ChefServerSubnets - # - SSLCertificateARN - # - InboundAdminSecurityGroupId - # - KeyName - # - DBPassword - # - ContactEmail - # - ContactDept - # - Label: - # default: "Performance Settings" - # Parameters: - # - InstanceType - # - MaxFrontendInstances - # - MinFrontendInstances - # - LicenseCount - # - DBInstanceClass - # - DBAllocatedStorage - # - DBIops - # - ElasticSearchInstanceType - # - ElasticSearchVersion - # - ElasticSearchShardCount - # - ElasticSearchReplicaCount - # - Label: - # default: "Package Versions & Locations" - # Parameters: - # - ChefServerPackage - # - ChefManagePackage - # - PushJobsPackage - # - BeforeScriptLocation - # - AfterScriptLocation - # - Label: - # default: "Security Settings" - # Parameters: - # - LoadBalancerScheme - # - LoadBalancerSecurityGroupId - # - FrontendSecurityGroupId - # - ChefServerAssociatePublicIpAddress - # - DisableSignup - # - ChefServerIamRole - # - ChefSecretsBucket + AWS::CloudFormation::Interface: + ParameterGroups: + - Label: + default: "Required Parameters" + Parameters: + - VPC + - ServerSubnets + - KeyName + - InboundAdminSecurityGroupId + - ContactEmail + - ContactDept + - Route53HostedZone + - ChefAutomateAdminPassword + - AutomateSSLCertificateARN + - ChefSSLCertificateARN + - SupermarketSSLCertificateARN + - ChefDBPassword + - Label: + default: "Chef Automate 2 Settings" + Parameters: + - ChefAutomateToken + - AutomateDnsRecordName + - AutomateInstanceType + - AutomateDataVolumeSize + - Label: + default: "Chef Server Settings" + Parameters: + - ChefServerDnsRecordName + - ChefInstanceType + - ChefDBInstanceClass + - ChefDBStorageType + - ChefDBAllocatedStorage + - ChefDBIops + - ChefDBBackupDaystoKeep + - ChefElasticSearchInstanceType + - ChefElasticSearchVersion + - ChefElasticSearchShardCount + - Label: + default: "Supermarket Settings" + Parameters: + - SupermarketDnsRecordName + - SupermarketInstanceType + - SupermarketDataVolumeSize + - Label: + default: "Security Settings" + Parameters: + - LoadBalancerScheme + - AssociatePublicIpAddress Conditions: CreateChefIamRole: @@ -240,62 +265,16 @@ Conditions: !Equals [ !Ref LoadBalancerSecurityGroupId, '' ] CreateFrontendSecurityGroup: !Equals [ !Ref FrontendSecurityGroupId, '' ] - CreateRoute53Record: - !Not [!Equals [ !Ref Route53HostedZone, '' ] ] - RHELAmiFlavor: - !Equals [ !Ref AMIFlavor, 'rhel' ] -# Periodically regenerate this list using the 'generate_os_mappings.sh' script +# STOP: Do not replace the AMI list unless you 100% know what you're doing +# and are fully prepared to lose all support. This template now depends +# on AMIs that are pre-baked with all needed packages and scripts. Mappings: AMI: - eu-north-1: - rhel: ami-66f67f18 - centos: ami-00d82e36d4cf13239 - ap-south-1: - rhel: ami-0963937a03c01ecd4 - centos: ami-0d121b70d4f2a7c07 - eu-west-3: - rhel: ami-0c4224e392ec4e440 - centos: ami-0cb3a832f7fa08045 - eu-west-2: - rhel: ami-0188c0c5eddd2d032 - centos: ami-0dd3bbb85610696bb - eu-west-1: - rhel: ami-0202869bdd0fc8c75 - centos: ami-0355255a4ed05d5af - ap-northeast-2: - rhel: ami-041b16ca28f036753 - centos: ami-0c3e242e39ef33638 - ap-northeast-1: - rhel: ami-00b95502a4d51a07e - centos: ami-02c58f1040d612a45 - sa-east-1: - rhel: ami-05c1c16cac05a7c0b - centos: ami-0c7108774ddaedbaf - ca-central-1: - rhel: ami-06ca3c0058d0275b3 - centos: ami-0dfd14de667c4e8c9 - ap-southeast-1: - rhel: ami-055c55112e25b1f1f - centos: ami-0d4af852545da36f7 - ap-southeast-2: - rhel: ami-036b423b657376f5b - centos: ami-0fa6aa803dc7f6191 - eu-central-1: - rhel: ami-09de4a4c670389e4b - centos: ami-0ba5a0a4a8e5d70c1 us-east-1: - rhel: ami-000db10762d0c4c05 - centos: ami-062a84bcfcb87a1ab - us-east-2: - rhel: ami-094720ddca649952f - centos: ami-05e444eb3b5bb499f - us-west-1: - rhel: ami-04642fc8fca1e8e67 - centos: ami-0719ffe132cd3bc6c + centos: ami-08d306e6de29d5cc1 us-west-2: - rhel: ami-0a7e1ebfee7a4570e - centos: ami-05178c5d62c39cb82 + centos: ami-06fbb1e0ba7c99cfd Resources: ChefRole: @@ -451,18 +430,15 @@ Resources: AutomateStack: Type: AWS::CloudFormation::Stack Properties: - TemplateURL: !Sub https://s3.amazonaws.com/${AutomationBucket}/automate.yaml + TemplateURL: !Sub https://s3.amazonaws.com/${AutomationBucket}/${TemplateVersion}/automate.yaml Parameters: - AutomationBucket: !Ref AutomationBucket VPC: !Ref VPC LoadBalancerSubnets: !Join [ ",", !Ref ServerSubnets ] AutomateSubnet: !Select [ "0", !Ref ServerSubnets ] SSLCertificateARN: !Ref AutomateSSLCertificateARN + AutomateAdminPassword: !Ref ChefAutomateAdminPassword InboundAdminSecurityGroupId: !Ref InboundAdminSecurityGroupId - ImageId: !If - - RHELAmiFlavor - - !FindInMap [AMI, !Ref "AWS::Region", rhel] - - !FindInMap [AMI, !Ref "AWS::Region", centos] + ImageId: !FindInMap [AMI, !Ref "AWS::Region", centos] KeyName: !Ref KeyName ContactEmail: !Ref ContactEmail ContactDept: !Ref ContactDept @@ -474,9 +450,8 @@ Resources: LoadBalancerSecurityGroupId: !If [CreateLoadBalancerSecurityGroup, !Ref LoadBalancerSecurityGroup, !Ref LoadBalancerSecurityGroupId] FrontendSecurityGroupId: !If [CreateFrontendSecurityGroup, !Ref FrontendSecurityGroup, !Ref FrontendSecurityGroupId] AutomateIamRole: !If [CreateChefIamRole, !Ref ChefRole, !Ref ExistingIamRole] - Route53HostedZone: !If [CreateRoute53Record, !Ref Route53HostedZone, ''] + Route53HostedZone: !Ref Route53HostedZone Route53RecordName: !Ref AutomateDnsRecordName - AMIFlavor: !Ref AMIFlavor SystemLogsGroup: !Ref SystemLogs AppLogsGroup: !Ref AutomateAppLogs @@ -485,17 +460,14 @@ Resources: ChefServerStack: Type: AWS::CloudFormation::Stack Properties: - TemplateURL: !Sub https://s3.amazonaws.com/${AutomationBucket}/chef_server_ha.yaml + TemplateURL: !Sub https://s3.amazonaws.com/${AutomationBucket}/${TemplateVersion}/chef_server_ha.yaml Parameters: AutomationBucket: !Ref AutomationBucket VPC: !Ref VPC ChefServerSubnets: !Join [ ",", !Ref ServerSubnets ] SSLCertificateARN: !Ref ChefSSLCertificateARN InboundAdminSecurityGroupId: !Ref InboundAdminSecurityGroupId - ImageId: !If - - RHELAmiFlavor - - !FindInMap [AMI, !Ref "AWS::Region", rhel] - - !FindInMap [AMI, !Ref "AWS::Region", centos] + ImageId: !FindInMap [AMI, !Ref "AWS::Region", centos] KeyName: !Ref KeyName DBPassword: !Ref ChefDBPassword ContactEmail: !Ref ContactEmail @@ -503,12 +475,13 @@ Resources: AlertNotificationTopic: !Ref AlertNotificationTopic InstanceType: !Ref ChefInstanceType DBInstanceClass: !Ref ChefDBInstanceClass + DBStorageType: !Ref ChefDBStorageType DBAllocatedStorage: !Ref ChefDBAllocatedStorage DBIops: !Ref ChefDBIops + DBBackupDaystoKeep: !Ref ChefDBBackupDaystoKeep ElasticSearchInstanceType: !Ref ChefElasticSearchInstanceType ElasticSearchVersion: !Ref ChefElasticSearchVersion ElasticSearchShardCount: !Ref ChefElasticSearchShardCount - ElasticSearchReplicaCount: !Ref ChefElasticSearchReplicaCount LoadBalancerScheme: !Ref LoadBalancerScheme LoadBalancerSecurityGroupId: !If [CreateLoadBalancerSecurityGroup, !Ref LoadBalancerSecurityGroup, !Ref LoadBalancerSecurityGroupId] FrontendSecurityGroupId: !If [CreateFrontendSecurityGroup, !Ref FrontendSecurityGroup, !Ref FrontendSecurityGroupId] @@ -517,29 +490,26 @@ Resources: ChefSecretsBucket: !If [CreateChefSecretsBucket, !Ref ChefBucket, !Ref ExistingSecretsBucket] ChefAutomateServerUrl: !Sub "https://${AutomateStack.Outputs.DNSName}" ChefAutomateToken: !Ref ChefAutomateToken - Route53HostedZone: !If [CreateRoute53Record, !Ref Route53HostedZone, ''] + SupermarketUrl: !Sub "https://${SupermarketDnsRecordName}.${Route53HostedZone}" + Route53HostedZone: !Ref Route53HostedZone Route53RecordName: !Ref ChefServerDnsRecordName - AMIFlavor: !Ref AMIFlavor SystemLogsGroup: !Ref SystemLogs AppLogsGroup: !Ref ChefAppLogs + TemplateVersion: !Ref TemplateVersion # Supermarket Stack ######################################################################################### SupermarketStack: Type: AWS::CloudFormation::Stack Properties: - TemplateURL: !Sub https://s3.amazonaws.com/${AutomationBucket}/supermarket.yaml + TemplateURL: !Sub https://s3.amazonaws.com/${AutomationBucket}/${TemplateVersion}/supermarket.yaml Parameters: - AutomationBucket: !Ref AutomationBucket VPC: !Ref VPC LoadBalancerSubnets: !Join [ ",", !Ref ServerSubnets ] SupermarketSubnet: !Select [ "0", !Ref ServerSubnets ] SSLCertificateARN: !Ref SupermarketSSLCertificateARN InboundAdminSecurityGroupId: !Ref InboundAdminSecurityGroupId - ImageId: !If - - RHELAmiFlavor - - !FindInMap [AMI, !Ref "AWS::Region", rhel] - - !FindInMap [AMI, !Ref "AWS::Region", centos] + ImageId: !FindInMap [AMI, !Ref "AWS::Region", centos] KeyName: !Ref KeyName ContactEmail: !Ref ContactEmail ContactDept: !Ref ContactDept @@ -553,9 +523,8 @@ Resources: SupermarketIamRole: !If [CreateChefIamRole, !Ref ChefRole, !Ref ExistingIamRole] ChefSecretsBucket: !If [CreateChefSecretsBucket, !Ref ChefBucket, !Ref ExistingSecretsBucket] ChefStackName: !GetAtt ChefServerStack.Outputs.StackName - Route53HostedZone: !If [CreateRoute53Record, !Ref Route53HostedZone, ''] + Route53HostedZone: !Ref Route53HostedZone Route53RecordName: !Ref SupermarketDnsRecordName - AMIFlavor: !Ref AMIFlavor SystemLogsGroup: !Ref SystemLogs AppLogsGroup: !Ref SupermarketAppLogs @@ -575,9 +544,3 @@ Outputs: Value: !Sub "https://${SupermarketStack.Outputs.DNSName}" Export: Name: !Sub "${AWS::StackName}-SupermarketURL" - AutomateAdminPassword: - Description: The password for the Chef Automate admin user - Value: !If - - RHELAmiFlavor - - !Sub "To retrieve, run: ssh ec2-user@${AutomateStack.Outputs.ServerIP} sudo cat /root/automate-credentials.toml" - - !Sub "To retrieve, run: ssh centos@${AutomateStack.Outputs.ServerIP} sudo cat /root/automate-credentials.toml" diff --git a/marketplace.yaml b/marketplace.yaml new file mode 100644 index 0000000..a584edf --- /dev/null +++ b/marketplace.yaml @@ -0,0 +1,543 @@ +AWSTemplateFormatVersion: '2010-09-09' +Description: AWS Native Chef Stack for Marketplace v5.0.2 + +Parameters: + # Required Parameters + AutomationBucket: + Default: 'aws-native-chef-server' + Type: String + Description: AWS S3 bucket name that contains all of the cloudformation templates and scripts for this stack + VPC: + Description: Choose VPC to use + Type: AWS::EC2::VPC::Id + ServerSubnets: + Description: Provide a list of Subnet IDs for the Chef Servers (must be within the specified VPC) + Type: List + KeyName: + Description: Name of an existing EC2 KeyPair to enable SSH access to the instance + Type: AWS::EC2::KeyPair::KeyName + InboundClientCidr: + Description: Your internal network range in CIDR notation, from where client traffic will originate + AllowedPattern: '((\d{1,3})\.){3}\d{1,3}/\d{1,2}' + ConstraintDescription: Must be in CIDR notation (e.g. 10.0.0.0/8 or 0.0.0.0/0) + Type: String + InboundAdminCidr: + Description: Your internal administrative network source range in CIDR notation + AllowedPattern: '((\d{1,3})\.){3}\d{1,3}/\d{1,2}' + ConstraintDescription: Must be in CIDR notation (e.g. 10.0.0.0/8 or 0.0.0.0/0) + Type: String + ContactEmail: + Description: Contact email for Cloudwatch notifications and instance tagging + Type: String + ContactDept: + Description: Contact department for resource tagging purposes + Type: String + Route53HostedZone: + Type: String + Default: '' + Description: Supply a Route 53 Hosted Zone name (eg. mydomain.com) for auto-creating a DNS record. Do not append a dot at the end. (Leave blank to disable) + ############################################################################### + # Automate Stack Settings + AutomateSSLCertificateARN: + Description: ARN for SSL Certficate, pre-create this in AWS Certificate Manager + Type: String + AllowedPattern: "arn:aws:acm:.*" + ChefAutomateAdminPassword: + Description: Provide a password for the Chef Automate admin user + NoEcho: true + Type: String + MinLength: 8 + MaxLength: 64 + ConstraintDescription: Must be at least 8 characters + ChefAutomateToken: + Description: The secret token used to communicate with the Chef Automate Server, for forwarding Visibility data. + Type: String + AllowedPattern: ^[a-z0-9]*$ + MinLength: 64 + MaxLength: 64 + ConstraintDescription: Must be 64 alphanumeric characters + AutomateDnsRecordName: + Description: The DNS A-record name to automatically create in the Route53 zone (if enabled). Do not include the domain name. + Type: String + Default: 'automate' + AutomateInstanceType: + Description: EC2 Instance type for Automate server (M5 class recommended) + Default: t3.medium + Type: String + AllowedValues: [t3.medium, t3.large, t3.xlarge, t3.2xlarge, + m5.large, m5.xlarge, m5.2xlarge, m5.4xlarge, m5.12xlarge, m5.24xlarge, + c5.large, c5.xlarge, c5.2xlarge, c5.4xlarge, c5.9xlarge, c5.18xlarge] + AutomateDataVolumeSize: + Description: Amount of storage space in GB to allocate on a dedicated data volume (must be between 500 and 16000) + Type: Number + Default: 500 + MinValue: 500 + MaxValue: 16000 + ############################################################################### + # Chef Server Stack Settings + ChefSSLCertificateARN: + Description: ARN for SSL Certficate, pre-create this in AWS Certificate Manager + Type: String + AllowedPattern: "arn:aws:acm:.*" + ChefServerDnsRecordName: + Description: The DNS A-record name to automatically create in the Route53 zone (if enabled). Do not include the domain name. + Type: String + Default: 'chef' + ChefInstanceType: + Description: EC2 Instance type for Chef Server Frontends (C5 recommended for performance) + Default: t3.small + Type: String + AllowedValues: [t3.small, t3.medium, t3.large, t3.xlarge, t3.2xlarge, + m5.large, m5.xlarge, m5.2xlarge, m5.4xlarge, m5.12xlarge, m5.24xlarge, + c5.large, c5.xlarge, c5.2xlarge, c5.4xlarge, c5.9xlarge, c5.18xlarge] + ChefDBInstanceClass: + Description: EC2 Instance type for RDS DBs (M5 class recommended for performance) + Default: 'db.t3.small' + Type: String + AllowedValues: [db.t3.small, db.t3.medium, db.t3.large, db.t3.xlarge, db.t3.2xlarge, + db.m4.large, db.m4.xlarge, db.m4.2xlarge, db.m4.4xlarge, db.m4.10xlarge, db.m4.16xlarge, + db.r4.large, db.r4.xlarge, db.r4.2xlarge, db.r4.4xlarge, db.r4.8xlarge, db.r4.16xlarge, + db.m5.large, db.m5.xlarge, db.m5.2xlarge, db.m5.4xlarge, db.m5.12xlarge, db.m5.24xlarge] + ChefDBStorageType: + Description: Select gp2 or io1 DB storage. io1 is much higher performing, must select minimum 100GB storage and an IOPS value + Default: 'gp2' + Type: String + AllowedValues: [gp2, io1] + ChefDBAllocatedStorage: + Description: Storage size allocated for the database (gp2 minimum 20, maximum 16000) (io1 minimum 100, maximum 32768) + Default: '20' + Type: String + ChefDBIops: + Description: IOPS reserved for DB storage, only applicable if ChefDBStorageType is io1 (ChefDBAllocatedStorage * 10) + Default: '1000' + Type: String + ChefDBBackupDaystoKeep: + Description: Number of days of automatic DB backups to keep + Default: 5 + Type: Number + MinValue: 0 + MaxValue: 35 + ChefDBPassword: + Description: Create a strong RDS admin password + NoEcho: true + Type: String + MinLength: 8 + MaxLength: 64 + ChefElasticSearchInstanceType: + Description: The Instance type to use for ElasticSearch instances (i3 recommended for performance) + Type: String + Default: 'm3.medium.elasticsearch' + AllowedValues: [ + 'i3.large.elasticsearch', 'i3.xlarge.elasticsearch', 'i3.2xlarge.elasticsearch', 'i3.4xlarge.elasticsearch', 'i3.8xlarge.elasticsearch', 'i3.16xlarge.elasticsearch', + 'i2.xlarge.elasticsearch', 'i2.2xlarge.elasticsearch', + 'm3.medium.elasticsearch', 'm3.large.elasticsearch', 'm3.xlarge.elasticsearch', 'm3.medium.elasticsearch', + 'r3.large.elasticsearch', 'r3.xlarge.elasticsearch', 'r3.2xlarge.elasticsearch', 'r3.4xlarge.elasticsearch', 'r3.8xlarge.elasticsearch' ] + ChefElasticSearchVersion: + Description: Version of ElasticSearch to use. (5.6 recommended for new installs) + Type: String + Default: '5.6' + AllowedValues: + - '5.6' + ChefElasticSearchShardCount: + Description: Number of ElasticSearch hosts to provision at launch (3 recommended, 2 provides HA) + Default: 2 + Type: Number + MinValue: 2 + MaxValue: 3 + ############################################################################### + # Automate Stack Settings + SupermarketSSLCertificateARN: + Description: ARN for SSL Certficate, pre-create this in AWS Certificate Manager + Type: String + AllowedPattern: "arn:aws:acm:.*" + SupermarketDnsRecordName: + Description: The DNS A-record name to automatically create in the Route53 zone (if enabled). Do not include the domain name. + Type: String + Default: 'supermarket' + SupermarketInstanceType: + Description: EC2 Instance type for Supermarket server (M5 class recommended) + Default: t3.small + Type: String + AllowedValues: [t3.small, t3.medium, t3.large, t3.xlarge, t3.2xlarge, + m5.large, m5.xlarge, m5.2xlarge, m5.4xlarge, m5.12xlarge, m5.24xlarge, + c5.large, c5.xlarge, c5.2xlarge, c5.4xlarge, c5.9xlarge, c5.18xlarge] + SupermarketDataVolumeSize: + Description: Amount of storage space in GB to allocate for the Supermarket OS+data volume + Type: Number + Default: 20 +############################################################################### +# Security Settings (these apply to all stacks) + LoadBalancerScheme: + Description: Network Scheme for the ELB + Type: String + Default: internet-facing + AllowedValues: + - 'internet-facing' + - 'internal' + AssociatePublicIpAddress: + Description: Assign public IP addresses to the Chef Servers or not + Type: String + Default: true + AllowedValues: + - true + - false + ############################################################################### + # Other Settings + TemplateVersion: + Description: The version of this template to use (do not change this unless directed by support) + Type: String + Default: "5.0.2" + LogsRetentionInDays: + Description: Specifies the number of days you want to retain cloudwatch log events. + Type: Number + Default: 90 + AllowedValues: [1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 3653] + +Conditions: + 2ZoneES: + !Equals [!Ref ChefElasticSearchShardCount, 2] + +Metadata: + cfn-lint: + config: + ignore_checks: + - W2507 + AWS::CloudFormation::Interface: + ParameterGroups: + - Label: + default: "Required Parameters" + Parameters: + - VPC + - ServerSubnets + - KeyName + - InboundClientCidr + - InboundAdminCidr + - ContactEmail + - ContactDept + - Route53HostedZone + - ChefAutomateAdminPassword + - AutomateSSLCertificateARN + - ChefSSLCertificateARN + - SupermarketSSLCertificateARN + - ChefDBPassword + - Label: + default: "Chef Automate 2 Settings" + Parameters: + - ChefAutomateToken + - AutomateDnsRecordName + - AutomateInstanceType + - AutomateDataVolumeSize + - Label: + default: "Chef Server Settings" + Parameters: + - ChefServerDnsRecordName + - ChefInstanceType + - ChefDBInstanceClass + - ChefDBStorageType + - ChefDBAllocatedStorage + - ChefDBIops + - ChefDBBackupDaystoKeep + - ChefElasticSearchInstanceType + - ChefElasticSearchVersion + - ChefElasticSearchShardCount + - Label: + default: "Supermarket Settings" + Parameters: + - SupermarketDnsRecordName + - SupermarketInstanceType + - SupermarketDataVolumeSize + - Label: + default: "Security Settings" + Parameters: + - LoadBalancerScheme + - AssociatePublicIpAddress + +# STOP: Do not replace the AMI list unless you 100% know what you're doing +# and are fully prepared to lose all support. This template now depends +# on AMIs that are pre-baked with all needed packages and scripts. +Mappings: + AMI: + us-east-1: + centos: ami-08d306e6de29d5cc1 + us-west-2: + centos: ami-06fbb1e0ba7c99cfd + +Resources: + ChefRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Statement: + - Action: + - sts:AssumeRole + Effect: Allow + Principal: + Service: + - ec2.amazonaws.com + Version: '2012-10-17' + Path: / + + RolePolicies: + Type: AWS::IAM::Policy + Properties: + PolicyName: !Sub ${AWS::StackName}-ChefStack-Policy + PolicyDocument: + Version: '2012-10-17' + Statement: + # Allow all actions to one bucket (the supplied one, or the one you provided) + - Action: s3:* + Effect: Allow + Resource: + - !Join ['', [ 'arn:aws:s3:::', !Ref ChefBucket]] + - !Join ['', [ 'arn:aws:s3:::', !Ref ChefBucket, '/*' ]] + # Allow ability to list all buckets + - Action: s3:List* + Effect: Allow + Resource: arn:aws:s3:::* + # Allow instances to set themselves as unhealthy if one of the scripts fail + - Action: autoscaling:* + Effect: Allow + Resource: "*" + # Allow instance to create a nightly snapshot + - Action: ["ec2:CreateSnapshot", "ec2:CreateTags", "ec2:DeleteSnapshot", "ec2:DescribeSnapshots", "ec2:DescribeVolumes"] + Effect: Allow + Resource: "*" + # Allow instances to read their own tags (needed for setup script below) + - Action: ec2:DescribeTags + Effect: Allow + Resource: "*" + - Action: cloudwatch:PutMetricData + Effect: Allow + Resource: "*" + - Action: cloudwatch:GetMetricStatistics + Effect: Allow + Resource: "*" + - Action: cloudwatch:ListMetrics + Effect: Allow + Resource: "*" + # Allow instances to write to cloudwatch logs + - Action: ["logs:PutLogEvents", "logs:CreateLogStream", "logs:CreateLogGroup"] + Effect: Allow + Resource: !Sub arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:* + Roles: + - !Ref ChefRole + + ChefBucket: + Type: AWS::S3::Bucket + DeletionPolicy: Retain + Properties: + AccessControl: Private + VersioningConfiguration: + Status: Enabled + + InboundAdminSecurityGroupId: + Type: AWS::EC2::SecurityGroup + Properties: + GroupDescription: Trusted IP addresses, ranges or security groups + SecurityGroupIngress: + - CidrIp: !Ref InboundAdminCidr + IpProtocol: tcp + FromPort: 22 + ToPort: 22 + - CidrIp: !Ref InboundAdminCidr + IpProtocol: tcp + FromPort: 443 + ToPort: 443 + Tags: + - Key: Name + Value: !Sub ${AWS::StackName}-Trusted-SG + VpcId: !Ref VPC + + FrontendSecurityGroup: + Type: AWS::EC2::SecurityGroup + Properties: + GroupDescription: Setup Ingress/Egress for Automate Server + SecurityGroupEgress: + - CidrIp: 0.0.0.0/0 + FromPort: 0 + IpProtocol: tcp + ToPort: 65535 + SecurityGroupIngress: + - FromPort: 80 + IpProtocol: tcp + SourceSecurityGroupId: !Ref LoadBalancerSecurityGroup + ToPort: 80 + - FromPort: 443 + IpProtocol: tcp + SourceSecurityGroupId: !Ref LoadBalancerSecurityGroup + ToPort: 443 + Tags: + - Key: Name + Value: !Sub ${AWS::StackName}-FE-SG + VpcId: !Ref VPC + + LoadBalancerSecurityGroup: + Type: AWS::EC2::SecurityGroup + Properties: + GroupDescription: Setup Ingress/Egress for Automate Load Balancer + SecurityGroupEgress: + - CidrIp: 0.0.0.0/0 + FromPort: 0 + IpProtocol: tcp + ToPort: 65535 + SecurityGroupIngress: + - CidrIp: !Ref InboundClientCidr + FromPort: 80 + IpProtocol: tcp + ToPort: 80 + - CidrIp: !Ref InboundClientCidr + FromPort: 443 + IpProtocol: tcp + ToPort: 443 + Tags: + - Key: Name + Value: !Sub ${AWS::StackName}-ELB-SG + VpcId: !Ref VPC + + AlertNotificationTopic: + Type: AWS::SNS::Topic + Properties: + Subscription: + - Endpoint: + !Ref ContactEmail + Protocol: email + +# Logging Groups (Cloudwatch Logs) +######################################################################################### + SystemLogs: + Type: AWS::Logs::LogGroup + Properties: + RetentionInDays: !Ref LogsRetentionInDays + ChefAppLogs: + Type: AWS::Logs::LogGroup + Properties: + RetentionInDays: !Ref LogsRetentionInDays + SupermarketAppLogs: + Type: AWS::Logs::LogGroup + Properties: + RetentionInDays: !Ref LogsRetentionInDays + AutomateAppLogs: + Type: AWS::Logs::LogGroup + Properties: + RetentionInDays: !Ref LogsRetentionInDays + +# Automate Stack +######################################################################################### + AutomateStack: + Type: AWS::CloudFormation::Stack + Properties: + TemplateURL: !Sub https://s3.amazonaws.com/${AutomationBucket}/${TemplateVersion}/automate.yaml + Parameters: + VPC: !Ref VPC + LoadBalancerSubnets: !Join [ ",", !Ref ServerSubnets ] + AutomateSubnet: !Select [ "0", !Ref ServerSubnets ] + SSLCertificateARN: !Ref AutomateSSLCertificateARN + AutomateAdminPassword: !Ref ChefAutomateAdminPassword + InboundAdminSecurityGroupId: !Ref InboundAdminSecurityGroupId + ImageId: !FindInMap [AMI, !Ref "AWS::Region", centos] + KeyName: !Ref KeyName + ContactEmail: !Ref ContactEmail + ContactDept: !Ref ContactDept + AlertNotificationTopic: !Ref AlertNotificationTopic + InstanceType: !Ref AutomateInstanceType + DataVolumeSize: !Ref AutomateDataVolumeSize + AutomateDataCollectorToken: !Ref ChefAutomateToken + LoadBalancerScheme: !Ref LoadBalancerScheme + LoadBalancerSecurityGroupId: !Ref LoadBalancerSecurityGroup + FrontendSecurityGroupId: !Ref FrontendSecurityGroup + AutomateIamRole: !Ref ChefRole + Route53HostedZone: !Ref Route53HostedZone + Route53RecordName: !Ref AutomateDnsRecordName + SystemLogsGroup: !Ref SystemLogs + AppLogsGroup: !Ref AutomateAppLogs + +# Chef Server Stack +######################################################################################### + ChefServerStack: + Type: AWS::CloudFormation::Stack + Properties: + TemplateURL: !Sub https://s3.amazonaws.com/${AutomationBucket}/${TemplateVersion}/chef_server_ha.yaml + Parameters: + AutomationBucket: !Ref AutomationBucket + VPC: !Ref VPC + ChefServerSubnets: !Join [ ",", !Ref ServerSubnets ] + SSLCertificateARN: !Ref ChefSSLCertificateARN + InboundAdminSecurityGroupId: !Ref InboundAdminSecurityGroupId + ImageId: !FindInMap [AMI, !Ref "AWS::Region", centos] + KeyName: !Ref KeyName + DBPassword: !Ref ChefDBPassword + ContactEmail: !Ref ContactEmail + ContactDept: !Ref ContactDept + AlertNotificationTopic: !Ref AlertNotificationTopic + InstanceType: !Ref ChefInstanceType + DBInstanceClass: !Ref ChefDBInstanceClass + DBStorageType: !Ref ChefDBStorageType + DBAllocatedStorage: !Ref ChefDBAllocatedStorage + DBIops: !Ref ChefDBIops + DBBackupDaystoKeep: !Ref ChefDBBackupDaystoKeep + ElasticSearchInstanceType: !Ref ChefElasticSearchInstanceType + ElasticSearchVersion: !Ref ChefElasticSearchVersion + ElasticSearchShardCount: !Ref ChefElasticSearchShardCount + ElasticSearchReplicaCount: !If [2ZoneES, 1, 2] + LoadBalancerScheme: !Ref LoadBalancerScheme + LoadBalancerSecurityGroupId: !Ref LoadBalancerSecurityGroup + FrontendSecurityGroupId: !Ref FrontendSecurityGroup + ChefServerAssociatePublicIpAddress: !Ref AssociatePublicIpAddress + ChefServerIamRole: !Ref ChefRole + ChefSecretsBucket: !Ref ChefBucket + ChefAutomateServerUrl: !Sub "https://${AutomateStack.Outputs.DNSName}" + ChefAutomateToken: !Ref ChefAutomateToken + SupermarketUrl: !Sub "https://${SupermarketDnsRecordName}.${Route53HostedZone}" + Route53HostedZone: !Ref Route53HostedZone + Route53RecordName: !Ref ChefServerDnsRecordName + SystemLogsGroup: !Ref SystemLogs + AppLogsGroup: !Ref ChefAppLogs + TemplateVersion: !Ref TemplateVersion + +# Supermarket Stack +######################################################################################### + SupermarketStack: + Type: AWS::CloudFormation::Stack + Properties: + TemplateURL: !Sub https://s3.amazonaws.com/${AutomationBucket}/${TemplateVersion}/supermarket.yaml + Parameters: + VPC: !Ref VPC + LoadBalancerSubnets: !Join [ ",", !Ref ServerSubnets ] + SupermarketSubnet: !Select [ "0", !Ref ServerSubnets ] + SSLCertificateARN: !Ref SupermarketSSLCertificateARN + InboundAdminSecurityGroupId: !Ref InboundAdminSecurityGroupId + ImageId: !FindInMap [AMI, !Ref "AWS::Region", centos] + KeyName: !Ref KeyName + ContactEmail: !Ref ContactEmail + ContactDept: !Ref ContactDept + AlertNotificationTopic: !Ref AlertNotificationTopic + ChefUrl: !Sub "https://${ChefServerStack.Outputs.DNSName}" + InstanceType: !Ref SupermarketInstanceType + DataVolumeSize: !Ref SupermarketDataVolumeSize + LoadBalancerScheme: !Ref LoadBalancerScheme + LoadBalancerSecurityGroupId: !Ref LoadBalancerSecurityGroup + FrontendSecurityGroupId: !Ref FrontendSecurityGroup + SupermarketIamRole: !Ref ChefRole + ChefSecretsBucket: !Ref ChefBucket + ChefStackName: !GetAtt ChefServerStack.Outputs.StackName + Route53HostedZone: !Ref Route53HostedZone + Route53RecordName: !Ref SupermarketDnsRecordName + SystemLogsGroup: !Ref SystemLogs + AppLogsGroup: !Ref SupermarketAppLogs + +Outputs: + AutomateUrl: + Description: The URL of the Chef Automate Server + Value: !Sub "https://${AutomateStack.Outputs.DNSName}" + Export: + Name: !Sub "${AWS::StackName}-AutomateURL" + ChefServerUrl: + Description: The URL of the Chef Automate Server + Value: !Sub "https://${ChefServerStack.Outputs.DNSName}" + Export: + Name: !Sub "${AWS::StackName}-ChefServerURL" + SupermarketUrl: + Description: The URL of the Chef Automate Server + Value: !Sub "https://${SupermarketStack.Outputs.DNSName}" + Export: + Name: !Sub "${AWS::StackName}-SupermarketURL" diff --git a/stack_parameters_main.json.example b/stack_parameters_main.json.example new file mode 100644 index 0000000..89c525f --- /dev/null +++ b/stack_parameters_main.json.example @@ -0,0 +1,50 @@ +[ + { + "ParameterKey": "VPC", + "ParameterValue": "vpc-41d45124" + }, + { + "ParameterKey": "ServerSubnets", + "ParameterValue": "subnet-7424b611,subnet-7fd06308,subnet-4dc71114" + }, + { + "ParameterKey": "InboundAdminSecurityGroupId", + "ParameterValue": "sg-c9beb2ac" + }, + { + "ParameterKey": "KeyName", + "ParameterValue": "id_rsa" + }, + { + "ParameterKey": "ContactEmail", + "ParameterValue": "me@mycompany.com" + }, + { + "ParameterKey": "ContactDept", + "ParameterValue": "Cool Automation Team" + }, + { + "ParameterKey": "ChefAutomateAdminPassword", + "ParameterValue": "SuperSecurePassword" + }, + { + "ParameterKey": "ChefSSLCertificateARN", + "ParameterValue": "arn:aws:acm:us-west-2:446539779517:certificate/60f573b3-f8ed-48d9-a6d1-e89f79da2e8f" + }, + { + "ParameterKey": "AutomateSSLCertificateARN", + "ParameterValue": "arn:aws:acm:us-west-2:446539779517:certificate/60f573b3-f8ed-48d9-a6d1-e89f79da2e8f" + }, + { + "ParameterKey": "SupermarketSSLCertificateARN", + "ParameterValue": "arn:aws:acm:us-west-2:446539779517:certificate/60f573b3-f8ed-48d9-a6d1-e89f79da2e8f" + }, + { + "ParameterKey": "ChefDBPassword", + "ParameterValue": "SuperSecurePassword" + }, + { + "ParameterKey": "Route53HostedZone", + "ParameterValue": "chef.mycompany.com" + } +] diff --git a/stack_parameters_marketplace.json.example b/stack_parameters_marketplace.json.example new file mode 100644 index 0000000..d24edd8 --- /dev/null +++ b/stack_parameters_marketplace.json.example @@ -0,0 +1,54 @@ +[ + { + "ParameterKey": "VPC", + "ParameterValue": "vpc-41d45124" + }, + { + "ParameterKey": "ServerSubnets", + "ParameterValue": "subnet-7424b611,subnet-7fd06308,subnet-4dc71114" + }, + { + "ParameterKey": "InboundClientCidr", + "ParameterValue": "0.0.0.0/0" + }, + { + "ParameterKey": "InboundAdminCidr", + "ParameterValue": "0.0.0.0/0" + }, + { + "ParameterKey": "KeyName", + "ParameterValue": "id_rsa" + }, + { + "ParameterKey": "ContactEmail", + "ParameterValue": "me@mycompany.com" + }, + { + "ParameterKey": "ContactDept", + "ParameterValue": "Cool Automation Team" + }, + { + "ParameterKey": "ChefAutomateAdminPassword", + "ParameterValue": "SuperSecurePassword" + }, + { + "ParameterKey": "ChefSSLCertificateARN", + "ParameterValue": "arn:aws:acm:us-west-2:446539779517:certificate/60f573b3-f8ed-48d9-a6d1-e89f79da2e8f" + }, + { + "ParameterKey": "AutomateSSLCertificateARN", + "ParameterValue": "arn:aws:acm:us-west-2:446539779517:certificate/60f573b3-f8ed-48d9-a6d1-e89f79da2e8f" + }, + { + "ParameterKey": "SupermarketSSLCertificateARN", + "ParameterValue": "arn:aws:acm:us-west-2:446539779517:certificate/60f573b3-f8ed-48d9-a6d1-e89f79da2e8f" + }, + { + "ParameterKey": "ChefDBPassword", + "ParameterValue": "SuperSecurePassword" + }, + { + "ParameterKey": "Route53HostedZone", + "ParameterValue": "chef.mycompany.com" + } +] diff --git a/supermarket.yaml b/supermarket.yaml index 84da691..e465730 100644 --- a/supermarket.yaml +++ b/supermarket.yaml @@ -1,12 +1,8 @@ AWSTemplateFormatVersion: '2010-09-09' -Description: AWS Native Supermarket (auto-recovering standalone) v4.1.0 +Description: AWS Native Supermarket (auto-recovering standalone) v5.0.2 Parameters: # Required Parameters - AutomationBucket: - Default: 'aws-native-chef-server' - Type: String - Description: AWS S3 bucket name that contains all of the cloudformation templates and scripts for this stack VPC: Description: Choose VPC to use Type: AWS::EC2::VPC::Id @@ -62,12 +58,6 @@ Parameters: Type: Number Default: 100 ############################################################################### - # Package Versions & Locations - SupermarketPackage: - Description: The URL to the chef server EL7 (chef-server-core) package which will be downloaded - Type: String - Default: 'https://packages.chef.io/files/stable/supermarket/3.1.63/el/7/supermarket-3.1.63-1.el7.x86_64.rpm' - ############################################################################### # Security Settings LoadBalancerScheme: Description: Network Scheme for the ELB @@ -96,18 +86,11 @@ Parameters: Route53HostedZone: Type: String Default: '' - Description: Supply a Route 53 Hosted Zone name (eg. mydomain.com.) for auto-creating a DNS record. Must end in a dot. (Leave blank to disable) + Description: Supply a Route 53 Hosted Zone name (eg. mydomain.com) for auto-creating a DNS record. Must NOT end in a dot. (Leave blank to disable) Route53RecordName: Type: String Default: 'supermarket' Description: Supply a DNS record name that will be prepended to the Route 53 Hosted Zone - AMIFlavor: - Description: Choose the AMI Flavor, the High-Perf CentOS image or the stock and paid official RHEL image - Type: String - Default: 'centos' - AllowedValues: - - 'centos' - - 'rhel' SystemLogsGroup: Description: The logical name of the Cloudwatch Logs log group for systemn logs Type: String @@ -124,8 +107,6 @@ Metadata: Conditions: CreateRoute53Record: !Not [!Equals [ !Ref Route53HostedZone, '' ] ] - RHELAmiFlavor: - !Equals [ !Ref AMIFlavor, 'rhel' ] Resources: ChefInstanceProfile: @@ -215,14 +196,6 @@ Resources: "Fn::Base64": !Sub | #!/bin/bash -ex trap '/opt/aws/bin/cfn-signal -e 1 --region ${AWS::Region} --stack ${AWS::StackName} --resource SupermarketServer' ERR - # Update the system first - yum upgrade -y - # Run before script, make sure to install cfn-init if not installed - # attempt to fetch the script first as a signed request (private bucket), then unsigned (public bucket) - aws s3 cp s3://${AutomationBucket}/files/before.sh /root/before.sh || \ - curl -s https://${AutomationBucket}.s3.amazonaws.com/files/before.sh -o /root/before.sh || \ - curl -s https://aws-native-chef-server.s3.amazonaws.com/files/before.sh -o /root/before.sh - bash -ex /root/before.sh # Execute AWS::CloudFormation::Init /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource SupermarketServer --region ${AWS::Region} /opt/aws/bin/cfn-signal -e 0 --region ${AWS::Region} --stack ${AWS::StackName} --resource SupermarketServer @@ -233,27 +206,12 @@ Resources: Metadata: AWS::CloudFormation::Init: configSets: - default: [awslogs, supermarket_setup, nightly_snapshots] + default: [awslogs, supermarket_setup, nightly_snapshots, final_stage] supermarket_setup: - packages: - rpm: - supermarket: !Sub ${SupermarketPackage} - yum: - perl: [] - perl-DateTime: [] - perl-Sys-Syslog: [] - perl-LWP-Protocol-https: [] - perl-Digest-SHA: [] - zip: [] - unzip: [] commands: 02_write_supermarket_json: command: !Sub - | - yum install -y jq || \ - rpm -ivh \ - https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/o/oniguruma-5.9.5-3.el7.x86_64.rpm \ - https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/j/jq-1.5-1.el7.x86_64.rpm aws s3 cp s3://${ChefSecretsBucket}/${ChefStackName}/etc_opscode/oc-id-applications/supermarket.json /tmp/supermarket.json OAUTH2_APP_ID=$(cat /tmp/supermarket.json | jq -r ".uid") OAUTH2_SECRET=$(cat /tmp/supermarket.json | jq -r ".secret") @@ -267,35 +225,18 @@ Resources: - { FQDN: !If [CreateRoute53Record, !Ref SupermarketDNSRecord, !GetAtt ChefALB.DNSName] } 03_configure_supermarket: command: | + rpm -ivh /var/cache/marketplace/supermarket.rpm supermarket-ctl reconfigure - 04_after_script: - command: !Sub | - # attempt to fetch the script first as a signed request (private bucket), then unsigned (public bucket) - aws s3 cp s3://${AutomationBucket}/files/after.sh /root/after.sh || \ - curl -s https://${AutomationBucket}.s3.amazonaws.com/files/after.sh -o /root/after.sh - bash -ex /root/after.sh 05_configure_cloudwatch_monitoring: command: | - mkdir /opt/cloudwatch_monitoring - cd /opt/cloudwatch_monitoring - curl http://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.1.zip -O - unzip CloudWatchMonitoringScripts-1.2.1.zip - rm -f CloudWatchMonitoringScripts-1.2.1.zip crontab -l | { cat; echo "*/5 * * * * /opt/cloudwatch_monitoring/aws-scripts-mon/mon-put-instance-data.pl --mem-util --disk-space-util --disk-path=/ --from-cron"; } | crontab - nightly_snapshots: - files: - /usr/local/bin/ebs-snapshot.sh: - source: https://raw.githubusercontent.com/CaseyLabs/aws-ec2-ebs-automatic-snapshot-bash/master/ebs-snapshot.sh - mode: '000755' commands: create_nightly_backup_cronjob: command: | crontab -l | { cat; echo "55 23 * * * root /usr/local/bin/ebs-snapshot.sh"; } | crontab - awslogs: files: - /root/awslogs-agent-setup.py: - source: 'https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py' - mode: '000755' /etc/awslogs/awscli.conf: content: !Sub | [default] @@ -374,7 +315,16 @@ Resources: 01_setup: command: !Sub | mkdir -p /var/lib/awslogs - python /root/awslogs-agent-setup.py --region ${AWS::Region} --non-interactive -c /etc/awslogs/awslogs.conf + python /var/cache/marketplace/awslogs-agent-setup.py --region ${AWS::Region} --non-interactive -c /etc/awslogs/awslogs.conf + final_stage: + commands: + 09_cleanup: + command: | + # clean up the chef package cache, to maximize available disk space after install + if [ -d /var/cache/marketplace ]; then + rm -f /var/cache/marketplace/*.rpm + rm -f /var/cache/marketplace/*.aib + fi # EC2 System AutoRecovery # https://aws.amazon.com/blogs/aws/new-auto-recovery-for-amazon-ec2/ @@ -423,9 +373,9 @@ Resources: Type: AWS::Route53::RecordSet Condition: CreateRoute53Record Properties: - HostedZoneName: !Ref Route53HostedZone + HostedZoneName: !Sub ${Route53HostedZone}. + Name: !Sub ${Route53RecordName}.${Route53HostedZone}. Comment: !Sub Created by Cloudformation ${AWS::StackName} - Name: !Sub ${Route53RecordName}.${Route53HostedZone} Type: CNAME TTL: '900' ResourceRecords: @@ -470,7 +420,7 @@ Resources: - Name: MountPath Value: '/' - Name: Filesystem - Value: !If [RHELAmiFlavor, "/dev/nvme0n1p2", "/dev/nvme0n1p1"] + Value: '/dev/nvme0n1p1' MemoryUtilization: Type: AWS::CloudWatch::Alarm