V5 and the road to a Marketplace-like experience (#50)

* Add a marketplace-style solution template.  Also a number of bug fixes and enhancements:
* Add template version-number support, so that multiple releases can be published to S3
* Fix an issue in automate.yaml where the PublicIp wasn't available in the Outputs, so it bombed
* Fix an issue where the dots appended to the domain name caused issues

Signed-off-by: Irving Popovetsky <[email protected]>

* Marketplace v5 - initial working version

Signed-off-by: Irving Popovetsky <[email protected]>

* move final_stage to its own configset, to resolve issues with purging Automate files too early

Signed-off-by: Irving Popovetsky <[email protected]>

* Cost savings part 1: enable gp2 storage for RDS, shrink default instance sizes in marketplace.yaml

Signed-off-by: Irving Popovetsky <[email protected]>

* Cost savings pt 2: Make the number of DB backups tunable and lower the default value

Signed-off-by: Irving Popovetsky <[email protected]>

* Elasticsearch cost reduction, allow 2 instead of 3

Signed-off-by: Irving Popovetsky <[email protected]>

* Update images

* Major cost reductions for the marketplace stack

Signed-off-by: Irving Popovetsky <[email protected]>

* ES6 is not supported with chef-server today

Signed-off-by: Irving Popovetsky <[email protected]>

* Improve browser and CLI launch experience

Signed-off-by: Irving Popovetsky <[email protected]>

* Fix one stray template version

Signed-off-by: Irving Popovetsky <[email protected]>

* fixes #51, fixes #52, fixes #53, bump version to 5.0.2

Signed-off-by: Irving Popovetsky <[email protected]>

* Increase Automate setup timeout to give smaller instances time to finish.  Update AMIs for TCP SACK vulnerability

Signed-off-by: Irving Popovetsky <[email protected]>

* Fix the redirect_uri for Supermarket

Signed-off-by: Irving Popovetsky <[email protected]>

* majorly revamp the Pre-reqs doc to be much clearer

Signed-off-by: Irving Popovetsky <[email protected]>

* New feature:  Provide an Automate admin password instead of the file reading shenanigans

Signed-off-by: Irving Popovetsky <[email protected]>

PREREQS.md

@@ -1,61 +1,34 @@
 ## Prerequisites
 
-### AMI Acceptance
+### Create an external Route53 zone that is resolvable
 
-Before you start, make sure the AMI that is listed in the `backendless_chef.yaml` for your region has been accepted/approved for use in the AWS Marketplace. We are using native Amazon Linux AMIs, the most up to date list is [always here](https://aws.amazon.com/amazon-linux-ami/).
+In order to use one of the "full stack" templates such as `main.yaml` or `marketplace.yaml`, you must have a domain that is hosted on Route53.  If you're not ready to take the plunge for your entire domain, you can create a subdomain and route all traffic to that (for example chef.mycompany.com).  AWS provides [instructions](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/CreatingNewSubdomain.html) and [a video](https://aws.amazon.com/premiumsupport/knowledge-center/create-subdomain-route-53/) on how to do that.
 
-### For SSL
-1. You must have a DNS Record Name and Zone Name planned for this cluster to match the SSL certificate.
-1. You must have already created and uploaded an SSL cert to AWS. Once you've uploaded the cert, you will need to follow these steps to update the `backendless_chef.yaml` template with the correct values:
-   * Update the `SSLCertificateARN` parameter with the `ARN` for the SSL Cert. _For info on how to get the ARN, please follow the [AWS Certificate Manger](https://aws.amazon.com/certificate-manager/) docs._
-   * **If you're using AWS Route 53**, fill in the `Route53RecordName` and `Route53HostedZone` parameters for the template
-   * **If you're not using AWS Route 53**, create the DNS record in your own nameservers to be a CNAME record that points at the stack's Application Load Balancer (e.g. my-chef-stack-1942464223.us-west-2.elb.amazonaws.com)
-
-### Using with Automate
-
-At this time, this template does not setup Automate as part of it's deployment. However, you can configure it to point to an Automate server that you will setup afterwards, or that is already setup.
-1. You should have a token already generated, follow the [instructions here](https://automate.chef.io/docs/data-collection/) on how to generate an Automate token.
-1. The Chef Automate URL, for more information on how this URL is formatted, [read this](https://automate.chef.io/docs/data-collection/).
+In the parameters, fill in the `Route53HostedZone` parameter to match your Route53 zone, as well as the `AutomateDnsRecordName`, `ChefServerDnsRecordName` and `SupermarketDnsRecordName` values.
 
-### Package Versions
 
-It's recommended to have the packages downloaded and hosted locally before proceeding, an S3 bucket, Artifactory/Nexus or YUM/APT repository cache works well for this purpose. Once you've downloaded the correct EL7 packages and have them hosted, adjust the following variables accordingly to point to the proper URLs.
-
-* `ChefServerPackage`, `ChefManagePackage`, `PushJobsPackage`
-
-_Here's an example of setting up an S3 cache:_
+### For SSL
 
-1. Install `mixlib-install` Ruby Gem from the [mixlib-install repo](https://github.com/chef/mixlib-install) on an EC2 instance that has access to the S3 bucket being used.
-1. Install the `aws` cli tool on the same instance from [aws cli](https://aws.amazon.com/cli/).
-1. Run the following commands:
-    ```
-    mixlib-install download chef-server --platform el --platform-version 7.5 --architecture x86_64
-    # Starting download https://packages.chef.io/files/stable/chef-server/12.17.33/el/7/chef-server-core-12.17.33-1.el7.x86_64.rpm
-    # Download saved to /Users/myname/chef-server-core-12.17.33-1.el7.x86_64.rpm
+You must create or upload at least one SSL certficate to AWS Certificate Manager (ACM).  If you wish, ACM can provide free SSL certificates for you and automatically manages renewals of those certificates.  You may create 3 separate certificates, or a single wildcard certificate (ex: `*.chef.mycompany.com`) that is used in all 3 places.  Since your Route53 zone is now working, use the `DNS Validation` option as it is far faster and more convenient.
 
-    aws s3 cp /Users/myname/chef-server-core-12.17.33-1.el7.x86_64.rpm s3://mybucket/package-cache/ --acl public-read
-    # upload: ./chef-server-core-12.17.33-1.el7.x86_64.rpm to s3://mybucket/package-cache/chef-server-core-12.17.33-1.el7.x86_64.rpm
+In the parameters, fill in the `ChefSSLCertificateARN`, `AutomateSSLCertificateARN` and `SupermarketSSLCertificateARN` values with the ARNs for the certificates. The ARNs are always viewable in the ACM console, an example ACM ARN looks like: `arn:aws:acm:us-west-2:446539779517:certificate/82d30a13-b420-4f43-80de-9e7872f70b96`
 
-    aws s3 presign s3://mybucket/package-cache/chef-server-core-12.17.33-1.el7.x86_64.rpm | cut -d '?' -f 1
-    # https://mybucket.s3.amazonaws.com/package-cache/chef-server-core-12.17.33-1.el7.x86_64.rpm
-    ```
-1. Set the last output (e.g. `https://mybucket.s3.amazonaws.com/package-cache/chef-server-core-12.17.33-1.el7.x86_64.rpm` as shown above) as your `ChefServerPackage` value in `backendless_chef.yaml`.
-    
 
 ## Network
 
 You must already have a VPC setup properly before continuing setting up the stack, it should;
 
 * Have enough IP's available to assign to nodes
 * Be split up into 3 subnets, each in different Availability Zones (AZ's)
+* Provide the VPC ID and associated subnets to the `VPC` and `ServerSubnets` parameters
 
 ## Security
 
-* You should already have created/uploaded an SSH key to AWS and have the ARN available.
-* You should already have an Admin SG created for inbound SSH connections. The Security Group ID should be provided for the `InboundAdminSecurityGroupId` parameter in `backendless_chef.yaml`, otherwise you won't have any ssh connectivity to your cluster.
+* You should already have created/uploaded an SSH key to AWS. Provide the keypair name to the `KeyName` parameter
+* If you're using the `main.yaml` stack, you must also create a security group in the referenced VPC to define your administrative access.  Provide sg ID to the `InboundAdminSecurityGroupId` parameter
 
-## Amazon ElasticSearch and Service Linked Role (SLR)
+## Amazon Elasticsearch and Service Linked Role (SLR)
 
-Amazon ElasticSearch requires a specific SLR to be created prior to running this CloudFormation template, specifically one called `AWSServiceRoleForAmazonElasticsearchService`. This role cannot be created programmatically as it is created automatically when setting up a VPC access domain in the AWS console. For more information on this [please see this doc from AWS](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/slr-es.html), at this time, even though the doc points to a way to create this manually via the CLI, it still only works via console setup, however AWS adds new features all the time, and by the time you do this, it may work programmatically, please follow their documentation. Once you've created the VPC access domain for AWS ElasticSearch, you can then delete this domain, the role will still be there and you should be able to continue.
+Amazon Elasticsearch requires a specific SLR to be created prior to running this CloudFormation template, specifically one called `AWSServiceRoleForAmazonElasticsearchService`. This role cannot be created programmatically as it is created automatically when setting up a VPC access domain in the AWS console. For more information on this [please see this doc from AWS](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/slr-es.html), at this time, even though the doc points to a way to create this manually via the CLI, it still only works via console setup, however AWS adds new features all the time, and by the time you do this, it may work programmatically, please follow their documentation. Once you've created the VPC access domain for AWS ElasticSearch, you can then delete this domain, the role will still be there and you should be able to continue.
 
 _Note: You will need to do this for each region you plan on setting up Chef in_

README.md

@@ -33,26 +33,35 @@ Before you fire it up, there are a few things you should make sure you have prep
 
 * Please review the [Prerequisites doc before proceeding](./PREREQS.md)
 
-## Fire up the Chef Server stack
+## Browser launch instructions (quickstart)
 
 You can launch this stack with the push of a button:
-<p><a href="https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https:%2F%2Fs3.amazonaws.com%2Faws-native-chef-server%2Fmain.yaml&amp;stackName=my-chef-stack" target="_blank"><img src="https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png" alt="Launch Stack" /></a></p>
 
-However, the most repeatable and least error-prone way to launch this stack is to use the `aws` command-line. First copy file `stack_parameters.json.example` to `stack_parameters.json`, make the necessary changes, then run this command:
+| Edition | Version | Est Monthly cost | Launch button |
+| ------- | ------- | ---------------- | ------------- |
+| High Performance | 5.0.2 | [$1500*](https://calculator.s3.amazonaws.com/index.html#r=IAD&s=EC2&key=calc-E9251374-D80C-45B7-BED7-C9C9778B1D6C) | <p><a href="https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https:%2F%2Fs3.amazonaws.com%2Faws-native-chef-server%2F5.0.2%2Fmain.yaml&amp;stackName=my-chef-stack" target="_blank"><img src="https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png" alt="Launch Stack" /></a></p> |
+| Lowest Cost | 5.0.2 | [$320*](https://calculator.s3.amazonaws.com/index.html#r=IAD&s=EC2&key=files/calc-165638db8e331664846c0d6654d743377bd3eac6&v=ver20190604sQ) | <p><a href="https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https:%2F%2Fs3.amazonaws.com%2Faws-native-chef-server%2F5.0.2%2Fmarketplace.yaml&amp;stackName=my-chef-stack" target="_blank"><img src="https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png" alt="Launch Stack" /></a></p> |
+
+_*NOTE: Estimated costs are just that, your mileage will vary. Use reserved instances to greatly reduce costs for long-running clusters._
+
+## CLI Launch instructions
+However, the most repeatable and least error-prone way to launch this stack is to use the `aws` command-line. First copy file `stack_parameters_main.json.example` to `stack_parameters_main.json`, make the necessary changes, then run this command:
 
 ```bash
 MYBUCKET=aws-native-chef-server
 MYID=mycompany
+VERSION=5.0.2
+EDITION=main  # main or marketplace
 # Configure the automate_stack_parameters.json and then launch the cloudformation stack:
 # If you're using your own bucket, uncomment the next line:
-# aws s3 sync . s3://$MYBUCKET/ --exclude "*" --include "*.yaml" --include "files/*" && \
-aws cloudformation validate-template --template-url https://s3.amazonaws.com/$MYBUCKET/main.yaml && \
+# aws s3 sync . s3://$MYBUCKET/$VERSION/ --exclude "*" --include "*.yaml" --include "files/*" && \
+aws cloudformation validate-template --template-url https://s3.amazonaws.com/$MYBUCKET/$VERSION/$EDITION.yaml && \
 aws cloudformation create-stack \
   --stack-name ${MYID}-chef-stack \
-  --template-url https://s3.amazonaws.com/$MYBUCKET/main.yaml \
+  --template-url https://s3.amazonaws.com/$MYBUCKET/$VERSION/$EDITION.yaml \
   --capabilities CAPABILITY_IAM \
   --on-failure DO_NOTHING \
-  --parameters file://stack_parameters.json
+  --parameters file://stack_parameters_$EDITION.json
 ```
 
 ## Updating the stack
@@ -62,14 +71,16 @@ If you've made changes to the template content or parameters and you wish to upd
 ```bash
 MYBUCKET=aws-native-chef-server
 MYID=mycompany
+VERSION=5.0.2
+EDITION=main  # main or marketplace
 # If you're using your own bucket, uncomment the next line:
-# aws s3 sync . s3://$MYBUCKET/ --exclude "*" --include "*.yaml" --include "files/*" && \
-aws cloudformation validate-template --template-url https://s3.amazonaws.com/$MYBUCKET/main.yaml && \
+# aws s3 sync . s3://$MYBUCKET/$VERSION/ --exclude "*" --include "*.yaml" --include "files/*" && \
+aws cloudformation validate-template --template-url https://s3.amazonaws.com/$MYBUCKET/$VERSION/$EDITION.yaml && \
 aws cloudformation update-stack \
   --stack-name ${MYID}-chef-stack \
-  --template-url https://s3.amazonaws.com/$MYBUCKET/main.yaml \
+  --template-url https://s3.amazonaws.com/$MYBUCKET/$VERSION/$EDITION.yaml \
   --capabilities CAPABILITY_IAM \
-  --parameters file://stack_parameters.json
+  --parameters file://stack_parameters_$EDITION.json
 ```
 
 Note: For production instances it is recommended to use the CloudFormation console so that you can get a report of all changes before executing them.  Particularly pay attention to any resources that are being replaced.
@@ -79,10 +90,8 @@ Note: For production instances it is recommended to use the CloudFormation conso
 If you're using a bastion host and need to SSH from the outside:
 
 ```bash
-ssh -o ProxyCommand="ssh -W %h:%p -q user@bastion" -l user <chef server private ip>
+ssh -o ProxyCommand="ssh -W %h:%p -q centos@bastion" -l centos <chef server private ip>
 ```
-Where "user" is `ec2-user` on the RHEL AMI, and `centos` on the CentOS AMI
-
 otherwise just SSH directly to the public IPs of the chef servers
 
 ## Upgrading the Chef Server