add opinionated tile author guide

TILE_AUTHOR_GUIDE.md

@@ -0,0 +1,385 @@
+# Tile Author Guide
+
+> Kiln provides utilities and conventions for maintaining the source to build and release ".pivotal" files.
+
+### Table of Conents
+- [Authoring Product Template Parts](#authoring-product-template-parts)
+- [Managing BOSH Release Tarballs](#bosh-release-tarballs)
+ - [Kilnfile BOSH Release Specification and Locking](#bosh-release-sources)
+ - Sources
+ - [BOSH.io](#release-source-boshio)
+ - [GitHub releases](#release-source-github)
+ - [Build Artifactory](#release-source-artifactory)
+ - [AWS S3](#release-source-s3)
+ - [Local Files](#release-source-directory)
+ - BOSH release compilation
+- Stemcell Version Management
+- Tile release note Generation
+- TanzuNet Release Publication
+- Provides importable utilities for Tile Authors
+
+### <a href="tile-source-conventions"></a> Conventional tile source files and directories
+
+In my exploration of tile source code, I found teams that use Kiln often have quite divergent file and directory names for their tile source code.
+This is partially because `kiln bake` required all directories and files to be passed as flags.
+It did not make assumptions based on convention.
+It now does.
+Your experience with `kiln bake` will be a lot easier if you follow some conventions.
+
+These are the default directory names Kiln expects:
+- bosh_variables
+- forms
+- instance_groups
+- jobs
+- migrations
+- properties
+- releases
+- runtime_configs
+
+It also expects these files:
+- base.yml
+- version
+- icon.png
+
+
+If you follow conventions you may experience the sweetness of running `kiln bake` without flags and getting a tile.
+Again, see [hello-tile](https://github.com/crhntr/hello-tile) (non-VMware employees) or the TAS repo (VMware employees) for tile source code that kiln "just works" with.
+
+#### Starting from scratch
+
+The following code does not create a working tile; it creates conventional defaults to start/standardize your tile authorship journey with Kiln.
+```shell
+# Make directories and keep them around
+mkdir -p bosh_variables forms instance_groups jobs migrations properties releases runtime_configs
+touch {bosh_variables,forms,instance_groups,jobs,migrations,properties,releases,runtime_configs}/.gitkeep
+
+# Add a tile image
+curl -L -o icon.png "https://github.com/crhntr/hello-tile/blob/main/icon.png?raw=true"
+
+# Add a version file (this is the default location Kiln reads from)
+echo '0.1.0' > 'version'
+
+# Create a "base.yml" with some minimal fields
+# See documentation here: `https://docs.pivotal.io/tiledev/2-9/property-template-references.html`
+cat << EOF > base.yml
+name: my-tile-name
+label: ""
+description: ""
+icon_image: $( icon )
+metadata_version: "3.0.0"
+minimum_version_for_upgrade: 0.0.0
+product_version: $( version )
+provides_product_versions:
+ - name: hello
+ version: $( version )
+rank: 1
+serial: false
+releases: []
+stemcell_criteria: []
+job_types: []
+runtime_configs: []
+property_blueprints: []
+form_types: []
+EOF
+```
+
+## <a id="authoring-product-template-parts"></a> Authoring Product Template Parts
+
+A tile zip file with a ".pivotal" suffix.
+The zip file must have a YAML file in the `metadata` subdirectory of the tile.
+The file specification is documented in the [**Property and Template References** page in the Ops Manager Tile Developer Guide](https://docs.pivotal.io/tiledev/2-9/property-template-references.html).
+While there are many other tools out there for YAML munging like
+[YTT](https://carvel.dev/ytt/),
+[bosh interpolate](https://bosh.io/docs/cli-int/),
+and [CUE](https://cuelang.org/).
+The Tanzu Application Service for VMs team has built our own YAML templating tooling that has proven itself in maintaining
+[TAS for VMs](https://docs.vmware.com/en/VMware-Tanzu-Application-Service/4.0/tas-for-vms/configure-pas.html)
+(and [small footprint TAS for VMs](https://docs.vmware.com/en/VMware-Tanzu-Application-Service/4.0/tas-for-vms/small-footprint.html)),
+[IST](https://docs.vmware.com/en/VMware-Tanzu-Application-Service/4.0/tas-for-vms/installing-pcf-is.html),
+[TASW](https://docs.vmware.com/en/VMware-Tanzu-Application-Service/4.0/tas-for-vms/installing-pcf-is.html).
+It has a bunch of functionality to help you generate identical Product Template files even if you need to refactor your source.
+While it has proven itself in "large" tiles with many bosh releases, it can be scaled down to work with smaller tiles.
+[@crhntr wrote an open source example tile here](https://github.com/crhntr/hello-tile).
+
+While you can write your entire Product Template in a single file, breaking up metadata parts into different yaml files and directories makes it easier to find configuration.
+
+[Please read and understand the implications of this rank field documentation.](https://docs.pivotal.io/tiledev/2-9/property-template-references.html#top-rank).
+
+## <a id="bosh-release-tarballs"></a> Managing BOSH Release Tarballs
+
+`kiln fetch` downloads BOSH Release Tarballs from any of the following "sources"
+and puts them in a "./releases" directory.
+
+Before Kiln can help you manage the BOSH Releases you put in the tile, you need to upload BOSH Release tarballs to a place accessable to Kiln.
+
+See the sources below to decide which is right for your release.
+
+Unlike Tile Generator.
+Kiln does not create releases.
+The Kiln way is to 
+- have BOSH releases each have their own source control
+- have CI build and upload (finalized) BOSH release tarballs somewhere
+- have a tile source repository
+ - in the tile source repository specify how Kiln can get the BOSH release tarballs
+
+While the following examples start from empty directories and mostly use S3 and BOSH.io.
+There are similar simple scripts for a small test tile illustrating similar usage patterns to the follwoing example.
+See [hello-tile](https://github.com/crhntr/hello-tile).
+Hello Tile consumes a single custom BOSH Release, [hello-release](https://github.com/crhntr/hello-release), from a GitHub release.
+It does not upload the release to TanzuNet but adds the built tile to a GitHub Release.
+
+#### <a id="kiln-fetch-example"></a> **EXAMPLE** Using Kiln to Download BOSH Release Tarballs
+This starts from an empty directory and downloads the latest BPM release from bosh.io.
+Note, the Kilnfile and Kilnfile.lock (unfortunately/frustratingly) must be created manually.
+
+```sh
+# Create and go into an empty directory
+mkdir -p /tmp/try-kiln-fetch
+cd !$
+mkdir -p releases
+touch release/.gitkeep # not required but a good idea
+
+# Hack a Kilnfile and Kilnfile lock
+echo '{"release_sources": [{type: bosh.io}], "releases": [{"name": "bpm"}]}' > Kilnfile
+yq '{"releases": [. | {"name": "bpm", "version": .version, "sha1": .sha, "remote_source": "bosh.io", "remote_path": .remote_path}]}' <(kiln find-release-version --release=bpm) > Kilnfile.lock
+
+# Call Kiln fetch
+kiln fetch
+
+# See the fetched release
+stat releases/bpm*.tgz
+```
+
+The files should look something like these
+```yaml
+# Expected Kilnfile
+release_sources:
+ - type: "bosh.io"
+releases:
+ - name: bpm
+```
+
+```yaml
+# Expected Kilnfile.lock
+releases:
+ - name: bpm
+ version: "1.2.3"
+ sha1: "ad12bb4e1d2c0b94ef679670a99adaff920850d0"
+ remote_source: bosh.io
+ remote_path: "https://bosh.io/d/github.com/cloudfoundry/bpm-release?v=1.2.3"
+```
+
+<details>
+<summary>PPE TODO</summary>
+
+The YQ expressions are a hack to get this to work from an empty directory.
+We need to improve this process.
+Kiln fetch was built around an existing "assets.lock";
+the developer experience for starting from an empty directory is not polished.
+</details>
+
+#### **EXAMPLE** Using Kiln to update BOSH Release Tarballs
+
+Updating a release in a lock file requires two kiln commands _(this developer experience needs work too IMO)_.
+
+Please follow the ["Kiln Fetch Example"](#kiln-fetch-example) before following this one.
+
+```sh
+# (optional) Add a version constraint to the Kilnfile.
+# This shows how Kiln will select a version that matches a constrint.
+yq '(.releases[] | select(.name == "bpm")) |= .version = "~1.1"' Kilnfile
+
+# Find a new BOSH Release Tarball version (on bosh.io)
+export NEW_RELEASE_VERSION="$(kiln find-release-version --release=bpm | yq '.version')"
+echo "Kiln found: bpm/${NEW_RELEASE_VERSION}"
+
+# Update the Kilnfile.lock with the new version
+kiln update-release --version="${NEW_RELEASE_VERSION}" --name="bpm"
+```
+
+The syntax for BOSH Release Tarball version constraints is [Masterminds/semver](https://github.com/Masterminds/semver#checking-version-constraints).
+Other parts of the Cloud Foundry ecosystem use [blang/semver](https://github.com/blang/semver#ranges).
+If you get unexpected results, this difference may be the cause.
+For simple version constraints they are similar enough.
+
+<details>
+<summary>PPE TODO</summary>
+
+The release name flag difference is awkward.
+In `find-release-version`, the flag is `--release`;
+In `kiln update-release`, the flag is `--name`.
+
+Maybe this should be one command and an optional flag
+- `kiln update-release` →`kiln update-bosh-release bpm`
+- `kiln find-release-version` → `kiln update-bosh-release --dry-run bpm`
+
+</details>
+
+### Release Sources
+
+While different credentials per release source element are currently supported.
+I would recomend one set of credentials per release source type.
+
+#### <a id="release-source-boshio"></a> BOSH.io
+
+This release source has minimal configuration.
+Just add it to your `release_sources` and you can get releases from [BOSH.io](https://bosh.io/releases/).
+
+#### <a id='release-source-github'></a> GitHub Release Artifacts
+
+To download BOSH Release Tarballs from GitHub Releases, add the following
+```
+release_sources:
+ - type: "github"
+ org: "crhntr"
+ github_token: $(variable "github_token")
+```
+
+You will need one entry per organization.
+Some examples are: "pivotal", "cloudfoundry", "pivotal-cf", or your personal/company GitHub username.
+
+You will need to add the following flag to most commands:
+
+```
+# Optional helper
+export GITHUB_TOKEN="$(gh auth status --show-token 2>&1 | grep 'Token:' | awk '{print $NF}')"
+
+# Example Kiln variable flag
+kiln fetch --variable="github_token=${GITHUB_TOKEN}"
+```
+
+#### <a id='release-source-artifactory'></a> Build Artifactory
+
+_TODO_
+
+#### <a id='release-source-s3'></a> AWS S3
+
+_TODO_
+
+#### <a id='release-source-directory'></a> Local files
+
+_TODO_
+
+#### Default credentials file
+
+You can add a default credentials file to `~/.kiln/credentials.yml` so you don't need to pass variables flags everywhere.
+Don't do this with production creds but if you have credentials you can safely write to your disk, consider using this functionality.
+The file can look like this
+```yaml
+# GitHub release sources credentials
+github_token: some-token
+
+# S3 release source credentials
+aws_secret_access_key: some-key
+aws_access_key_id: some-id
+
+# Artifactory release source credentials
+artifactory_username: some-username
+artifactory_password: some-password
+```
+
+### Release Compilation
+
+_WORK IN PROGRESS EXAMPLE_
+
+```
+# create a tile with the releases you want compiled
+kiln bake
+
+# Add an S3 (or Artifactory) release source to your Kilnfile
+yq -i '.release_sources = [{"type": "s3", "id": "my_compiled_release_bucket", bucket": "some_bucket", "publishable": true, "access_key_id": "some_id", "secret_access_key": "some_id"}] + .release_sources' Kilnfile
+
+# claim a bosh director and configure the environment
+smith claim -p us_4_0
+eval "$(smith om)"
+eval "$(smith bosh)"
+
+# deploy your Product (the commands should look something like this)
+om upload-product --product=tile.pivotal
+om configure-product --config=simple_config.yml
+om apply-changes --product-name=my-tile-name
+
+kiln cache-compiled-releases --upload-target-id=my_compiled_release_bucket --name=hello
+```
+
+### Temporary BOSH Release Tarball Locking
+
+_TODO_
+
+`kiln glaze`
+
+## Stemcell Version Management
+
+_TODO_
+
+`kiln find-stemcell-version` and `kiln update-stemcell`
+
+## Tile Release Note Generation
+
+_TODO_
+
+`kiln release-notes`
+
+## TanzuNet Release Publication
+
+_TODO? we might deprecate this functionality_
+
+`kiln publish`
+
+
+## Provides importable utilities for Tile Authors
+
+_TODO_
+
+Kiln is written in Go.
+Some parts of the Kiln CLI source code hase been made available in the "pkg" subdirectory to be used in your own Go projects.
+
+This source is useful for writing manifest/metadata tests and for writing tile utilities. 
+
+## Tile Build Packs
+
+There is an internal VMware intiative to build stuff using TAP and buildpacks.
+The Kiln Buildpack can take tile source code and create a tile.
+For it to work,
+you need to have your BOSH Release Tarballs fetch-able by Kiln (and only using GitHub or BOSH.io release sources)
+and it is nice if your bake command not require too many flags (see [Tile Source Conventions](#tile-source-conventions)).
+
+The private repository [kiln buildpack](https://github.com/pivotal/kiln-buildpack) has the Pakito buildpack source.
+You can run the acceptance tests with a `TILE_DIRECTORY` environment variable set to your tile source to see if your tile will build with the buildpack.
+
+```
+mkdir -p /tmp/try-kiln-buildpack
+cd !$
+
+set -e
+
+# Clone the Buildpack source
+git clone [email protected]:pivotal/kiln-buildpack
+cd kiln-buildpack
+
+# Check the path to your tile directory
+stat ${HOME}/workspace/my-tile-source/Kilnfile.lock
+
+# WARNING this does a git clean. This is important to simulate building from source.
+# If you have releases fetched already, you won't get an acurate test. 
+cd stat ${HOME}/workspace/my-tile-source && git clean -ffd && cd -
+
+# Run the acceptance test against your tile source
+TILE_DIRECTORY="${HOME}/workspace/my-tile-source" go test -v --tags=acceptance .
+
+stat /tmp/try-kiln-buildpack/kiln-buildpack/tile.pivotal
+```
+
+Just becuase it works here, does not mean it will work for you.
+
+
+## Other Tips
+
+### .gitignore
+
+The following are some useful .gitignore contents.
+
+```.gitignore
+releases/*.tgz
+*.pivotal
+```