From 74f84af761250fa50b7b84b93c9572ac4735f614 Mon Sep 17 00:00:00 2001 From: Christopher Hunter Date: Fri, 30 Jun 2023 20:00:39 -0700 Subject: [PATCH] add opinionated tile author guide --- TILE_AUTHOR_GUIDE.md | 385 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 385 insertions(+) create mode 100644 TILE_AUTHOR_GUIDE.md diff --git a/TILE_AUTHOR_GUIDE.md b/TILE_AUTHOR_GUIDE.md new file mode 100644 index 000000000..f99cdf35e --- /dev/null +++ b/TILE_AUTHOR_GUIDE.md @@ -0,0 +1,385 @@ +# Tile Author Guide + +> Kiln provides utilities and conventions for maintaining the source to build and release ".pivotal" files. + +### Table of Conents +- [Authoring Product Template Parts](#authoring-product-template-parts) +- [Managing BOSH Release Tarballs](#bosh-release-tarballs) + - [Kilnfile BOSH Release Specification and Locking](#bosh-release-sources) + - Sources + - [BOSH.io](#release-source-boshio) + - [GitHub releases](#release-source-github) + - [Build Artifactory](#release-source-artifactory) + - [AWS S3](#release-source-s3) + - [Local Files](#release-source-directory) + - BOSH release compilation +- Stemcell Version Management +- Tile release note Generation +- TanzuNet Release Publication +- Provides importable utilities for Tile Authors + +### Conventional tile source files and directories + +In my exploration of tile source code, I found teams that use Kiln often have quite divergent file and directory names for their tile source code. +This is partially because `kiln bake` required all directories and files to be passed as flags. +It did not make assumptions based on convention. +It now does. +Your experience with `kiln bake` will be a lot easier if you follow some conventions. + +These are the default directory names Kiln expects: +- bosh_variables +- forms +- instance_groups +- jobs +- migrations +- properties +- releases +- runtime_configs + +It also expects these files: +- base.yml +- version +- icon.png + + +If you follow conventions you may experience the sweetness of running `kiln bake` without flags and getting a tile. +Again, see [hello-tile](https://github.com/crhntr/hello-tile) (non-VMware employees) or the TAS repo (VMware employees) for tile source code that kiln "just works" with. + +#### Starting from scratch + +The following code does not create a working tile; it creates conventional defaults to start/standardize your tile authorship journey with Kiln. +```shell +# Make directories and keep them around +mkdir -p bosh_variables forms instance_groups jobs migrations properties releases runtime_configs +touch {bosh_variables,forms,instance_groups,jobs,migrations,properties,releases,runtime_configs}/.gitkeep + +# Add a tile image +curl -L -o icon.png "https://github.com/crhntr/hello-tile/blob/main/icon.png?raw=true" + +# Add a version file (this is the default location Kiln reads from) +echo '0.1.0' > 'version' + +# Create a "base.yml" with some minimal fields +# See documentation here: `https://docs.pivotal.io/tiledev/2-9/property-template-references.html` +cat << EOF > base.yml +name: my-tile-name +label: "" +description: "" +icon_image: $( icon ) +metadata_version: "3.0.0" +minimum_version_for_upgrade: 0.0.0 +product_version: $( version ) +provides_product_versions: + - name: hello + version: $( version ) +rank: 1 +serial: false +releases: [] +stemcell_criteria: [] +job_types: [] +runtime_configs: [] +property_blueprints: [] +form_types: [] +EOF +``` + +## Authoring Product Template Parts + +A tile zip file with a ".pivotal" suffix. +The zip file must have a YAML file in the `metadata` subdirectory of the tile. +The file specification is documented in the [**Property and Template References** page in the Ops Manager Tile Developer Guide](https://docs.pivotal.io/tiledev/2-9/property-template-references.html). +While there are many other tools out there for YAML munging like +[YTT](https://carvel.dev/ytt/), +[bosh interpolate](https://bosh.io/docs/cli-int/), +and [CUE](https://cuelang.org/). +The Tanzu Application Service for VMs team has built our own YAML templating tooling that has proven itself in maintaining +[TAS for VMs](https://docs.vmware.com/en/VMware-Tanzu-Application-Service/4.0/tas-for-vms/configure-pas.html) +(and [small footprint TAS for VMs](https://docs.vmware.com/en/VMware-Tanzu-Application-Service/4.0/tas-for-vms/small-footprint.html)), +[IST](https://docs.vmware.com/en/VMware-Tanzu-Application-Service/4.0/tas-for-vms/installing-pcf-is.html), +[TASW](https://docs.vmware.com/en/VMware-Tanzu-Application-Service/4.0/tas-for-vms/installing-pcf-is.html). +It has a bunch of functionality to help you generate identical Product Template files even if you need to refactor your source. +While it has proven itself in "large" tiles with many bosh releases, it can be scaled down to work with smaller tiles. +[@crhntr wrote an open source example tile here](https://github.com/crhntr/hello-tile). + +While you can write your entire Product Template in a single file, breaking up metadata parts into different yaml files and directories makes it easier to find configuration. + +[Please read and understand the implications of this rank field documentation.](https://docs.pivotal.io/tiledev/2-9/property-template-references.html#top-rank). + +## Managing BOSH Release Tarballs + +`kiln fetch` downloads BOSH Release Tarballs from any of the following "sources" +and puts them in a "./releases" directory. + +Before Kiln can help you manage the BOSH Releases you put in the tile, you need to upload BOSH Release tarballs to a place accessable to Kiln. + +See the sources below to decide which is right for your release. + +Unlike Tile Generator. +Kiln does not create releases. +The Kiln way is to +- have BOSH releases each have their own source control +- have CI build and upload (finalized) BOSH release tarballs somewhere +- have a tile source repository + - in the tile source repository specify how Kiln can get the BOSH release tarballs + +While the following examples start from empty directories and mostly use S3 and BOSH.io. +There are similar simple scripts for a small test tile illustrating similar usage patterns to the follwoing example. +See [hello-tile](https://github.com/crhntr/hello-tile). +Hello Tile consumes a single custom BOSH Release, [hello-release](https://github.com/crhntr/hello-release), from a GitHub release. +It does not upload the release to TanzuNet but adds the built tile to a GitHub Release. + +#### **EXAMPLE** Using Kiln to Download BOSH Release Tarballs +This starts from an empty directory and downloads the latest BPM release from bosh.io. +Note, the Kilnfile and Kilnfile.lock (unfortunately/frustratingly) must be created manually. + +```sh +# Create and go into an empty directory +mkdir -p /tmp/try-kiln-fetch +cd !$ +mkdir -p releases +touch release/.gitkeep # not required but a good idea + +# Hack a Kilnfile and Kilnfile lock +echo '{"release_sources": [{type: bosh.io}], "releases": [{"name": "bpm"}]}' > Kilnfile +yq '{"releases": [. | {"name": "bpm", "version": .version, "sha1": .sha, "remote_source": "bosh.io", "remote_path": .remote_path}]}' <(kiln find-release-version --release=bpm) > Kilnfile.lock + +# Call Kiln fetch +kiln fetch + +# See the fetched release +stat releases/bpm*.tgz +``` + +The files should look something like these +```yaml +# Expected Kilnfile +release_sources: + - type: "bosh.io" +releases: + - name: bpm +``` + +```yaml +# Expected Kilnfile.lock +releases: + - name: bpm + version: "1.2.3" + sha1: "ad12bb4e1d2c0b94ef679670a99adaff920850d0" + remote_source: bosh.io + remote_path: "https://bosh.io/d/github.com/cloudfoundry/bpm-release?v=1.2.3" +``` + +
+PPE TODO + +The YQ expressions are a hack to get this to work from an empty directory. +We need to improve this process. +Kiln fetch was built around an existing "assets.lock"; +the developer experience for starting from an empty directory is not polished. +
+ +#### **EXAMPLE** Using Kiln to update BOSH Release Tarballs + +Updating a release in a lock file requires two kiln commands _(this developer experience needs work too IMO)_. + +Please follow the ["Kiln Fetch Example"](#kiln-fetch-example) before following this one. + +```sh +# (optional) Add a version constraint to the Kilnfile. +# This shows how Kiln will select a version that matches a constrint. +yq '(.releases[] | select(.name == "bpm")) |= .version = "~1.1"' Kilnfile + +# Find a new BOSH Release Tarball version (on bosh.io) +export NEW_RELEASE_VERSION="$(kiln find-release-version --release=bpm | yq '.version')" +echo "Kiln found: bpm/${NEW_RELEASE_VERSION}" + +# Update the Kilnfile.lock with the new version +kiln update-release --version="${NEW_RELEASE_VERSION}" --name="bpm" +``` + +The syntax for BOSH Release Tarball version constraints is [Masterminds/semver](https://github.com/Masterminds/semver#checking-version-constraints). +Other parts of the Cloud Foundry ecosystem use [blang/semver](https://github.com/blang/semver#ranges). +If you get unexpected results, this difference may be the cause. +For simple version constraints they are similar enough. + +
+PPE TODO + +The release name flag difference is awkward. +In `find-release-version`, the flag is `--release`; +In `kiln update-release`, the flag is `--name`. + +Maybe this should be one command and an optional flag +- `kiln update-release` →`kiln update-bosh-release bpm` +- `kiln find-release-version` → `kiln update-bosh-release --dry-run bpm` + +
+ +### Release Sources + +While different credentials per release source element are currently supported. +I would recomend one set of credentials per release source type. + +#### BOSH.io + +This release source has minimal configuration. +Just add it to your `release_sources` and you can get releases from [BOSH.io](https://bosh.io/releases/). + +#### GitHub Release Artifacts + +To download BOSH Release Tarballs from GitHub Releases, add the following +``` +release_sources: + - type: "github" + org: "crhntr" + github_token: $(variable "github_token") +``` + +You will need one entry per organization. +Some examples are: "pivotal", "cloudfoundry", "pivotal-cf", or your personal/company GitHub username. + +You will need to add the following flag to most commands: + +``` +# Optional helper +export GITHUB_TOKEN="$(gh auth status --show-token 2>&1 | grep 'Token:' | awk '{print $NF}')" + +# Example Kiln variable flag +kiln fetch --variable="github_token=${GITHUB_TOKEN}" +``` + +#### Build Artifactory + +_TODO_ + +#### AWS S3 + +_TODO_ + +#### Local files + +_TODO_ + +#### Default credentials file + +You can add a default credentials file to `~/.kiln/credentials.yml` so you don't need to pass variables flags everywhere. +Don't do this with production creds but if you have credentials you can safely write to your disk, consider using this functionality. +The file can look like this +```yaml +# GitHub release sources credentials +github_token: some-token + +# S3 release source credentials +aws_secret_access_key: some-key +aws_access_key_id: some-id + +# Artifactory release source credentials +artifactory_username: some-username +artifactory_password: some-password +``` + +### Release Compilation + +_WORK IN PROGRESS EXAMPLE_ + +``` +# create a tile with the releases you want compiled +kiln bake + +# Add an S3 (or Artifactory) release source to your Kilnfile +yq -i '.release_sources = [{"type": "s3", "id": "my_compiled_release_bucket", bucket": "some_bucket", "publishable": true, "access_key_id": "some_id", "secret_access_key": "some_id"}] + .release_sources' Kilnfile + +# claim a bosh director and configure the environment +smith claim -p us_4_0 +eval "$(smith om)" +eval "$(smith bosh)" + +# deploy your Product (the commands should look something like this) +om upload-product --product=tile.pivotal +om configure-product --config=simple_config.yml +om apply-changes --product-name=my-tile-name + +kiln cache-compiled-releases --upload-target-id=my_compiled_release_bucket --name=hello +``` + +### Temporary BOSH Release Tarball Locking + +_TODO_ + +`kiln glaze` + +## Stemcell Version Management + +_TODO_ + +`kiln find-stemcell-version` and `kiln update-stemcell` + +## Tile Release Note Generation + +_TODO_ + +`kiln release-notes` + +## TanzuNet Release Publication + +_TODO? we might deprecate this functionality_ + +`kiln publish` + + +## Provides importable utilities for Tile Authors + +_TODO_ + +Kiln is written in Go. +Some parts of the Kiln CLI source code hase been made available in the "pkg" subdirectory to be used in your own Go projects. + +This source is useful for writing manifest/metadata tests and for writing tile utilities. + +## Tile Build Packs + +There is an internal VMware intiative to build stuff using TAP and buildpacks. +The Kiln Buildpack can take tile source code and create a tile. +For it to work, +you need to have your BOSH Release Tarballs fetch-able by Kiln (and only using GitHub or BOSH.io release sources) +and it is nice if your bake command not require too many flags (see [Tile Source Conventions](#tile-source-conventions)). + +The private repository [kiln buildpack](https://github.com/pivotal/kiln-buildpack) has the Pakito buildpack source. +You can run the acceptance tests with a `TILE_DIRECTORY` environment variable set to your tile source to see if your tile will build with the buildpack. + +``` +mkdir -p /tmp/try-kiln-buildpack +cd !$ + +set -e + +# Clone the Buildpack source +git clone git@github.com:pivotal/kiln-buildpack +cd kiln-buildpack + +# Check the path to your tile directory +stat ${HOME}/workspace/my-tile-source/Kilnfile.lock + +# WARNING this does a git clean. This is important to simulate building from source. +# If you have releases fetched already, you won't get an acurate test. +cd stat ${HOME}/workspace/my-tile-source && git clean -ffd && cd - + +# Run the acceptance test against your tile source +TILE_DIRECTORY="${HOME}/workspace/my-tile-source" go test -v --tags=acceptance . + +stat /tmp/try-kiln-buildpack/kiln-buildpack/tile.pivotal +``` + +Just becuase it works here, does not mean it will work for you. + + +## Other Tips + +### .gitignore + +The following are some useful .gitignore contents. + +```.gitignore +releases/*.tgz +*.pivotal +``` \ No newline at end of file