Skip to content

Commit

Permalink
replace some todos with first draft documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
@crhntr
crhntr committed Jul 5, 2023
1 parent 6008f59 commit 7105a85
Showing 1 changed file with 172 additions and 45 deletions.
217 changes: 172 additions & 45 deletions TILE_AUTHOR_GUIDE.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
_The shell script examples use [kiln 0.85.0](https://github.com/pivotal-cf/kiln/releases/tag/v0.85.0), [yq 4.34.1](https://github.com/mikefarah/yq), and **curl 7.88.1**._

### Table of Conents
### Table of Contents
- [Authoring Product Template Parts](#authoring-product-template-parts)
- [Managing BOSH Release Tarballs](#bosh-release-tarballs)
- [Kilnfile BOSH Release Specification and Locking](#bosh-release-sources)
Expand All @@ -18,38 +18,12 @@ _The shell script examples use [kiln 0.85.0](https://github.com/pivotal-cf/kiln/
- Stemcell Version Management
- Testing
- Tile release note Generation
- TanzuNet Release Publication
- PivNet Release Publication
- Provides importable utilities for Tile Authors

### <a href="tile-source-conventions"></a> Conventional tile source files and directories

In my exploration of tile source code, I found teams that use Kiln often have quite divergent file and directory names for their tile source code.
This is partially because `kiln bake` required all directories and files to be passed as flags.
It did not make assumptions based on convention.
It now does.
Your experience with `kiln bake` will be a lot easier if you follow some conventions.

These are the default directory names Kiln expects:
- bosh_variables
- forms
- instance_groups
- jobs
- migrations
- properties
- releases
- runtime_configs

It also expects these files:
- base.yml
- version
- icon.png


If you follow conventions you may experience the sweetness of running `kiln bake` without flags and getting a tile.
Again, see [hello-tile](https://github.com/crhntr/hello-tile) (non-VMware employees) or the TAS repo (VMware employees) for tile source code that kiln "just works" with.


#### Starting from scratch
## Bootstrapping a tile repository

The following code bootstraps Product template parts from an empty directory working tile; it creates conventional defaults to start/standardize your tile authorship journey with Kiln.
It does not generate source for a working tile.
Expand Down Expand Up @@ -112,7 +86,45 @@ Kiln expects certain directories to contain Product Template (metadata) parts.
- `./releases` should be an empty directory (maybe containing .gitkeep) and will be where Kiln writes BOSH Release Tarballs
- `./runtime_configs` should contain YAML files that have [runtime configs](https://docs.pivotal.io/tiledev/2-9/runtime-config.html)

_TODO show how to add Product Template parts from the above to base.yml._
Also add the following files
- `base.yml` should be the entrypoint for the manifest template.
- `version` should contain the tile version
- `icon.png` should be the tile image

While you can use other directory and filenames and pas those to Kiln,
you would have a better experience if you follow the above naming conventions.

### Adding product template parts to your Metadata

#### Product Template baking functions

Each product template part has a "name" field.
When you `kiln bake`, the contents of `base.yml` go through a few templating steps.
One of those steps exposes the following functions.
These functions will read, interpolate, and format metadata parts for use in your tile's metadata.yml.

You can reference the part in base.yml by using the follwing template functions:
- `bosh_variable` reads, interpolates, and formats a product template part from `./bosh_variables`
- `form` reads, interpolates, and formats a product template part from `./forms`
- `property` reads, interpolates, and formats a product template part from `./properties`
- `release` reads, interpolates, and formats a BOSH release data from either the `./releases` directory or the Kilnfile.lock
- `stemcell` reads, interpolates, and formats a Stemcell Criteria data from the Kilnfile.lock
- `version` returns the contents of the `./version` file
- `variable` finds the named variables from either a `--variables-file` or `--variable`
- `icon` returns the base64 encoded content of `icon.png`
- `instance_group` reads, interpolates, and formats a product template part from `./instance_groups`
- `job` reads, interpolates, and formats a product template part from `./jobs`
- `runtime_config` reads, interpolates, and formats a product template part from `./runtime_configs`

Other functions:
- `tile` _TODO function documentation_
- `select` _TODO function documentation_
- `regexReplaceAll` _TODO function documentation_

See the [crhntr/hello-tile/base.yml](https://github.com/pivotal/hello-tile/blob/main/base.yml) for some use of the above functions.
The property definition in [crhntr/hello-tile/properties/hello.yml](https://github.com/pivotal/hello-tile/blob/main/properties/hello.yml)
in referenced in `base.yml` using `$( property "port" )`.
Most other product template part functions behave similarly.

## <a id="bosh-release-tarballs"></a> Managing BOSH Release Tarballs

Expand All @@ -135,7 +147,7 @@ While the following examples start from empty directories and mostly use S3 and
There are similar simple scripts for a small test tile illustrating similar usage patterns to the follwoing example.
See [hello-tile](https://github.com/crhntr/hello-tile).
Hello Tile consumes a single custom BOSH Release, [hello-release](https://github.com/crhntr/hello-release), from a GitHub release.
It does not upload the release to TanzuNet but adds the built tile to a GitHub Release.
It does not upload the release to PivNet but adds the built tile to a GitHub Release.

#### <a id="kiln-fetch-example"></a> **EXAMPLE** Using Kiln to Download BOSH Release Tarballs
This starts from an empty directory and downloads the latest BPM release from bosh.io.
Expand Down Expand Up @@ -189,7 +201,7 @@ the developer experience for starting from an empty directory is not polished.
#### **EXAMPLE** Using Kiln to update BOSH Release Tarballs
Updating a release in a lock file requires two kiln commands _(this developer experience needs work too IMO)_.
Updating a release in a lock file requires two kiln commands.
Please follow the ["Kiln Fetch Example"](#kiln-fetch-example) before following this one.
Expand Down Expand Up @@ -217,6 +229,8 @@ This can cause `kiln validate` to fail when a version passed to `kiln update-rel
<details>
<summary>PPE TODO</summary>

This developer experience needs work IMO.

The release name flag difference is awkward.
In `find-release-version`, the flag is `--release`;
In `kiln update-release`, the flag is `--name`.
Expand All @@ -232,13 +246,89 @@ Maybe this should be one command and an optional flag
While different credentials per release source element are currently supported.
I would recommend one set of credentials per release source type.

In the Kilnfile.lock, BOSH release tarballs lock elements have a few fields.
- `name`: The BOSH release name
- `version`: The BOSH release version
- `sha1`: The sha1 sum of the BOSH release tarball
- `remote_source`: The identifier of the BOSH Release tarball source specified in the Kilnfile where the tarball is stored
- `remote_path`: A source specific string to identify where the tarball is. This _may_ be a URL.

##### <a id='kilnfile-secrets'></a> Kilnfile Secrets

A Kilnfile (currently) specifies all the strings required to configure a BOSH Release tarball source.
This includes secrets.
While you _can_ just add the secrets to the Kilnfile, don't.
The Kilnfiles go through an initial templating step before being parsed.
Please don't use this for anything but secret injection.
Most kiln commands recieve a `--variable` or a `--variables-file` flag.

To use the `--variable` flag run something like this:

`kiln fetch --variable=fruit=banana`

In your Kilnfile use the fruit variable like this.

```
release_source:
- some_field: $(variable "banana")
- some_field: "$(variable "banana")"
- some_field: '$(variable "banana")'
```

##### <a id='release-source-id'></a> Explicit Release Source ID

Please set an explicit ID for each release source.
Kiln has fall-back behavior to use other fields to identify a release source (like bucket for S3 or owner for GitHub...)
but this fall-back behavior can be hard to follow.
Just set `id` on all of your release sources and make mapping releases in the Kilnfile.lock to the release source in Kilnfile easier to follow.

##### <a id='path-templates'></a> Path Templates

While some BOSH release tarball sources use URLs as the `remote_path` in their release locks,
others (S3 and Artifactory) rely on a path template in their configuration.

The `path_template` is uses [`text/template`](https://pkg.go.dev/text/template) and is passed a value with the following type.

```go
package cargo

// PathTemplateData is passed to template.Execute along with the parsed Release Source path template.
// this type is not the real one in the source.
type PathTemplateData struct{
// Name is set to the BOSH Release name
Name string

// Version is set to the BOSH Release version
Version string

// Name is set to the Kilnfile.lock StemcellCriteria OS value
StemcellOS string

// Name is set to the Kilnfile.lock StemcellCriteria Version value
StemcellVersion string
}
```

Here are some example path templates (the rest of the release source config has been omitted).

```yaml
release_sources:
- path_template: "bla-bla/{{.Name}}/{{.Version}}/{{.Name}}-{{.Version}}-{{.StemcellOS}}-{{.StemcellVersion}}.tgz"
- path_template: "bla-bla/{{.Name}}/{{.Version}}/{{.Name}}-{{.Version}}.tgz"
- path_template: "bla-bla/{{.Name}}-{{.Version}}-{{.StemcellOS}}-{{.StemcellVersion}}.tgz"
```
Avoid using other Go text/template actions like `{{if pipeline}}` and the like.

#### <a id="release-source-boshio"></a> BOSH.io

Kiln can only download releases from BOSH.io and can not upload BOSH Releases to BOSH.io.

This release source has minimal configuration.
Just add it to your `release_sources` and you can get releases from [BOSH.io](https://bosh.io/releases/).

The value of `remote_path` in the BOSH release tarball lock is a URL.

#### <a id='release-source-github'></a> GitHub Release Artifacts

Kiln can only download releases from GitHub Releases and can not upload BOSH Releases to BOSH.io.
Expand All @@ -264,26 +354,48 @@ export GITHUB_TOKEN="$(gh auth status --show-token 2>&1 | grep 'Token:' | awk '{
kiln fetch --variable="github_token=${GITHUB_TOKEN}"
```
The value of `remote_path` in the BOSH release tarball lock is a URL.
#### <a id='release-source-artifactory'></a> Build Artifactory
Kiln can fetch and upload releases to/from Build Artifactory.
_TODO: write this section._
The release source specification should look like this:
```yaml
release_sources:
- type: "artifactory"
repo: "some-repository"
username: "some-username"
password: "some-password"
path_template: "some-path-template/tarball.tgz"
```

Note `repo` is not a GitHub repository. It is an Artifactory thing.

Please see [Path Templates](#path-templates). The value of `remote_path` in the BOSH release tarball lock is part of the path needed to construct a URL to download the release.

#### <a id='release-source-s3'></a> AWS S3

Kiln can fetch and upload releases to/from AWS S3.

_TODO: write this section._
```yaml
release_sources:
- type: "artifactory"
bucket: "some-bucket"
region: "some-region"
access_key_id: "some-access-key-id"
secret_access_key: "some-secret-access-key"
path_template: "some-path-template/tarball.tgz"
```
Please see [Path Templates](#path-templates). The value of `remote_path` in the BOSH release tarball lock is part of the path needed to make the S3 object name.

#### <a id='release-source-directory'></a> Local tarballs

`kiln bake` adds the BOSH release tarballs in the releases directory to the tile reguardless of if they match the Kilnfile.lock
(There may be a regression in this behavior but that change may be fixed/reverted.)
`kiln bake` adds the BOSH release tarballs in the releases directory to the tile reguardless of if they match the Kilnfile.lock.
Building a tile with arbitrary releases in the tarball is not secure; this behavior should only be used for development not for building production tiles.

_TODO: test out this behavior (a slack comment mentioned it is not working)._

#### Default credentials file

You can add a default credentials file to `~/.kiln/credentials.yml` so you don't need to pass variables flags everywhere.
Expand Down Expand Up @@ -345,25 +457,40 @@ PPE uses this command for TAS/IST/TASW prior to new major versions.

## Stemcell Version Management

_TODO_

`kiln find-stemcell-version` and `kiln update-stemcell`

Find the latest stemcell releases on PivNet. They behave similarly to the bosh release commands above.

_If I remember right, the find-stemcell-version command has a bug where the stemcell criteria version in the Kilnfile is not respected and the result of the command is always the latest version._

## Tile Release Note Generation

_TODO_
If you GitHub BOSH Release tarball sources,
you can generate release notes for your tile that contain release notes for each BOSH release.

`kiln release-notes`
This feature requires you to have a previous tile release that used a Kilnfile.lock to specify the BOSH releases packaged.
You pass two references
- the Git Reference or SHA of the commit of the source used to generate the previously published tile
- Git Reference or SHA of the commit of the source used to generate the **next** tile

## TanzuNet Release Publication
While you can override the template and the regular expression used to make these notes.
They are quite hard to craft.
I recommend you use the defaults.

_TODO? we might deprecate this functionality_
`kiln release-notes --update-docs=path-to-release-notes-file/notes.md "${PREVIOUS_RELEASE_SHA}" "${NEXT_RELEASE_SHA}"`

`kiln publish`
If you omit `--update-docs` the notes will be written to standard out.

## PivNet Release Publication

`kiln publish` does not in-fact publish a tile.
It changes some of the configuration on a previously created PivNet release.
While we use it for TAS, it is not ready/intended to be used by other tiles quite yet.

## Importing Go Source Code [![Go Reference](https://pkg.go.dev/badge/github.com/pivotal-cf/kiln.svg)](https://pkg.go.dev/github.com/pivotal-cf/kiln/pkg).

**Note the Kiln repository is pre-1.0.0. While we _try_ to maintain backwards compatablility with the commands. The package API is subject to change without notice.**

See the [Hello Tile manifest test](https://github.com/pivotal/hello-tile/blob/main/test/manifest/manifest_test.go) to see how to use this in tests.
Follow the conventions you see in hello-tile, and you should be able to run `kiln test`.
The github.com/pivotal-cf/kiln/pkg/proofing/upgrade package can help you detect changes that would require foundation provider/operator) intervention.
Expand Down

0 comments on commit 7105a85

Please sign in to comment.