Refactor project for 0.9.0 release

It's a big one, probably incomplete changes list:

* Raise Bash min version to 4.2, add jq as host dependency for Docker usage
* Add a Kubler bash-completion script
* Prepare Kubler for installation as system app
* Add support for user managed custom commands/engines or overrides
* Prepare for split of example images into a separate git repo
* Minor refactor of the build process, -i now builds missing parents for example
* Add configure_builder hook, replaces configure_bob. Both will be supported
  for a while though
* The portage container is no longer updated via git, the amount of
  upstream changes is too large, snapshots all the way now per default
* Run a Docker health-check or custom image test script as part of the build
* Review/improve all ui output for consistency, also now in glorius 8bit color!
* Add compact output mode, if enabled almost all cmd output is logged to file logs
* Add dep-graph command to visualize image dependencies in dot/ascii/png
* Add a few clean command options to remove built and dangling images
* Improve 'new' command user handling to be less confusing
* Lots of minor improvements and some fixes for edge cases

See also #154

Author: edannenberg

.travis.yml

@@ -9,5 +9,5 @@ before_install:
   - docker pull koalaman/shellcheck
 
 script:
- - docker run -w /scripts -v $(pwd):/scripts koalaman/shellcheck -x
-   kubler.sh lib/*.sh lib/engine/docker.sh lib/cmd/*.sh lib/bob-core/*.sh lib/argbash/argbash-refresh.sh
+ - docker run --rm -w /scripts -v $(pwd):/scripts koalaman/shellcheck -x
+   kubler.sh lib/*.sh lib/engine/docker.sh lib/cmd/*.sh lib/bob-core/*.sh lib/cmd/argbash/argbash-refresh.sh

README.md

@@ -10,174 +10,139 @@ and the German name Schmidt, the cooper trade is also the origin of German names
 > There is still demand for high-quality ~~wooden barrels~~ containers, and it is thought that the
 highest-quality ~~barrels~~ containers are those hand-made by professional ~~coopers~~ kublers.
 
-At the core Kubler is just a simple ~~craftsman~~ bash script that, well, builds things.. and things that
-can depend on other things. It does'nt really care all too much about the details as long as it gets 
-to build. So what, some ~~people~~ scripts just like to build things. Don't judge.
+## Why Should You Care?
 
-What kind of things? In theory your imagination is the limit, but we provide batteries for building
-[Docker][] images, with [acbuild][] (read: rkt and OCI) support in the works. PR are always welcome. ;)  
+Perhaps:
 
-Due to it's unrivaled flexibility [Gentoo][] is used under the hood as build container base, 
-however the final images hold just the runtime dependencies for selected software packages, resulting
-in very slim images. To achieve this a 2 phase build process is employed, essentially the often requested, but
-still missing, Docker feature for [nested](https://github.com/docker/docker/issues/7115) image builds.
+1. You love Docker but are annoyed by some of the restrictions of it's `build` command that keep
+   getting into your way. Wouldn't it be nice if you could `build` your images with all `docker run`
+   args, like `-v`, at your disposal?
+2. You are a SysAdmin or DevOps engineer who seeks complete governance for the contents of their
+   Docker images, with full control of the update cycle and the ability to track software version
+   changes across the board from a centralized vcs repository.
+3. You need to manage a **lot** of Docker base/service images in a sane way and want peace of mind
+   with automated post-build tests.
+4. You are a Gentoo user that wants to build slim Gentoo based images without having to wrestle
+   with CrossDev.
+5. You are looking for an interactive OS host agnostic Gentoo playground or a portable ebuild
+   development environment.
+6. You want to create custom root file systems, possibly for different cpu architectures, in a safe
+   and repeatable manner.
 
-## Goals
+## Cool. So What Exactly Is A Container Image Meta Builder?
 
-* Central, organization-wide management of base images
-* Containers should only contain the bare minimum to run
-  * Separate build and runtime dependencies
-  * Only deploy runtime dependencies
-* Maximum flexibility while assembling the root file system, but with minimal effort
-* Keep things maintainable as the stack grows
+While Kubler was designed primarily for building and managing container images it doesn't
+particularly care about the way those images are built. At the core it's just a glorified directory
+crawler, with a simple dependency mechanism, that fires a command on a selected image or namespace
+dependency graph.
 
-## Status
+The actual build logic is abstracted away into pluggable engines that may orchestrate other tools,
+like Docker, to create the final image, or whatever the selected namespace's configured engine
+produces.
 
-* Stable for a while now and used in production
-* Monthly update cycle for all reference images
+Kubler is extendable, users may provide their own commands and/or build engines in a maintainable
+way. As both are just plain old Bash scripts this is usually a simple* and straight forward process
+with almost no limitations.
 
-## Features
+`{ font-size: 2px; }` * Additional rates of blood, sweat and tears may apply when implementing a new engine
 
-* Decoupled build logic
-* Maintain multiple image stacks with differing build engines
-* Generic [root-fs][bob-core] build script to quickly bootstrap a [Gentoo][] based build container
-* Utilizes Gentoo's [binary package][] features for quick rebuilds
-* Simple hook system allows for full control of the build process while retaining modularity
-* Generic image and builder dependencies that can be utilized by build engines
-* Automated image [documentation][nginx-packages] and history when using a CVS
+## Requirements
 
-### Docker Features
+#### Kubler
 
-* Essentially enables [nested](https://github.com/docker/docker/issues/7115) docker builds
-* Everything happens in docker containers except for some bash glue on the build host
-* Glibc, Musl and Uclibc based build containers, each with complete toolchain out of the box
-* Tiny static busybox-musl root image (~1.2mb), FROM scratch is fine too
-* Shared layer support for final images, images are not squashed and can depend on other images
-* [s6][] instead of [OpenRC][] as default supervisor (small footprint (<1mb) and proper docker SIGTERM handling),
-optional of course
-* Reference images are available on [docker hub][kubler-docker]
-* Push built image stack(s) to a public or private docker registry
+* Bash version 4.2+, using 4.4+ is highly recommended due to bugs in previous versions.
 
-### Requirements
+Optional:
+
+* GPG for download verification
+
+#### Docker Build Engine
 
-* Bash 4.x
 * Working Docker setup
+* GIT
+* jq to parse Docker json output
 
-Optional:
+## Installation
 
-* GPG for download verification
+#### On Gentoo
 
-Kubler has been tested on Gentoo, CoreOS and macOS. It should run on all Linux distributions.
+An ebuild can be found at https://github.com/edannenberg/kubler-overlay/
 
-## How much do I save?
+Add the overlay and install as usual:
 
-* Quite a bit, the Nginx Docker image, for example, clocks in at ~17MB, compared to >1GB for a full Gentoo version
-or ~300MB for a similiar Ubuntu version
+    emerge -av kubler
 
-## Quick Start
+#### Manual Installation
 
-    $ git clone https://github.com/edannenberg/kubler.git
+Kubler has been tested on Gentoo, CoreOS and macOS. It should run on all Linux distributions.
 
-Kubler needs a `working-dir` to operate from, much like `git` needs to be called from inside a git repo for most of its
-functionality. You may also call Kubler from any sub directory and it will detect the proper path. The Kubler git repo 
-comes with an example image stack, let's build a provided `glibc` image:
+1. Clone the repo or download/extract the release archive to a location of your choice, i.e.
 
-    $ cd kubler/
-    $ ./kubler.sh build kubler/glibc
+    $ cd ~/tools/
+    $ curl -L https://github.com/edannenberg/kubler/archive/master.tar.gz | tar xz
 
-This will build a `kubler/busybox` and `kubler/glibc` image. You also get a glibc and musl based build container for
-free, which you can utilize for your own images.
+2. Add `kubler.sh` to your path
 
-* You may add `kubler.sh` to your `PATH`, one-liner: `export PATH="${PATH}:/path/to/kubler/bin"` 
-* If you don't have GPG available use `build -s ..` to skip verification of downloaded files (SHA512 is still checked)
-* The directories in `./dock/kubler/images/` contain image specific documentation
+If you are unsure add the following at the end of your `~/.bashrc` file, don't forget to adjust the
+path for each line accordingly:
 
-Note: If you get a 404 error on downloading a Gentoo stage3 tar ball try running `kubler update` to resolve the issue.
-The Gentoo servers only keep those files for a few weeks.
+    export PATH="${PATH}:/path/to/kubler/bin"
+    # optional but highly recommended, adds bash completion support for all kubler commands
+    source /path/to/kubler/lib/kubler-completion.bash
 
-The first run will take quite a bit of time, don't worry, once the build containers and binary package cache
-are seeded future runs will be much faster.
+Note: You will need to open a new shell for this to take effect, if this fails on a Linux SystemD
+host re-logging might be required instead.
 
-## Creating a new namespace
+#### Uninstall
 
-Images are kept in a `namespace` directory in `--working-dir`. You may have any number of namespaces. A helper is
-provided to take care of the boiler plate for you: 
+1. Remove any build artifacts and Docker images created by Kubler:
 
-```
- $ cd kubler/
- $ ./kubler.sh new namespace testing
- 
- --> Who maintains the new namespace?
- Name (John Doe): My Name
- EMail ([email protected]): [email protected]
- --> What type of images would you like to build?
- Engine (docker):
+    $ kubler clean -N
 
- --> Successfully added "testing" namespace at ./dock/testing
+2. Delete the two entries from `~/.bashrc` you possibly added during manual installation
 
- $ tree dock/testing/
- dock/testing/
- |-- .gitignore
- |-- kubler.conf
- .-- README.md
-```
+3. Delete any namespace dirs and configured `KUBLER_DATA_DIR` (default is `~/.kubler/`) you had in
+   use, this may require su permissions.
 
-You are now ready to work on your shiny new image stack.
+## Quick Start
 
-## Adding Docker images
+#### The Basics
 
-Let's create a [Figlet](http://www.figlet.org/) test image in our new namespace. If you chose a more
-sensible namespace name above replace `testing` accordingly:
+To get a quick overview/reminder of available commands/options run:
 
-```
- $ ./kubler.sh new image testing/figlet
+    $ kubler --help
 
- --> Extend an existing image? Fully qualified image id (i.e. kubler/busybox) if yes or scratch
- Parent Image (scratch): kubler/glibc
+To view details for specific command:
 
- --> Successfully added testing/figlet image at ./dock/testing/images/figlet
-```
+    $ kubler build -h
 
-We used `kubler/glibc` as parent image, or what you probably know as `FROM` in your `Dockerfiles`.
-The namespace now looks like this:
- 
-```
- $ tree dock/testing/
- dock/testing/
- |-- kubler.conf
- |-- images
- |   .-- figlet
- |       |-- build.conf
- |       |-- build.sh
- |       |-- Dockerfile.template
- |       .-- README.md
- .-- README.md
-```
+Almost all of Kubler's commands will need to be run from a `--working-dir`, if the option is
+omitted the current working dir of the executing shell is used. It functions much like Git in that
+regard, executing any Kubler command from a sub directory of a valid working dir will also work as
+expected.
 
-Edit the new image's build script located at `./dock/testing/images/figlet/build.sh` and add `app-misc/figlet` to the
-`_packages` variable:
+A `--working-dir` is considered valid if it has a `kubler.conf` file and either an `images/` dir or
+one ore more namespace dirs, which are just a collection of images.
 
-```
-_packages="app-misc/figlet"
-```
+#### Setup A New Namespace
 
-When it's time to build this will instruct the build container in the *first build phase* to install the given package(s)
-from Gentoo's package tree at an empty directory. It's content is then exported to the host as a `rootfs.tar` file.
-In the *second build phase* a normal Docker build is started and the `rootfs.tar` file is added to the final image.
+First switch to a directory where you would like to store your Kubler managed images or namespaces:
 
-See the 'how does it work' section below for more details on the build process. Also make sure to read the comments
-in `build.sh`. But let's build the darn thing already:
+    $ cd ~/workspace
 
-```
- $ ./kubler.sh build testing
-```
+Then use the `new` command to take care of the boiler plate for you, choose `single` as namespace
+type when asked:
 
-Once that finishes we are ready to take the image for a test drive:
+    $ kubler new namespace mytest
+    $ cd mytest
 
-```
- $ docker images | grep /figlet
- $ docker run -it --rm kubler/figlet figlet kubler sends his regards
-```
+### Hello Image
+
+To create a new image in the current working dir:
+
+    $ kubler new image mytest/figlet
+
+#TODO: finish docs
 
 Some useful options for while working on an image:
 
@@ -245,6 +210,18 @@ this preserves exact build state
 Build container names generally start with `*/bob`, when a new build container state is committed the current image
 name gets appended. For example `kubler/bob-openssl` refers to the container used to build the `kubler/openssl` image.
 
+## Other Resources
+
+* An excellent blog post, written by [@berney][], can be found at https://www.elttam.com.au/blog/kubler/
+
+## Discord Community
+
+For questions or chatting with other users you may join our Discord server at:
+
+https://discord.gg/rH9R7bc
+
+You just need a username, email verification with Discord is not required.
+
 ## Thanks
 
 [@wking][] for his [gentoo-docker][] repo which served as an excellent starting point

kubler.conf

@@ -1,33 +1,59 @@
-AUTHOR="${AUTHOR:-Erik Dannenberg <[email protected]>}"
-# Global tag (a.k.a version) for all images
-IMAGE_TAG="${IMAGE_TAG:-20181130}"
-# Cheeky example that always sets the current date:
-#IMAGE_TAG="${IMAGE_TAG:-$(date +%Y%m%d)}"
-
-# Portage snapshot date that is used to bootstrap the portage container
-PORTAGE_DATE="${PORTAGE_DATE:-latest}"
+# The first config file read by Kubler. System wide settings, users may override via KUBLER_DATA_DIR/kubler.conf or
+# current --working-dir.
+
+# Image version, ideally set via user config so it's shared by all --working-dirs of current user
+#IMAGE_TAG='20190123'
+# Default maintainer, override via namespace kubler.conf
+AUTHOR='Erik Dannenberg <[email protected]>'
+
+# Kubler's runtime data dir, needs to be writable by the current user
+#KUBLER_DATA_DIR="${HOME}/.kubler}"
+# Gentoo's stage3 and portage files download location
+#KUBLER_DOWNLOAD_DIR="${KUBLER_DATA_DIR}/downloads"
+# Gentoo's build related downloads, like source code, patches, etc
+#KUBLER_DISTFILES_DIR="${KUBLER_DATA_DIR}/distfiles"
+# Gentoo's binary package cache location
+#KUBLER_PACKAGES_DIR="${KUBLER_DATA_DIR}/packages"
+
+# Ouput related config
+# Disable compact output, effectively always passes -v to all commands
+#KUBLER_VERBOSE='false'
+# If true and compact output is enabled send output to log file instead of /dev/null
+#KUBLER_CMD_LOG='true'
+# Shall we ring the terminal bell on error?
+#KUBLER_BELL_ON_ERROR='true'
+# Disabling this only works when set as ENV before starting Kubler
+#KUBLER_COLORS='true'
+
+# Update the portage container via git. Not recommended as it can be quite slow due to the amount of upstream changes.
+#KUBLER_PORTAGE_GIT='false'
+# Not recommended unless you are building your image stack from scratch and with your own stage3 build containers
+#KUBLER_DISABLE_KUBLER_NS='false'
+# Effectively always enables -s for the build command
+#KUBLER_DISABLE_GPG='false'
+
+# Portage snapshot date that is used to bootstrap the portage container, 'latest' is highly recommended
+#PORTAGE_DATE='latest'
 # Download location for stage3 and Portage files, use whitespace to set multiple servers
 # You may visit https://www.gentoo.org/downloads/mirrors/ and pick a http or ftp url near your physical location
-MIRROR="${MIRROR:-http://distfiles.gentoo.org/}"
-
-DOWNLOAD_PATH="${DOWNLOAD_PATH:-${_KUBLER_DIR}/tmp/downloads}"
+MIRROR='http://distfiles.gentoo.org/'
 
 # You can also define these per namespace conf
-BUILD_ENGINE="${BUILD_ENGINE:-docker}"
-DEFAULT_BUILDER="${DEFAULT_BUILDER:-kubler/bob}"
+BUILD_ENGINE='docker'
+DEFAULT_BUILDER='kubler/bob'
 
 # Variables starting with BOB_ are exported as ENV to all build containers
 
 # init Portage's make.conf defaults
 
-BOB_GENTOO_MIRRORS="${BOB_GENTOO_MIRRORS:-${MIRROR}}"
+BOB_GENTOO_MIRRORS="${MIRROR}"
 
-BOB_FEATURES="${BOB_FEATURES:-parallel-fetch nodoc noinfo noman binpkg-multi-instance}"
-BOB_EMERGE_DEFAULT_OPTS="${BOB_EMERGE_DEFAULT_OPTS:--b -k --binpkg-respect-use=y}"
+BOB_FEATURES='-parallel-fetch nodoc noinfo noman binpkg-multi-instance'
+BOB_EMERGE_DEFAULT_OPTS='-b -k --binpkg-respect-use=y'
 
 # Timezone for build containers
-BOB_TIMEZONE="${BOB_TIMEZONE:-UTC}"
+BOB_TIMEZONE='UTC'
 
 # Options passed on to the make jobs launched from Portage
 # -jX = number of cpu cores used for compiling, rule of thumb: amount_of_cores+1, i.e. -j9
-BOB_MAKEOPTS="${BOB_MAKEOPTS:--j9}"
+BOB_MAKEOPTS='-j9'