Skip to content

Commit

Permalink
Refactor project for 0.9.0 release
Browse files Browse the repository at this point in the history 
It's a big one, probably incomplete changes list:

* Raise Bash min version to 4.2, add jq as host dependency for Docker usage
* Add a Kubler bash-completion script
* Prepare Kubler for installation as system app
* Add support for user managed custom commands/engines or overrides
* Prepare for split of example images into a separate git repo
* Minor refactor of the build process, -i now builds missing parents for example
* Add configure_builder hook, replaces configure_bob. Both will be supported
  for a while though
* The portage container is no longer updated via git, the amount of
  upstream changes is too large, snapshots all the way now per default
* Run a Docker health-check or custom image test script as part of the build
* Review/improve all ui output for consistency, also now in glorius 8bit color!
* Add compact output mode, if enabled almost all cmd output is logged to file logs
* Add dep-graph command to visualize image dependencies in dot/ascii/png
* Add a few clean command options to remove built and dangling images
* Improve 'new' command user handling to be less confusing
* Lots of minor improvements and some fixes for edge cases

See also #154
  • Loading branch information
@edannenberg
edannenberg committed Jan 25, 2019
1 parent 9927836 commit 7abf51a
Show file tree
Hide file tree
Showing 41 changed files with 2,452 additions and 1,052 deletions.
4 changes: 2 additions & 2 deletions .travis.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ before_install:
- docker pull koalaman/shellcheck

script:
- docker run -w /scripts -v $(pwd):/scripts koalaman/shellcheck -x
kubler.sh lib/*.sh lib/engine/docker.sh lib/cmd/*.sh lib/bob-core/*.sh lib/argbash/argbash-refresh.sh
- docker run --rm -w /scripts -v $(pwd):/scripts koalaman/shellcheck -x
kubler.sh lib/*.sh lib/engine/docker.sh lib/cmd/*.sh lib/bob-core/*.sh lib/cmd/argbash/argbash-refresh.sh
223 changes: 100 additions & 123 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,174 +10,139 @@ and the German name Schmidt, the cooper trade is also the origin of German names
> There is still demand for high-quality ~~wooden barrels~~ containers, and it is thought that the
highest-quality ~~barrels~~ containers are those hand-made by professional ~~coopers~~ kublers.

At the core Kubler is just a simple ~~craftsman~~ bash script that, well, builds things.. and things that
can depend on other things. It does'nt really care all too much about the details as long as it gets
to build. So what, some ~~people~~ scripts just like to build things. Don't judge.
## Why Should You Care?

What kind of things? In theory your imagination is the limit, but we provide batteries for building
[Docker][] images, with [acbuild][] (read: rkt and OCI) support in the works. PR are always welcome. ;)
Perhaps:

Due to it's unrivaled flexibility [Gentoo][] is used under the hood as build container base,
however the final images hold just the runtime dependencies for selected software packages, resulting
in very slim images. To achieve this a 2 phase build process is employed, essentially the often requested, but
still missing, Docker feature for [nested](https://github.com/docker/docker/issues/7115) image builds.
1. You love Docker but are annoyed by some of the restrictions of it's `build` command that keep
getting into your way. Wouldn't it be nice if you could `build` your images with all `docker run`
args, like `-v`, at your disposal?
2. You are a SysAdmin or DevOps engineer who seeks complete governance for the contents of their
Docker images, with full control of the update cycle and the ability to track software version
changes across the board from a centralized vcs repository.
3. You need to manage a **lot** of Docker base/service images in a sane way and want peace of mind
with automated post-build tests.
4. You are a Gentoo user that wants to build slim Gentoo based images without having to wrestle
with CrossDev.
5. You are looking for an interactive OS host agnostic Gentoo playground or a portable ebuild
development environment.
6. You want to create custom root file systems, possibly for different cpu architectures, in a safe
and repeatable manner.

## Goals
## Cool. So What Exactly Is A Container Image Meta Builder?

* Central, organization-wide management of base images
* Containers should only contain the bare minimum to run
* Separate build and runtime dependencies
* Only deploy runtime dependencies
* Maximum flexibility while assembling the root file system, but with minimal effort
* Keep things maintainable as the stack grows
While Kubler was designed primarily for building and managing container images it doesn't
particularly care about the way those images are built. At the core it's just a glorified directory
crawler, with a simple dependency mechanism, that fires a command on a selected image or namespace
dependency graph.

## Status
The actual build logic is abstracted away into pluggable engines that may orchestrate other tools,
like Docker, to create the final image, or whatever the selected namespace's configured engine
produces.

* Stable for a while now and used in production
* Monthly update cycle for all reference images
Kubler is extendable, users may provide their own commands and/or build engines in a maintainable
way. As both are just plain old Bash scripts this is usually a simple* and straight forward process
with almost no limitations.

## Features
`{ font-size: 2px; }` * Additional rates of blood, sweat and tears may apply when implementing a new engine

* Decoupled build logic
* Maintain multiple image stacks with differing build engines
* Generic [root-fs][bob-core] build script to quickly bootstrap a [Gentoo][] based build container
* Utilizes Gentoo's [binary package][] features for quick rebuilds
* Simple hook system allows for full control of the build process while retaining modularity
* Generic image and builder dependencies that can be utilized by build engines
* Automated image [documentation][nginx-packages] and history when using a CVS
## Requirements

### Docker Features
#### Kubler

* Essentially enables [nested](https://github.com/docker/docker/issues/7115) docker builds
* Everything happens in docker containers except for some bash glue on the build host
* Glibc, Musl and Uclibc based build containers, each with complete toolchain out of the box
* Tiny static busybox-musl root image (~1.2mb), FROM scratch is fine too
* Shared layer support for final images, images are not squashed and can depend on other images
* [s6][] instead of [OpenRC][] as default supervisor (small footprint (<1mb) and proper docker SIGTERM handling),
optional of course
* Reference images are available on [docker hub][kubler-docker]
* Push built image stack(s) to a public or private docker registry
* Bash version 4.2+, using 4.4+ is highly recommended due to bugs in previous versions.

### Requirements
Optional:

* GPG for download verification

#### Docker Build Engine

* Bash 4.x
* Working Docker setup
* GIT
* jq to parse Docker json output

Optional:
## Installation

* GPG for download verification
#### On Gentoo

Kubler has been tested on Gentoo, CoreOS and macOS. It should run on all Linux distributions.
An ebuild can be found at https://github.com/edannenberg/kubler-overlay/

## How much do I save?
Add the overlay and install as usual:

* Quite a bit, the Nginx Docker image, for example, clocks in at ~17MB, compared to >1GB for a full Gentoo version
or ~300MB for a similiar Ubuntu version
emerge -av kubler

## Quick Start
#### Manual Installation

$ git clone https://github.com/edannenberg/kubler.git
Kubler has been tested on Gentoo, CoreOS and macOS. It should run on all Linux distributions.

Kubler needs a `working-dir` to operate from, much like `git` needs to be called from inside a git repo for most of its
functionality. You may also call Kubler from any sub directory and it will detect the proper path. The Kubler git repo
comes with an example image stack, let's build a provided `glibc` image:
1. Clone the repo or download/extract the release archive to a location of your choice, i.e.

$ cd kubler/
$ ./kubler.sh build kubler/glibc
$ cd ~/tools/
$ curl -L https://github.com/edannenberg/kubler/archive/master.tar.gz | tar xz

This will build a `kubler/busybox` and `kubler/glibc` image. You also get a glibc and musl based build container for
free, which you can utilize for your own images.
2. Add `kubler.sh` to your path

* You may add `kubler.sh` to your `PATH`, one-liner: `export PATH="${PATH}:/path/to/kubler/bin"`
* If you don't have GPG available use `build -s ..` to skip verification of downloaded files (SHA512 is still checked)
* The directories in `./dock/kubler/images/` contain image specific documentation
If you are unsure add the following at the end of your `~/.bashrc` file, don't forget to adjust the
path for each line accordingly:

Note: If you get a 404 error on downloading a Gentoo stage3 tar ball try running `kubler update` to resolve the issue.
The Gentoo servers only keep those files for a few weeks.
export PATH="${PATH}:/path/to/kubler/bin"
# optional but highly recommended, adds bash completion support for all kubler commands
source /path/to/kubler/lib/kubler-completion.bash

The first run will take quite a bit of time, don't worry, once the build containers and binary package cache
are seeded future runs will be much faster.
Note: You will need to open a new shell for this to take effect, if this fails on a Linux SystemD
host re-logging might be required instead.

## Creating a new namespace
#### Uninstall

Images are kept in a `namespace` directory in `--working-dir`. You may have any number of namespaces. A helper is
provided to take care of the boiler plate for you:
1. Remove any build artifacts and Docker images created by Kubler:

```
$ cd kubler/
$ ./kubler.sh new namespace testing
--> Who maintains the new namespace?
Name (John Doe): My Name
EMail ([email protected]): [email protected]
--> What type of images would you like to build?
Engine (docker):
$ kubler clean -N

--> Successfully added "testing" namespace at ./dock/testing
2. Delete the two entries from `~/.bashrc` you possibly added during manual installation

$ tree dock/testing/
dock/testing/
|-- .gitignore
|-- kubler.conf
.-- README.md
```
3. Delete any namespace dirs and configured `KUBLER_DATA_DIR` (default is `~/.kubler/`) you had in
use, this may require su permissions.

You are now ready to work on your shiny new image stack.
## Quick Start

## Adding Docker images
#### The Basics

Let's create a [Figlet](http://www.figlet.org/) test image in our new namespace. If you chose a more
sensible namespace name above replace `testing` accordingly:
To get a quick overview/reminder of available commands/options run:

```
$ ./kubler.sh new image testing/figlet
$ kubler --help

--> Extend an existing image? Fully qualified image id (i.e. kubler/busybox) if yes or scratch
Parent Image (scratch): kubler/glibc
To view details for specific command:

--> Successfully added testing/figlet image at ./dock/testing/images/figlet
```
$ kubler build -h

We used `kubler/glibc` as parent image, or what you probably know as `FROM` in your `Dockerfiles`.
The namespace now looks like this:

```
$ tree dock/testing/
dock/testing/
|-- kubler.conf
|-- images
|   .-- figlet
|   |-- build.conf
|   |-- build.sh
|   |-- Dockerfile.template
|   .-- README.md
.-- README.md
```
Almost all of Kubler's commands will need to be run from a `--working-dir`, if the option is
omitted the current working dir of the executing shell is used. It functions much like Git in that
regard, executing any Kubler command from a sub directory of a valid working dir will also work as
expected.

Edit the new image's build script located at `./dock/testing/images/figlet/build.sh` and add `app-misc/figlet` to the
`_packages` variable:
A `--working-dir` is considered valid if it has a `kubler.conf` file and either an `images/` dir or
one ore more namespace dirs, which are just a collection of images.

```
_packages="app-misc/figlet"
```
#### Setup A New Namespace

When it's time to build this will instruct the build container in the *first build phase* to install the given package(s)
from Gentoo's package tree at an empty directory. It's content is then exported to the host as a `rootfs.tar` file.
In the *second build phase* a normal Docker build is started and the `rootfs.tar` file is added to the final image.
First switch to a directory where you would like to store your Kubler managed images or namespaces:

See the 'how does it work' section below for more details on the build process. Also make sure to read the comments
in `build.sh`. But let's build the darn thing already:
$ cd ~/workspace

```
$ ./kubler.sh build testing
```
Then use the `new` command to take care of the boiler plate for you, choose `single` as namespace
type when asked:

Once that finishes we are ready to take the image for a test drive:
$ kubler new namespace mytest
$ cd mytest

```
$ docker images | grep /figlet
$ docker run -it --rm kubler/figlet figlet kubler sends his regards
```
### Hello Image

To create a new image in the current working dir:

$ kubler new image mytest/figlet

#TODO: finish docs

Some useful options for while working on an image:

Expand Down Expand Up @@ -245,6 +210,18 @@ this preserves exact build state
Build container names generally start with `*/bob`, when a new build container state is committed the current image
name gets appended. For example `kubler/bob-openssl` refers to the container used to build the `kubler/openssl` image.

## Other Resources

* An excellent blog post, written by [@berney][], can be found at https://www.elttam.com.au/blog/kubler/

## Discord Community

For questions or chatting with other users you may join our Discord server at:

https://discord.gg/rH9R7bc

You just need a username, email verification with Discord is not required.

## Thanks

[@wking][] for his [gentoo-docker][] repo which served as an excellent starting point
Expand Down
62 changes: 44 additions & 18 deletions kubler.conf
View file
Original file line number Diff line number Diff line change
@@ -1,33 +1,59 @@
AUTHOR="${AUTHOR:-Erik Dannenberg <[email protected]>}"
# Global tag (a.k.a version) for all images
IMAGE_TAG="${IMAGE_TAG:-20181130}"
# Cheeky example that always sets the current date:
#IMAGE_TAG="${IMAGE_TAG:-$(date +%Y%m%d)}"

# Portage snapshot date that is used to bootstrap the portage container
PORTAGE_DATE="${PORTAGE_DATE:-latest}"
# The first config file read by Kubler. System wide settings, users may override via KUBLER_DATA_DIR/kubler.conf or
# current --working-dir.

# Image version, ideally set via user config so it's shared by all --working-dirs of current user
#IMAGE_TAG='20190123'
# Default maintainer, override via namespace kubler.conf
AUTHOR='Erik Dannenberg <[email protected]>'

# Kubler's runtime data dir, needs to be writable by the current user
#KUBLER_DATA_DIR="${HOME}/.kubler}"
# Gentoo's stage3 and portage files download location
#KUBLER_DOWNLOAD_DIR="${KUBLER_DATA_DIR}/downloads"
# Gentoo's build related downloads, like source code, patches, etc
#KUBLER_DISTFILES_DIR="${KUBLER_DATA_DIR}/distfiles"
# Gentoo's binary package cache location
#KUBLER_PACKAGES_DIR="${KUBLER_DATA_DIR}/packages"

# Ouput related config
# Disable compact output, effectively always passes -v to all commands
#KUBLER_VERBOSE='false'
# If true and compact output is enabled send output to log file instead of /dev/null
#KUBLER_CMD_LOG='true'
# Shall we ring the terminal bell on error?
#KUBLER_BELL_ON_ERROR='true'
# Disabling this only works when set as ENV before starting Kubler
#KUBLER_COLORS='true'

# Update the portage container via git. Not recommended as it can be quite slow due to the amount of upstream changes.
#KUBLER_PORTAGE_GIT='false'
# Not recommended unless you are building your image stack from scratch and with your own stage3 build containers
#KUBLER_DISABLE_KUBLER_NS='false'
# Effectively always enables -s for the build command
#KUBLER_DISABLE_GPG='false'

# Portage snapshot date that is used to bootstrap the portage container, 'latest' is highly recommended
#PORTAGE_DATE='latest'
# Download location for stage3 and Portage files, use whitespace to set multiple servers
# You may visit https://www.gentoo.org/downloads/mirrors/ and pick a http or ftp url near your physical location
MIRROR="${MIRROR:-http://distfiles.gentoo.org/}"

DOWNLOAD_PATH="${DOWNLOAD_PATH:-${_KUBLER_DIR}/tmp/downloads}"
MIRROR='http://distfiles.gentoo.org/'

# You can also define these per namespace conf
BUILD_ENGINE="${BUILD_ENGINE:-docker}"
DEFAULT_BUILDER="${DEFAULT_BUILDER:-kubler/bob}"
BUILD_ENGINE='docker'
DEFAULT_BUILDER='kubler/bob'

# Variables starting with BOB_ are exported as ENV to all build containers

# init Portage's make.conf defaults

BOB_GENTOO_MIRRORS="${BOB_GENTOO_MIRRORS:-${MIRROR}}"
BOB_GENTOO_MIRRORS="${MIRROR}"

BOB_FEATURES="${BOB_FEATURES:-parallel-fetch nodoc noinfo noman binpkg-multi-instance}"
BOB_EMERGE_DEFAULT_OPTS="${BOB_EMERGE_DEFAULT_OPTS:--b -k --binpkg-respect-use=y}"
BOB_FEATURES='-parallel-fetch nodoc noinfo noman binpkg-multi-instance'
BOB_EMERGE_DEFAULT_OPTS='-b -k --binpkg-respect-use=y'

# Timezone for build containers
BOB_TIMEZONE="${BOB_TIMEZONE:-UTC}"
BOB_TIMEZONE='UTC'

# Options passed on to the make jobs launched from Portage
# -jX = number of cpu cores used for compiling, rule of thumb: amount_of_cores+1, i.e. -j9
BOB_MAKEOPTS="${BOB_MAKEOPTS:--j9}"
BOB_MAKEOPTS='-j9'
Loading

0 comments on commit 7abf51a

Please sign in to comment.