Add instructions for running Drupal VM inside Docker

[geerlingguy]

Issue Type

• Feature Idea

Your OS

• macOS (10.12.3)

Summary

While doing some debugging for a 4.2.2 release (see #1200), I was just thinking to myself... we already have test rigs set up for Travis CI that builds and runs Drupal VM inside Docker.

And someday there may be an official set of Docker Images for Drupal VM (though maintaining a similar architecture to the current incarnation of Drupal VM)... but in the interim, we could at least provide some instructions for how you could run Drupal VM on top of Docker, bypassing Vagrant entirely.

This would be fully experimental for now, and there are a bunch of conveniences Vagrant adds that Docker can't at this point (at least not without an insane amount of extra spaghetti work), but it's doable, and in some cases it's faster than Vagrant-based builds.

[geerlingguy]

Scenario 1: Just building a local Docker container to replicate what our tests do:

1. docker run --detach --add-host $(HOSTNAME):127.0.0.1 --volume=$(pw d):/var/www/drupalvm:rw --privileged --volume=/sys/fs/cgroup:/sys/fs/cgroup:ro geerlingguy/docker-centos7-ansible:latest "/usr/lib/systemd/systemd"
2. docker exec [container-id] mkdir -p /var/www/drupalvm/drupal
3. docker exec [container-id] env TERM=xterm ansible-playbook /var/www/drupalvm/ tests/test-setup.yml
4. docker exec [container-id] env TERM=xterm ansible-playbook /var/www/drupalvm/ provisioning/playbook.yml

[thom8]

@geerlingguy have found this incredibly very useful for CI - https://hub.docker.com/r/beet/box/

Can run docker run -v $(pwd):/var/beetbox -p 80:80 beet/box in a project dir to quickly test stuff out, as it's a full clone of the preprovisioned base box.

[thom8]

Also converted tests to run with docker & robo -- https://github.com/beetboxvm/beetbox/blob/master/RoboFile.php

loving robo :)

& composer scripts -- https://github.com/beetboxvm/beetbox/blob/master/composer.json#L29

composer test

[oxyc]

I would love it if I could run the tests locally with just a simple composer test.

[thom8]

@oxyc all my projects now have a composer test which is usually just an alias to a set of test suites for that particular project.

Makes it easy to remember how to run tests, like composer polymorphism.

[geerlingguy]

++ to all the above; right now I manually run a few docker commands when I want to run tests on a given role/project. I'd be happy to wrap things a little better to get the tests runnable with one command.

[oxyc]

Two thoughts:

1. One minor annoyance is that we currently allow config files and pre/post provision scripts to reside anywhere within the project and this makes it a requirement to sync both the codebase to /var/www/drupalvm and the entire project tree to /vagrant (or something similar in Docker's case). Would be nice if we could remove the /vagrant dependency by looking up the config files, the pre/post provision scripts and then copy them to the remote together with the playbook and the roles. With Vagrant the current setup doesn't bother me as I have /vagrant as an NFS mount, but as a macos user with docker I guess I would use rsync which would transfer the sites/default/files directory twice.
   Furthermore, users can also override template files for some roles, and if the files I mentioned before would be somewhat easily solvable by a few tasks that normalize the locations on the remote, this one makes it pretty much impossible to solve unless we somehow enforce a config/ directory.

2. I can live without most of Vagrant's convenience features but being able to configure the synced folders without hacking a Dockerfile would be my top priority. Is docker-compose.yml the solution here?

[thom8]

@oxyc could having a .drupalvm directory work? along with a config playbook

This means we can assume the config location -- https://github.com/beetboxvm/beetbox/blob/master/Dockerfile

[oxyc]

Yes that would work but would remove the convenience of being able to point pre/post provision scripts straight into to the examples directory (or another composer dependency).

Edit: Not sure if we should prioritize Vagrant and just live with the annoyance, or make docker more of a first class citizen.

[thom8]

This was one of the primary reasons behind moving to composer only support, we can now also assume that the project directory is at [vendor directory]/beet/box which would be available in the sync directory/volume.

Therefore, the config can be pulled directly from either location.
Still WIP on the composer side of things

[geerlingguy]

(And to reply to the question earlier, yes, I think it would be easiest to do what you want to do via a docker-compose.yml file rather than verbose commands.

Will either of you be present at DrupalCon Baltimore this week? Would be awesome to meet up!

[oxyc]

Will either of you be present at DrupalCon Baltimore this week? Would be awesome to meet up!

Sadly no, unfortunately it's a bit out of my budget range.

[thom8]

@geerlingguy unfortunately not this time, hoping to make it to Vienna this year.

[geerlingguy]

Picking this up this morning.

[geerlingguy]

I'm adding some tests to Drupal VM's composer.json, and it is surprisingly quite simple. For now, they're inline in composer.json, and I might stick with that for the time being... at some point, it would be nice to find a clean way to make comprehensive tests runnable between Travis and composer test.

Also, because the volume is slow on Mac, I'm getting this error:

[Symfony\Component\Process\Exception\ProcessTimedOutException]                                                       
  The process "docker exec drupal-vm-test env TERM=xterm ANSIBLE_FORCE_COLOR=true ansible-playbook /var/www/drupalvm/  
  provisioning/playbook.yml" exceeded the timeout of 300 seconds.

So I'll set COMPOSER_PROCESS_TIMEOUT=1800, and also use Docker's Edge channel build for Mac to use cached so disk reads are somewhat faster (https://docs.docker.com/docker-for-mac/install/#download-docker-for-mac).

[geerlingguy]

And thinking further, maybe it would be good to abstract the tests all into a script/scripts that are accessed through Composer, then we can switch .travis.yml over to running composer install and composer test ?

[thom8]

Recommend robo for docker based stuff -- http://robo.li/tasks/Docker/

[geerlingguy]

Now I'm hitting: geerlingguy/ansible-role-mysql#57

TASK [geerlingguy.mysql : Disallow root login remotely] ************************
failed: [localhost] (item=DELETE FROM mysql.user WHERE User='root' AND Host NOT IN ('localhost', '127.0.0.1', '::1')) => {"changed": false, "cmd": ["mysql", "-NBe", "DELETE FROM mysql.user WHERE User='root' AND Host NOT IN ('localhost', '127.0.0.1', '::1')"], "delta": "0:00:00.060785", "end": "2017-04-28 15:02:47.804283", "failed": true, "item": "DELETE FROM mysql.user WHERE User='root' AND Host NOT IN ('localhost', '127.0.0.1', '::1')", "rc": 1, "start": "2017-04-28 15:02:47.743498", "stderr": "ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)", "stderr_lines": ["ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)"], "stdout": "", "stdout_lines": []}

[thom8]

Yep, .mysql not running during build

[thom8]

https://github.com/beetboxvm/beetbox/blob/master/provisioning/ansible/roles/beetbox-docker/tasks/main.yml

Requirements for docker on 14.04

[thom8]

@geerlingguy sorry was on the train... let me know if you need any help.

This is essentially building DVM on 14.04 -- https://hub.docker.com/r/beet/box/~/dockerfile/

[geerlingguy]

@thom8 - Oops, I meant to try Ubuntu 16.04 (for a POC) instead of 14.04, because yeah, 14.04 is kinda annoying :P

See also: geerlingguy/ansible-role-mysql#84 (comment)

[thom8]

So looking forward to dropping all the 14.04 hacks...

[geerlingguy]

So, so far the quick and dirty way is to do the following:

Build a Docker container using the configuration in your Drupal VM instance:

TODO (this will go into a docker-up command, I think).

Commit the built container as a new image and save that image:

docker commit dvm-test drupal-vm/test
docker save drupal-vm/test > ~/Downloads/drupal-vm-test.tar

(After this, you can safely remove the built image with docker rmi dvm-test.)

Load the saved image:

docker load < ~/Downloads/drupal-vm-test.tar

(On the same computer or on another.)

Run a new container using the loaded image:

docker run --name=dvm-test -d --add-host "$HOSTNAME  drupalvm:127.0.0.1" -v $PWD:/var/www/drupalvm/:rw,cached --privileged -p 8888:80 drupal-vm/test:latest /lib/systemd/systemd

Win!

Visit http://localhost:8888/

[geerlingguy]

Working on the composer script now.

[geerlingguy]

Checking now to see if I can wrap all the docker testing stuff inside composer run-tests, which calls a shell script (this will make it a bit easier to build the tests, since it calls a bash script... it also means we can run tests locally!).

Also, it's more easy/doable to use something like Robo this way (but I'm fine with Bash, at least for now... if it works, it works.)

[geerlingguy]

So, that's nice. I'm going to merge that work, then second half of this issue will be making sure you can use Drupal VM to build yourself a nice Docker image (it seems with Ubuntu 16.04 the image ends up being around 1.2 GB with the defaults; I've yet to test Debian 8).

[geerlingguy]

With the changes above, you can now run the exact same tests that are run on Travis CI using the same command: composer run-tests

See new section in README: https://github.com/geerlingguy/drupal-vm#tests

We'll see how far I can get in the airport tomorrow :)

Also, note that some of this work may affect our approach to how we want Drupal VM to be a Composer plugin over in #1256

[thom8]

Nice, I guess robo makes it more platform agnostic but it's probably becoming less of an issue with WSL.

[geerlingguy]

A few outstanding issues:

• Making the database persist (maybe just mount a volume for it?)
• Create docker-bake script to bring up a Docker container using current Drupal VM settings.
• Make docker-bake script work with variables like vagrant_box, vagrant_hostname, vagrant_machine_name, and vagrant_ip (at a minimum)—or create alternatives?
• Create docker-save-image script to save a Docker container (from docker-up) as an image archive (.tar).
• Create docker-load-image script to load a Docker image from an archive (.tar)
• Create example docker-compose.yml file for baked/loaded Docker image.
• Doing some performance tests (with cached for the volume on my Mac... it actually feels faster after first page load than VirtualBox!)
• Add complete documentation for all of the above.

[geerlingguy]

Apparently the container has some semblence of persistence (between stop/start) already, so I can docker stop and docker start the container and the database persists. I didn't think that was done automatically, but it seems like it is. (I could be more explicit about it... but I'd rather leave that up to the user (e.g. if you want to mount a volume for the MySQL data dir, you can add that to your own compose... see example).

[geerlingguy]

Also note that this experimental branch is likely far from what we'll eventually get to in the 'primarily Vagrant but possibly Docker' stage of Drupal VM's existence... right now the instructions involve committing the database inside the container (makes for a larger image, and can cause annoying extra work in rebuilding/syncing/etc.).

But as with all things Docker there are some tradeoffs here and there (e.g. use a separate volume for MySQL dir... mount a volume... etc.).

Right now I'm trying to see what would be the simplest way to implement networking so you could use the same IP as vagrant_ip, basically, exposing all the ports defined in the list of firewall_allowed_tcp_ports on just that IP.

There will most definitely be a few manual steps involved in the generation and use of Docker images:

• Setting a few docker-specific variables.
• Running a few commands to get everything to happen (mostly automated at least).
• Adjusting the hosts file manually when using Docker (at least initially).

[geerlingguy]

PROGRESS!

Special thanks to moby/moby#22753 (comment) — I was going insane trying to figure out why I couldn't get bridged networking working. Every. Single. Guide. And. Docs. Page assumes you are running on Linux or Windows—there's a small caveat buried in the Docker for Mac pages where it talks about docker0 not being available.

Between this and the lack of write caching in the filesystem layer (though read caching is here, and helps Drupal tremendously), it's still highly annoying to do 'native' Docker development on a Mac.

For now, Docker support in Drupal VM will be experimental. But at least I have a path forward, woot!

[geerlingguy]

Left a follow-up comment here: moby/moby#22753 (comment)

[geerlingguy]

Using the IP trickery in the initial bake.sh script too... if it works, and tests pass, I'm going to commit it and just leave things 'experimental' for now.

But there's a good chance I'm going to switch many if not most of my personal projects to using Docker, just because it's so darn fast (well, with anything that doesn't require writing a lot of code). And technically I can tweak Drupal VM settings and reprovision if I need to.

[geerlingguy]

Merged what I had worked on so far. It works well enough; definitely room for improvement.