How to manage database and uploads migration for multiple sites on multiple servers

[davidyv]

Hi,

First of all I would like to thank you for this awesome project. I use it daily for all my projects.

I'm wondering how to go about manage database and uploads migration for a combination of multiple websites (not multisite) across multiple servers.

I have provisioned and deployed multiple sites on multiple servers with hosts/production file like this:

## Begin for database and uploads migration(https://github.com/valentinocossar/trellis-database-uploads-migration)
# production_host ansible_host=IP1
## production_host ansible_host=IP2
## production_host ansible_host=IP3
## production_host ansible_host=IP4

# [production]
# production_host

# [web]
# production_host

## End for database and uploads migration(https://github.com/valentinocossar/trellis-database-uploads-migration)

## Begin for provision and deploy a combination of multiple websites (not multisite) across multiple servers(https://discourse.roots.io/t/best-practice-for-multiple-sites-on-multiple-servers/5385)
[web]
production_site1 ansible_host=IP1
production_site2 ansible_host=IP2
production_site3 ansible_host=IP3
production_site4 ansible_host=IP4

[production:children]
web
## End for provision and deploy a combination of multiple websites (not multisite) across multiple servers(https://discourse.roots.io/t/best-practice-for- multiple- sites-on-multiple-servers/5385)

And the hosts_folder like this:

host_vars/production_site1/
host_vars/production_site2/
host_vars/production_site3/
host_vars/production_site4/

In those folders, I made copies of the vault.yml, wordpress_sites.yml and main.yml.

During provision and deploy, I comment out those lines for Hosts configuration of trellis-database-uploads-migration and uncomment those line for multiple sites on multiple servers provision and deploy, then:

ansible-playbook server.yml -e env=production --limit production_site1
ansible-playbook deploy.yml -i hosts/production -e "site=example.com env=production" --limit production_site1

During the database and uploads migration, I uncomment those lines for Hosts configuration of trellis-database-uploads-migration and comment out those line for multiple sites on multiple servers provision and deploy, then

ansible-playbook database-push.yml -e "env=production site=example.com"
ansible-playbook uploads.yml -e "env=production site=example.com mode=push"

But I’d rather like to find a more elegant and simpler workaround instead like use the --limit option.

[valentinocossar]

Hi, in our company we use a single Trellis project for each customer and, if necessary, we insert more sites in the wordpress_sites.yml file so that we have multiple sites on the same machine. We have never used a single Trellis installation to handle multiple servers and multiple sites (in addition to the classic staging and production servers).
Surely this is an interesting request, my question is: did you try to configure the hosts file this way and use --limit option also for pulling/pushing database and uploads?

production_host ansible_host=IP1
production_host ansible_host=IP2
production_host ansible_host=IP3
production_host ansible_host=IP4

[web]
production_site1
production_site2
production_site3
production_site4

[production]
production_site1
production_site2
production_site3
production_site4

I currently have no way of trying to make changes to the application, if you were willing to help me, I could integrate the tool to make it compatible with what you're asking me.

[davidyv]

Hello @valentinocossar! I get the following error when pulling with your hosts configure:

$ ansible-playbook database-pull.yml -e "env=production site=example.com" --limit production_site1

PLAY [Pull example.com database from production to development] **********************

TASK [Gathering Facts] **************************************************************************
System info:
  Ansible 2.3.1.0; Darwin
  Trellis at "Option to install WP-CLI packages"
---------------------------------------------------
SSH Error: data could not be sent to remote host "production_site1". Make
sure this host can be reached over ssh
fatal: [production_site1]: UNREACHABLE! => {"changed": false, "unreachable": true}
	to retry, use: --limit @/Users/david/projects/wordpress/sites/example.com/trellis/database-pull.retry

PLAY RECAP **************************************************************************************
production_site1       : ok=0    changed=0    unreachable=1    failed=0

Then I changed the hosts configure to:

production_host ansible_host=IP1
production_host ansible_host=IP2
production_host ansible_host=IP3
production_host ansible_host=IP4

[web]
production_site1 ansible_host=IP1
production_site2 ansible_host=IP2
production_site3 ansible_host=IP3
production_site4 ansible_host=IP4

[production]
production_site1 ansible_host=IP1
production_site2 ansible_host=IP2
production_site3 ansible_host=IP3
production_site4 ansible_host=IP4

or:

production_host ansible_host=IP1
production_host ansible_host=IP2
production_host ansible_host=IP3
production_host ansible_host=IP4

[web]
production_site1 ansible_host=IP1
production_site2 ansible_host=IP2
production_site3 ansible_host=IP3
production_site4 ansible_host=IP4

[production:children]
web

I get the following error when pulling:

$ ansible-playbook database-pull.yml -e "env=production site=example.com" --limit production_site1

PLAY [Pull example.com database from production to development] **********************

TASK [Gathering Facts] **************************************************************************
ok: [production_site1]

TASK [Abort if environment variable is equal to development] ************************************
skipping: [production_site1]

TASK [Check if example.com folder exists] ********************************************
ok: [production_site1 -> 192.168.50.5]

TASK [Abort if example.com folder doesn't exists] ************************************
skipping: [production_site1]

TASK [Create database_backup directory if it doesn't exist] *************************************
ok: [production_site1 -> 192.168.50.5]

TASK [Create database dump on production] *******************************************************
changed: [production_site1]

TASK [Pull database dump from production to development] ****************************************
changed: [production_site1]

TASK [Delete database dump from production] *****************************************************
changed: [production_site1]

TASK [Export development database before importing dump (backup)] *******************************
changed: [production_site1 -> 192.168.50.5]

TASK [Import database dump on development] ******************************************************
changed: [production_site1 -> 192.168.50.5]

TASK [Delete database dump from development] ****************************************************
changed: [production_site1 -> 192.168.50.5]

TASK [Search for {{ url_from }} and replace with {{ url_to }} on development] *******************
System info:
  Ansible 2.3.1.0; Darwin
  Trellis at "Option to install WP-CLI packages"
---------------------------------------------------
the field 'args' has an invalid value, which appears to include a variable
that is undefined. The error was: {{
from_host.wordpress_sites[site].site_hosts.0.canonical }}: 'dict object' has
no attribute 'wordpress_sites'

The error appears to have been in
'/Users/david/projects/wordpress/sites/example.com/trellis/database-
pull.yml': line 79, column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


    - name: Search for {{ url_from }} and replace with {{ url_to }} on
development
      ^ here
We could be wrong, but this one looks like it might be an issue with
missing quotes.  Always quote template expression brackets when they
start a value. For instance:

    with_items:
      - {{ foo }}

Should be written as:

    with_items:
      - "{{ foo }}"

fatal: [production_site1]: FAILED! => {"failed": true}
	to retry, use: --limit @/Users/david/projects/wordpress/sites/example.com/trellis/database-pull.retry

PLAY RECAP **************************************************************************************
production_site1       : ok=9    changed=6    unreachable=0    failed=1

[valentinocossar]

Hi @davidyv, sorry, I think I was wrong, the correct hosts file should be this:

production_site1 ansible_host=IP1
production_site2 ansible_host=IP2
production_site3 ansible_host=IP3
production_site4 ansible_host=IP4

[web]
production_site1
production_site2
production_site3
production_site4

[production]
production_site1
production_site2
production_site3
production_site4

However, I think it gives the same problem or even others. I'm sorry, I understand the utility in supporting the functionality you require but I do not think I have time to test the implementation in the short term.

Certainly, the problem is tied to the hosts file and the fact that the tool uses the name of the environment as a parameter for many operations.

If you find the way to fix the problem, I'll ask you to open a pull request to integrate the changes into the tool. Of course, it does not have to alter the way it works now (a least not too much).