Trellis and Apple Silicon

[intelligence]

• I've read the guidelines for Contributing to Roots Projects
• This request isn't a duplicate of an existing request
• This is not a personal support request that should be posted on the Roots Discourse community

Summary

Apple Silicon is here, VirtualBox support is far away (if ever) – what is the road going forward?

Motivation

Apple have started to ship their first arm based macs to customers and as many of us developers are sitting on macOS systems I'm curious if you've considered how the road forward will look? As I understand VirtualBox support seem far away, Docker is more probable. I love working with the Roots stack when doing WP work, would be nice to be able to do so in the future.

Some links with discussion around M1 based macs.
https://forums.virtualbox.org/viewtopic.php?f=8&t=98742
https://www.docker.com/blog/apple-silicon-m1-chips-and-docker/
https://localwp.com/community/t/local-on-apple-silicon-macs/22834/14

[swalkinshaw]

I haven't looked into this enough yet, but my interpretation of the current state of things is:

• it seems like virtualization support won't come until the next gen M2 chips (and that's just people's hopes)
• VirtualBox likely won't ever work on M* chips until/if the above happens
• Trellis is quite tied to x86 right now

I think the key point is that Trellis is coupled to x86 more so than VirtualBox itself. Trellis already supports other Vagrant providers which some people use.

There's two biggest questions:

1. will it become more common place to run ARM on production servers?
2. or are people willing to run ARM in development and then deploy to x86 in production?

I personally don't really know the implications of running difference architectures in dev/prod yet. So even if Trellis found a way to migrate to Docker... there's still a lot of potential problems and big open questions.

My stance for now is nothing will change for a while. Anyone running M1 chips is unfortunately out of luck running for Trellis for at least the short-term and almost certainly for the medium-term too.

[nikitasol]

1 sounds a bit farther away, but 2 looks like a necessity already for anyone with ARM macs.

@swalkinshaw Scott, as for short-term, do you have weeks or rather months in mind?

[swalkinshaw]

If you want to run ARM in development now then I'd try and find another Vagrant provider that supports it. VirtualBox isn't the only choice so Trellis doesn't really need to do anything to support that (as far as I know).

[nikitasol]

A bit over my head, but I would like to try it. Would you suggest replacing virtualbox with docker? I remember seeing a couple of years ago a docker feed on discourse, but not sure if it gave any solid fruits in the end.

[nikitasol]

From what I can see, there is no working provider at the moment. Docker is on standby waiting for upstream updates from Go and others with no ETA, Parallels is still in development with no ETA either, VMware is committed as well, but completely silent about roadmap. So there is no possibility to try out the 2nd option for the time being. But it would be wise to at least leave a note for any silicon users stating that the Trellis framework is not compatible with Apple M1 laptops until at least some of the Vagrant providers start rolling out arm tailored versions.

[swalkinshaw]

Ah, yeah sorry I wasn't sure what the status of Parallels and VMware was yet. I had just read they had betas or were working on it.

But yes we definitely need to document this limitation somewhere.

[intelligence]

Seems like great progress is being made!

https://www.xda-developers.com/apple-silicon-mac-boot-windows-10-and-linux-virtualization/

[Pls]

So developers who bought new Macbook Pro with M1 are currently not supported by Trellis? Docker seems to be working on M1, but Trellis is not compatible with Docker. Maby Trellis should really consider Docker as other provider than VirtualBox?

[nikitasol]

@Pls I think Docker is a bit different solution to the question as it basically replaces trellis instead of just creating a vm. But you can try out docker approach. You can find more information here: https://roots.io/docs/bedrock/master/local-development/#additional-resources.

[raphaelparent]

I thought about Rosetta 2 since it allows some apps not made for the M1 chip to actually run, but unfortunately they don't support virtual machines.

What can’t Rosetta 2 translate?
Rosetta cannot translate kernel extensions or Virtual Machine apps that virtualize x86_64 computer platforms. Developers should be aware that Rosetta is also unable to translate AVX, AVX2, and AVX512 vector instructions.

Leaving it as a reference for anyone stumbling here.

[artshostak]

Just got a new iMac with the M1 chip, would love to know what the gameplan is for making it compatible with the Roots suite. Is it just a waiting game at this point?

[amenbreak]

Has anyone attempted Trellis on ARM in development on the M1? Parallels should support that. https://www.parallels.com/blogs/parallels-desktop-m1/

[tangrufus]

Can we get support for m1 macs through docker? It looks like virtualbox and even vmware are dragging their feet on support the macs and some people got vagrant working on docker:
https://dev.to/taybenlor/running-vagrant-on-an-m1-apple-silicon-using-docker-3fh4

It would be great if we could flip a flag and spin up a docker image instead of a virtualbox machine, not sure how hard this would be in practice though.

-- @Digital-Nomad #1302 (comment)

[swalkinshaw]

I've been looking into this a bit more lately, and here's what I've found.

New M1 chips

The new M1 Pro and M1 Max CPUs don't change anything virtualization wise unfortunately. They are the same as the original M1 in that regard.

Virtualbox

There has been absolutely no movement on Virtualbox offering ARM support and I highly doubt there ever will be. It defeats the purpose of VirtualBox which is very tied, by design, to x86.

Other virtualization solutions

While Trellis defaults to VirtualBox (since it's open source and free), it's always supported other providers like VMWare and Parallels.

Parallels

As mentioned above, Parallels does have support for M1 macs but only for ARM based operating systems.

VMware

Likewise, VMware Fusion now supports ARM-based Linux distributions just like Parallels.

Others

• lima-vm - promises all combinations of VMs?
• UTM - QEMU wrapper with GUI
• libvirt - lower-level QEMU wrapper
• docker

One issue with all of these is integrating them with Vagrant. docker has by far the most mature Vagrant integration with a 1st-party provider. Lima and UTM have none so far. vagrant-libvirt exists but has a lot of issues on macOS

Parity between Intel & ARM

The best bet at this point is running Virtualbox on Intel Macs and some other ARM-based virtualization solution on M1 chips. This isn't 100% ideal since it's likely your production server will be running x86 for the foreseeable future; however, I'm not sure it matters in reality. While there are difference between x86 and ARM, it likely does not matter for the purposes of running a web server with WordPress. PHP should be PHP; Nginx should be Nginx, etc.

It's a small trade-off we'll have to make and be okay with in the end.

Conclusion

This space is changing every day and it's hard to keep track. There's a lot of solutions which are like 50-80% of the way there, but not 100% unfortunately. Right now the most promising solution is:

1. stick with Vagrant as the main way to create development VMs
2. continue to use Virtualbox for Intel-based Macs
3. decide on the best option for running Ubuntu ARM VMs for M1 Macs
4. default to Docker currently because it has the most mature Vagrant driver

Of course Docker's default design is not compatible with Trellis because docker images/containers don't like to run multiple services and lack an initd system. This has previously been discussed, but I did find a useful base docker image which solves most of these problems: https://medium.com/nerd-for-tech/developing-on-apple-m1-silicon-with-virtual-environments-4f5f0765fd2f

I've been testing this out on an x86 Mac, and I've run into a few problems (NFS and networking related), but are hopefully solvable.

If anyone wants to help test out any of the other Vagrant providers or VM solutions, please let me know or post your results. It's also helpful for people to post updates about these various projects if they come across them.

Further reading:

• Provider: utm qemu libvirt Varying-Vagrant-Vagrants/VVV#2468
• https://medium.com/@lizrice/linux-vms-on-an-m1-based-mac-with-vscode-and-utm-d73e7cb06133
• Provider for UTM (VM for M1 processors) hashicorp/vagrant#12518
• https://gist.github.com/aserhat/91c1d5633d395d45dc8e5ab12c6b4767

[swalkinshaw]

Help wanted

If you have an M1 Mac and have Parallels or VMware, you can help by trying out an ARM based Ubuntu box.

Since Trellis already supports Parallels and VMware fusion, it's possible to try using an ARM-based Ubuntu vagrant image and get Trellis working "as is".

The main Ubuntu vagrant box we use is bento/ubuntu-20-04 which doesn't yet have support for ARM. However, chef/bento#1374 looks like it has most of it ready. I think you'd need to packer to actually create that box/image and use it locally.

[jgarib]

Ok! I got this to work with parallels on an M1 Pro macbook pro. I'm not the smartest person here so I'm going to lay out some steps in hopes of helping others like me. The machine provisions without errors, but I have not extensively tested beyond this.

**note I believe parallels pro is needed which is what my trial is for this test. The basic version does not provide the cli tools. https://www.parallels.com/products/desktop/pro/#compare

Following swalkinshaw's extensive legwork, using the chef/bento vagrant box on parallels is the only one I could get working. I tried using VMware fusion but I think more work needs to be done to get that working. Jeffnoxon already has a vagrant box compatible with parallels up on the vagrant cloud (https://app.vagrantup.com/jeffnoxon/boxes/ubuntu-20.04-arm64). I'm not sure if it's safe to just reference this box or if something more formal needs to be done.

I added that box to the vagrant.default.yml

vagrant_box: 'jeffnoxon/ubuntu-20.04-arm64'
vagrant_box_version: '>= 1.0.0'

I also needed to make a slight change in the maria-db role. roles > mariadb > defaults > main.yml -- change the arch from amd64 to arm64

mariadb_ppa: "deb [arch=arm64] http://nyc2.mirrors.digitalocean.com/mariadb/repo/10.5/ubuntu {{ ansible_distribution_release }} main"

after that, vagrant up using the parallels provider (make sure you install the parallels vagrant provider plugin (https://parallels.github.io/vagrant-parallels/docs/installation/)

vagrant up --provider=parallels

And success! https://prnt.sc/1xujta1. Loading the site at example.test works and everything appears good. Obviously this will only work for a development environment. I have not tested provisioning a staging / production server on arm64... one step at a time. I can now at least continue developing in glorious real all day battery land and I'll just copy the files to an x86 based machine for the rest of the workflow until I can get more experience with the rest of it.

I'll follow up here with any issues as I find them.

[swalkinshaw]

@jgarib awesome, thank you so much! I'm not that surprised that this "just" worked. Ubuntu should be Ubuntu, so it makes sense it worked once you got that up and running.

re the jeffnoxon/ubuntu-20.04-arm64 box: the base Vagrant box shouldn't matter that much, as long as it's a standard Ubuntu image 👍

I'll grep the source again, but assuming MariaDB is the only place with a hardcoded architecture, we can use Ansible's ansible_architecture to support both (it just needs a mapping).

@jgarib did Nginx not fail with the same issue? It has deb [arch=amd64]. Either way, I'll fix both 👍

[jgarib]

No issue with nginx, though you're right.. you'd think it would fail the same as mariadb. Here is the output of the nginx section of the provisioning with arch=amd64 still set. I can run a quick test with the same change to verify it still works if you'd like?

https://prnt.sc/1xv8e7c

[swalkinshaw]

I'm pretty sure we can just remove the deb arch options entirely. It defaults to the arch that dpkg supports. Running dpkg --print-architecture on my x86 VM just outputs amd64 as you'd expect. I'm assuming that if you run it you'll get back arm64. This makes sense to me since you shouldn't have to explicitly tell apt what your architecture is; it should just default to using your current arch.

[swalkinshaw]

Sorry this is all on the virtual machine side (in Ubuntu)

[swalkinshaw]

This is working on the x86 side for me and I'm assuming it will work for M1s. #1318

[jgarib]

Confirmed working on ARM side. So once that is merged the only change needed to use trellis in a development environment on an M1 mac is:

• parallels pro
• reference the arm based vagrant box mentioned above (jeffnoxon/ubuntu-20.04-arm64) in vagrant.default.yml or whatever change you make to support both!
• ???
• profit

Thanks much Scott!

[swalkinshaw]

🎉 yep, but I can probably update the Vagrantfile to use that box automatically as well so ideally there's no manual steps needed.

[jamesknelson]

Just chiming in to say that I've got Trellis working on an M1 Mac Mini by following these instructions with Parallels Pro.

[jgarib]

Is it possible to set a default provider in the vagrant.default.yml file or anywhere else? It's not ideal to have to type the provider flag every time I provision a machine, though not the end of the world.

[tangrufus]

Trellis’s Vagrantfile tries to auto select a provider in this order:

1. virtualbox
2. vmware_fusion
3. vmware_workstation
4. parallels
5. hyperv

if multiple providers are installed, first one will be used.

Alternatively, you can overrides it via the VAGRANT_DEFAULT_PROVIDER environmental variable.

See: https://www.vagrantup.com/docs/providers/basic_usage#default-provider

[swalkinshaw]

Yeah if you only have Parallels installed it should be the default? Is that not what's happening?

If you have an M1 mac then you definitely shouldn't have VirtualBox installed since it won't work anyway.

[rinatkhaziev]

@swalkinshaw

I also tried the VMWare Fusion preview (which is free right now) and had nothing but trouble

Out of curiosity, what did you encounter?

[swalkinshaw]

• setup was a bit more complicated and I had to manually install some additional tool
• it was really slow to boot up; it was hanging for minutes it seemed like
• despite provisioning succeeding, I couldn't actually access the site due to some dns/port issue

I actually abandoned it after that last issue and got Parallels working fine 🤷‍♂️

[jamesknelson]

After trying both, I ended up shelling out for Parallels for a year. There were too many issues with VMWare, namely:

• After suspending/resuming the VM, VMWare would somehow lose a bunch of files and end up in an unusable state. So I'd need to recreate and reprovision the VM from scratch after each reboot.
• I also ran into the taking-minutes-to-start-up-and-suspend issue.
• Then after all that, it randomly stopped working 🤷‍♂️

[swalkinshaw]

Should have published this sooner, but Roots joined the Parallels affiliate program. So if people do have to purchase Parallels, you can support us by doing it through this link: https://prf.hn/l/KzkNLZB

Note: just make sure to select your proper country/currency.

[swalkinshaw]

I've finally published some docs on the Apple Silicon situation and Parallels: https://docs.roots.io/trellis/master/vagrant/

I'll keep this issue open for discussion and any updates.

[mmoga-uf]

Following the docs for Parallels with an M1, I get this error:

Stderr: Failed to start the VM: Unable to start "trellis-site.com" because your Mac is equipped with the Apple M1 chip that does not support Intel-based operating systems. To create a compatible virtual machine, use an ISO or VHDX image file with an ARM-based operating system.

I've installed the vagrant-parallels plugin and made the changes to the vagrant.default.yml file:

vagrant_box: 'jeffnoxon/ubuntu-20.04-arm64'
vagrant_box_version: '>= 1.0.0'

The only thing I really deviated from was the Vagrant version because I have 2.2.19 installed on my Mac. Has anyone else encountered this? I have Parallels trial version running.

[mmoga-uf]

Sorry, y'all! I had to delete the .vagrant folder and try again. Going to backdate Vagrant now...

[fowlercraig]

@mmoga-uf how are you backdating Vagrant? I've been attempting with Brew, but haven't had any luck.

[swalkinshaw]

See https://discourse.roots.io/t/downgrading-vagrant-version/21742/4

[mmoga-uf]

@mmoga-uf how are you backdating Vagrant? I've been attempting with Brew, but haven't had any luck.

I may have been missing some important point, but I actually kept Vagrant 2.2.19 and then made this change hashicorp/vagrant#12583 (comment).

I got Trellis/Bedrock/Sage all working locally, but then had to bail before doing anything remotely because I work at a RHEL LAMP institution. 😥

[techieshark]

This may be a bit far afield but… 

I wonder if the virtualization changes announced at WWDC 2022 will pave the way for some new alternatives.

Here's an Apple demo video showing how some Swift code can be used to set up a Linux VM (jump to 15:13 to see the setup of Linux on Mac). Here's something similar but in writing.

From this blog post:

In macOS Ventura, Apple is introducing support for the EFI boot loader, which is able to discover any virtual device attached to a VM from which to boot. This makes it possible to download an ISO Linux image, attach it to a virtual storage device, create a boot loader, and boot into the OS, as Poulin shows. This is possible both on Intel and ARM CPUs provided you download the appropriate distribution for your hardware.

Additionally, with macOS Ventura, you will be able to use VirtioGPU 2D, a virtualization device that allows Linux to provide interfaces to the host. This makes it possible to display a Linux GUI inside a macOS window, similarly to how you can display a macOS GUI within a macOS window.

Possibly the most interesting new virtualization feature coming with macOS Ventura is support for Rosetta to run unmodified x86-64 Linux binaries.

(I'm also interested in the open source options… @mikaelwedemeyer I wonder if you had any luck with multipass from Canonical?)

[swalkinshaw]

I briefly read about those macOS Ventura changes too but honestly was a bit confused. So it might improve compatibility for these cases, but performance would still suffer since it's under emulation (unless I'm mistaken). So ultimately still running ARM-based OS's would give the best performance.

[techieshark]

@swalkinshaw it sounds like the performance impact might not be so bad… I'm only just reading up on it now but it seems like to some extent they're able translate the machine instructions and store the translation, so the second run is faster. Also my understanding is they're offloading some or all of that to the GPU.

With new Apple Silicon machines, this is done by Rosetta 2. When you try to open an Intel binary on an M1-based machine, MacOS hands it off to a program called Rosetta 2. Rosetta 2 does an on-the-fly translation of the x86 code into Apple Silicon code, saves it, and then runs the translated code. Some elements are emulated while other elements are completely transcoded.
The first time you run an Intel-based program, it may take a little while for the program to start running. That's because Rosetta is doing a translation pass first. Subsequent runs will then be faster because the translation has already been done. (ref)

One major concern with Rosetta is performance. With the transition from PPC to x86, one factor slowing down Rosetta was the different byte ordering used by the two platforms, with PowerPC being a big-endian architecture, and x86 little-endian. While byte ordering is not a problem for the transition from x86 to ARM, another issue related to memory, namely the memory consistency model total store ordering (TSO), could hamper performance in this case. To prevent this from happening, Apple added support for x86 memory ordering to the M1 CPU, as Robert Graham noted on Twitter. (ref)

But yeah I haven't seen much of a technical analysis for Ventura so … I suppose safest bet for now is to use ARM-based OS like you say.

BTW I have mad some progress with the Trellis + Vagrant + Docker setup using the tips above (which uses ARM-based Linux) - #1253 (comment)

[jgarib]

Today, while spinning up a new project I had to change the vagrant default to:

vagrant_box: 'jeffnoxon/ubuntu-20.04-arm64'
vagrant_box_version: '1.0.0'

The 1.0.1 release of the arm box wasn't booting so removing the >= from the version definition got me going. I'll look into it more later, unless anyone here has already run into similar issue and wants to chime in.

[craigpearson]

Yeah there's an issue with the new vagrant box version, and only version 1.0.0 works without issue

It seems to be network related, and I assume it's either due to the creator exporting the VM without configuring a new MAC address for virtual network cards or related to some of the other changes with parallel tools which are included in v1.0.1

I think this only becomes problematic when you have multiple boxes, due to the mac addresses of the boxes being the same this seems to break parallels due to the same DHCP address being assigned on the required parallels shared network.

Once you hit this issue you have to destroy all your machines, do a hard system restart and rebuild - so it's a bit of a pain

Will see if I can raise with the creator

[jgarib]

Good info. Thanks! Thankfully all my previous boxes (at least the three I've run since) are OK and avoided any collateral damage... at least so far.

[techieshark]

@j-funk you noted the networking issue with Docker - #1253 (comment)

After hitting that myself and digging in a bit, it seems like it might be due to the firewall settings Trellis includes (via ferm). I seem to be having a little luck working around it by adding nameserver 1.1.1.1 to the /etc/resolv.conf on the docker box ; see here for how I persisted this change via a new little Ansible role - https://discourse.roots.io/t/trellis-provisioning-breaks-dns-in-docker/23808/2?u=techieshark

[Bozzified]

Yeah there's an issue with the new vagrant box version, and only version 1.0.0 works without issue

It seems to be network related, and I assume it's either due to the creator exporting the VM without configuring a new MAC address for virtual network cards or related to some of the other changes with parallel tools which are included in v1.0.1

I think this only becomes problematic when you have multiple boxes, due to the mac addresses of the boxes being the same this seems to break parallels due to the same DHCP address being assigned on the required parallels shared network.

Once you hit this issue you have to destroy all your machines, do a hard system restart and rebuild - so it's a bit of a pain

Will see if I can raise with the creator

I'm glad I wasn't going crazy. I noticed this happening and it is huge pain for me. I have multiple boxes running for multiple sites and this has been giving me nightmares.

Hopefully it gets fixed.

[swalkinshaw]

While Trellis has supported Apple Silicon for a while now, it still required two manual changes (Vagrant box + MailHog).
This is long overdue, but I'm happy to say that both those issues are fixed and there's no more manual changes needed. Apple Silicon should just work out of the box (via a Vagrant provider that supports ARM of course).

• Automate arm64 Vagrant box support for Apple Silicon #1431
• Support ARM64 arch in Mailhog role #1432

Thanks for everyone's feedback and help 🚀

[jkananen]

@swalkinshaw it seems trellis new misses these last two updates.

As trellis up failed, I compared my trellis files against trellis master, and clearly #1431 and #1432 weren't in. I've patched my workarea and all is now well.

[swalkinshaw]

Ah, yeah apologies @jkananen I need to do a new Trellis release which I'll do when I'm back from vacation.

In the meantime you can always use trellis new --trellis-version=dev to get the version from master.

[nikitasol]

Anyone had a chance to try Virtualbox ARM version yet? -> https://download.virtualbox.org/virtualbox/7.0.4/VirtualBox-7.0.4_BETA4-154605-macOSArm64.dmg

[tomdevisser]

Anyone had a chance to try Virtualbox ARM version yet? -> https://download.virtualbox.org/virtualbox/7.0.4/VirtualBox-7.0.4_BETA4-154605-macOSArm64.dmg

Doesn't work yet.

The box you're attempting to add doesn't support the provider
you requested. Please find an alternate box or use an alternate
provider. Double-check your requested provider to verify you didn't
simply misspell it.

If you're adding a box from HashiCorp's Vagrant Cloud, make sure the box is
released.

Name: bento/ubuntu-22.04-arm64
Address: https://vagrantcloud.com/bento/ubuntu-22.04-arm64
Requested provider: [:virtualbox]```

[swalkinshaw]

We just released VM integration into trellis-cli powered by Lima 🎉 See https://roots.io/introducing-lima-to-trellis-for-faster-local-development/ for more details and roots/trellis-cli#346 for some documentation.

This has many benefits over Vagrant + VirtualBox/Parallels:

• it's free and open source
• it lighter weight and simpler
• commands will run much faster since Vagrant is slow
• VMs should run faster as well since it uses macOS virtualization framework which is as close to native speed as you can get

[stevedsross]

This is awesome. Thanks for sharing. I'll start testing this in the next couple of weeks.

[minemindmedia]

I am trying to follow along here and get some projects booted up on my new m2 macbook but run into this error which is not very detailed and I've hit a wall. Any ideas how to get passed this?

==> default: Booting VM...

There was an error while command execution. The command and stderr is shown below.

Command: ["/usr/local/bin/prlctl", "start", "03376801-9a15-421d-a445-ef786b52765a"]

Stderr: Failed to start the VM: The virtual machine cannot be started due to a critical error. 

I've updated my vagrantfile as suggested and have destroyed and attempted to launch over and over with no success.

vagrant_box: 'jeffnoxon/ubuntu-20.04-arm64'
vagrant_box_version: '>= 1.0.0'