From #2 (comment):
The other thing I noticed is that if rosetta is enabled, bumping the nixpkgs input means you need an aarch64-linux VM to rebuild the x86_64-linux vm (which means disabling the virby service, then enabling the linux-builder, then rebuilding, then enabling virby, etc).
That's in the case that a cached version of the VM isn't available - which #2 is about; I wonder if it wouldn't be more appropriate to set up the rosetta vm as a second guest, in addition (and not instead of) the regular aarch64 vm.