Permissions errors Catalina running hdiutil, write to user home, and sudo

[sidney]

I have only tried this in Catalina, as I don't have an older MacOS handy right now. I wouldn't be surprised if it has to do with Catalina's extra security.

I'm trying to write a formula for something that has a Makefile that runs hdiutil to create a dmg. When it tries to run the hdiutil create command to make the dmg file, it fails with an exit code 66. Running the formula with debug so I can break to a shell when the error is raised, I find that trying the hdiutil command by hand gets the same error. If I try to run the same hdiutil command from an ordinary shell it works fine. I also notice that in that shell if I try to touch /Users/myname/foo I get a permissions error. Also I am not allowed to run sudo when in that shell. When I run the id command, it shows the same uid and groups that I get in an ordinary shell.

With hdiutil -debug the lines the output at point of error look like

2020-11-08 13:05:40.699 diskimages-helper[15143:12640914] format: about to formatDevEntry:/dev/disk6s1 executable:/System/Library/Filesystems/apfs.fs/Contents/Resources/newfs_apfs arguments:(
    "-i",
    "-v",
    Enigma
)
newfs_apfs: /dev/rdisk6s1: Operation not permitted
2020-11-08 13:05:40.766 diskimages-helper[15143:12640914] format: formatDevEntry: returned 66
2020-11-08 13:05:40.766 diskimages-helper[15143:12640914] -format: exiting with 66

I can put together an example project and formula if this isn't enough to explain or reproduce the problem, but I thought I would just start with the questions 1) What is it in the environment the formula runs in that has less permission than an ordinary shell? 2) Any suggestions about the problem running hdiutil?

To try this for yourself, in Catalina run a formula in debug that will raise an error so you end up in a shell that has the homebrew shell environment. Try running the commands

mkdir testhdiutil
echo testhdiutil testhdiutil/foo
hdiutil create -ov -srcfolder testhdiutil testhdi.dmg

That just produces the error "hdiutil: create failed - Directory not empty" but you can see the exit code 66 by adding the -debug option after the create.

Also try writing to a file in your users home directory (not $HOME in that environment) and try running something with sudo to verify those aren't allowed either. I don't have a problem with those not being allowed, but I thought maybe the explanation as to why those aren't would shed light on the permissions problem running hdiutil.

[gromgit]

1. What is it in the environment the formula runs in that has less permission than an ordinary shell?

All Homebrew builds are done in a temporary sandbox, so trying to do anything that ventures outside the directory where the build happens is probably best done in a post_install block. (If it's required for the build to proceed, find another way to do it, because I'm pretty sure the Homebrew maintainers would never permit it.)

2. Any suggestions about the problem running hdiutil?

If it requires root, add an explanation for the user to do the necessary steps in the caveats block.

[sidney]

1. The environment is the sandbox. If that is the explanation for not being able to write outside the directory or running sudo, then that's fine. The fact of the existence of the sandbox is exactly the type of answer I was hoping to elicit by mentioning those two operations, and I don't need to do them.

2. No, running hdiutil does not require sudo. What I'm trying to do in the the formula is to run a Makefile to build an app bundle, which does work fine, then run a make step that runs hdiutil create to create a dmg file. The hdiutil create command creates a virtual disk image, copies files to it, then writes the virtual disk image to a dmg file. When, within the sandbox, it tries to write to the virtual disk, that fails with the permission error. I can see how that could happen if the sandbox does not allow writing outside of the current directory.

Is post_install the right way to get the formula to run hdiutil create outside of the sandbox? Where is use of post_install documented? Is the temporary directory used by install still there when post_install runs or would I have to install the files that hdiutil will use before getting to the post_install? I googled for homebrew post_install and didn't see any explanation of it other than that it runs for both source and bottle installs, no other details.

[gromgit]

post_install runs after all the formula's files have been installed in /usr/local (hence the name). That's how it can run "for both source and bottle installs", and going by what you've written so far, it's probably not suitable for your purposes.

In fact, if your formula's sole purpose is to build a DMG, with no intention of installing anything under the Homebrew hierarchy, you're much better off just writing a simple shell script instead. The only use for Homebrew I can think of for your use case is the cask mechanism, but that only tells Homebrew how to manage the installation of your DMG. You'd still have to build the DMG yourself.

[sidney]

The main purpose of the formula is to build the app bundle, with homebrew handling the installation of all required library dependencies. The hdiutil step is to create a dmg for distribution so that it can be downloaded by people who don't have to install homebrew. From what you say, the output of install should be the app bundle, then the one line call to hdiutil can either be in a shell script that the person building runs, or a post_install step that does the same thing, if indeed post_install runs outside the sandbox. I think that solves the problem. Thanks

[sidney]

I now have more complete information that I want to leave here to be seen as part of the marked answer.

The failures I encountered are caused by the fact that brew formulas are run in the MacOS Sandbox. There is only one mention of that in the Homebrew FAQ in a discussion of sudo, and in the Formula Cookbook where the word "sandbox" is used in passing. I think that at least in the Cookbook a sentence should be added to let people know that everything, not just the interactive shell, runs in the sandbox and provide a reference to find exactly what restrictions are imposed by MacOS Sandbox.

postinstall runs in the sandbox just like install, so moving the use of hdutil to a post_install block doesn't help me.

The naive use of hdiutil to create a dmg will not run in the sandbox. There is a project named create-dmg that already has a formula in core which provides an easy wrapper around hdiutil for creating dmg files with many useful options, including an option to perform the creation in the sandbox. It looks like the best solution for me is to make create-dmg a build dependency and then use a call to create-dmg with the proper option for running in the sandbox. If there is any problem with that, an alternative would be to look at the code in create-dmg and call hdiutil the same it does when it needs to run in a sandbox.

[jonchang]

Homebrew is not a generic build system, it's a package manager. You'd be better off using something like Bazel if your goal is to build software for eventual distribution outside of Homebrew.

Pull requests are always welcome to improve Homebrew's documentation.