Building an omnibus package requires write access to the install destination #841

drbrain · 2018-06-12T20:37:15Z

Description

When I'm building the next version of an omnibus-packaged application omnibus obliterates the existing files I installed using a previous package. If the build fails I may have a corrupt or non-working application.

If I'm building for the first time omnibus expects to be able to write to the installed locations. If this directory is only root-writable the omnibus build will fail with Errno::EACCESS as it tries to delete or overwrite the installed files.

Additionally, a macOS .pkg requires root permissions to install the package, which omnibus should not need to build that package.

Instead omnibus should build the installer in a directory completely under its ownership.

Omnibus Version

v5.6.8

Platform Version

macOS 10.13.5: Darwin asx.local 17.6.0 Darwin Kernel Version 17.6.0: Tue May 8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASE_X86_64 x86_64

ruby 2.5.0: ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-darwin17]

Replication Case

This starts by following the Get Sarted steps in the omnibus README, then extends that to a scenario where you've installed your software and want to package when you have a previous package installed.

omnibus new MY_PROJECT_NAME
bundle install --binstubs
bin/omnibus build MY_PROJECT_NAME
1. Discover this doesn't work because /var/cache/omnibus is not writable
2. File Building an omnibus package writes to the install destination #840 because this is not documented as a necessary step
3. Find out you need to create /var/cache/omnibus and make it writable
4. sudo mkdir /var/cache/omnibus
5. sudo chown $USER /var/cache/omnibus
Re-run bin/omnibus build MY_PROJECT_NAME
1. Discover this also doesn't work because you are missing "software named `preparation'"
2. Find and copy that file into the project
3. Skip filing a bug that will probably get closed like Building an omnibus package writes to the install destination #840 did, where you tried to re-open it after following up with a repeated reproduction
Re-run bin/omnibus build MY_PROJECT_NAME
1. Discover this also doesn't work because you are missing "software named `version-manifest'"
2. Find and copy that file into the project
3. Skip filing a bug that will probably get closed like Building an omnibus package writes to the install destination #840 did, where you tried to re-open it after following up with a repeated reproduction
Re-run bin/omnibus build MY_PROJECT_NAME
1. Discover this doesn't work because it's trying to write to /opt/MY_PROJECT_NAME
2. sudo mkdir /opt/MY_PROJECT_NAME
3. sudo chown $USER /opt/MY_PROJECT_NAME/
Re-run bin/omnibus build MY_PROJECT_NAME
1. This completes with pkg/myprojectname-0.0.0+20180612202154-1.pkg being created
sudo rm -r /opt/MY_PROJECT_NAME/
open pkg/myprojectname-0.0.0+20180612202154-1.pkg
1. Click yes/accept/etc. through all the prompts and enter your password such that the software would be installed
Make some change to my omnibus project
bin/omnibus build MY_PROJECT_NAME
1. Discover this fails with Permission denied @ apply2files - /opt/MY_PROJECT_NAME/LICENSES/.gitkeep (Errno::EACCES)

Expected results

I want to be able to build a package without corrupting and replacing my currently installed project. If there is some issue with the package it may not complete replace my currently installed and working version.

I should not require write access by default to the package installation destination. Since the installer requires my password to install, and they're installed such that I can't mess them up by accident, then building the software should not touch that same location.

Step 6 must not require 6.ⅱ or 6.ⅲ and step 11 must work without changing the permissions of /opt/MY_PROJECT_NAME

Additionally, the Get Started steps are incomplete and the README generated in the omnibus-MY_PROJECT_NAME directory does not include any of the extra steps I had to perform to reproduce the bug I care about.

Actual results

The install location must be writable in order to build an omnibus package

Additionally, if you file a bug about how the Get Started steps are missing steps (#840) it gets closed and the steps are not updated for correctness.

Build Output

Step 6:

$ bin/omnibus build MY_PROJECT_NAME
                    [CLI] I | 2018-06-12T13:19:36-07:00 | Using config from 'omnibus.rb'
[2018-06-12T13:19:36-07:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
           [BuildVersion] W | 2018-06-12T13:19:37-07:00 | Could not extract version information from 'git describe'! Setting version to 0.0.0.
Building MY_PROJECT_NAME 0.0.0+20180612201937...
  [Software: preparation] I | 2018-06-12T13:19:37-07:00 | Resolving manifest entry for preparation
[NullFetcher: preparation] I | 2018-06-12T13:19:37-07:00 | Fetching `preparation' (nothing to fetch)
[Software: version-manifest] I | 2018-06-12T13:19:37-07:00 | Resolving manifest entry for version-manifest
[NullFetcher: version-manifest] I | 2018-06-12T13:19:37-07:00 | Fetching `version-manifest' (nothing to fetch)
Traceback (most recent call last):
	17: from bin/omnibus:29:in `<main>'
	16: from bin/omnibus:29:in `load'
	15: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/bin/omnibus:16:in `<top (required)>'
	14: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/cli.rb:42:in `execute!'
	13: from /usr/local/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
	12: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/cli/base.rb:33:in `dispatch'
	11: from /usr/local/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
	10: from /usr/local/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
	 9: from /usr/local/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
	 8: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/cli.rb:89:in `build'
	 7: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/project.rb:1073:in `build'
	 6: from /usr/local/lib/ruby/2.5.0/fileutils.rb:193:in `mkdir_p'
	 5: from /usr/local/lib/ruby/2.5.0/fileutils.rb:193:in `each'
	 4: from /usr/local/lib/ruby/2.5.0/fileutils.rb:208:in `block in mkdir_p'
	 3: from /usr/local/lib/ruby/2.5.0/fileutils.rb:208:in `reverse_each'
	 2: from /usr/local/lib/ruby/2.5.0/fileutils.rb:210:in `block (2 levels) in mkdir_p'
	 1: from /usr/local/lib/ruby/2.5.0/fileutils.rb:232:in `fu_mkdir'
/usr/local/lib/ruby/2.5.0/fileutils.rb:232:in `mkdir': Permission denied @ dir_s_mkdir - /opt/MY_PROJECT_NAME (Errno::EACCES)

Step 11:

$ bin/omnibus build MY_PROJECT_NAME
                    [CLI] I | 2018-06-12T13:23:57-07:00 | Using config from 'omnibus.rb'
[2018-06-12T13:23:57-07:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
           [BuildVersion] W | 2018-06-12T13:23:58-07:00 | Could not extract version information from 'git describe'! Setting version to 0.0.0.
Building MY_PROJECT_NAME 0.0.0+20180612202358...
  [Software: preparation] I | 2018-06-12T13:23:58-07:00 | Resolving manifest entry for preparation
[NullFetcher: preparation] I | 2018-06-12T13:23:58-07:00 | Fetching `preparation' (nothing to fetch)
[Software: version-manifest] I | 2018-06-12T13:23:58-07:00 | Resolving manifest entry for version-manifest
[NullFetcher: version-manifest] I | 2018-06-12T13:23:58-07:00 | Fetching `version-manifest' (nothing to fetch)
Traceback (most recent call last):
	18: from bin/omnibus:29:in `<main>'
	17: from bin/omnibus:29:in `load'
	16: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/bin/omnibus:16:in `<top (required)>'
	15: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/cli.rb:42:in `execute!'
	14: from /usr/local/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
	13: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/cli/base.rb:33:in `dispatch'
	12: from /usr/local/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
	11: from /usr/local/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
	10: from /usr/local/lib/ruby/gems/2.5.0/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
	 9: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/cli.rb:89:in `build'
	 8: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/project.rb:1075:in `build'
	 7: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/licensing.rb:57:in `create_incrementally'
	 6: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/licensing.rb:57:in `tap'
	 5: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/licensing.rb:59:in `block in create_incrementally'
	 4: from /usr/local/lib/ruby/gems/2.5.0/gems/omnibus-5.6.8/lib/omnibus/licensing.rb:119:in `prepare'
	 3: from /usr/local/lib/ruby/2.5.0/fileutils.rb:1051:in `touch'
	 2: from /usr/local/lib/ruby/2.5.0/fileutils.rb:1051:in `each'
	 1: from /usr/local/lib/ruby/2.5.0/fileutils.rb:1054:in `block in touch'
/usr/local/lib/ruby/2.5.0/fileutils.rb:1054:in `utime': Permission denied @ apply2files - /opt/MY_PROJECT_NAME/LICENSES/.gitkeep (Errno::EACCES)

The text was updated successfully, but these errors were encountered:

stevendanna · 2019-05-01T09:36:28Z

Apologies this sat for so long without a response. You've reported a number of issues here. I've tried to list them all below:

/var/cache/omnibus needs to exist and be writable
preparation software not included by default
version_manifest software not included by default
/opt/MY_PROJECT_NAME needs to exist and be writable
Can't rebuild MY_PROJECT_NAME when it is already installed

1 and 4 should probably be fixed with a combination of better docs and better error messages. We could potentially default the omnibus cache dir to another location if not running as root.

2 and 3 should just be fixed. It isn't clear to me we need the version_manifest software anymore since omnibus can do that directly. I think this was broken here: 2d1d714 since these definitions are included in omnibus-software which was included in the bundle.

Unfortunately, issue 5 is much trickier. The issue is that building the software in any directory other than /opt/PACKAGE_NAME is likely to (1) require pretty drastic changes to how omnibus builds C software and (2) will break compatibility with existing plans. I think a better error message is definitely in order; but, at this stage in the project, I'm not sure it will be worth the trade-off in compatibility to completely fix.

Omnibus was previously generating an invalid project by default because it: - Didn't include a version-manifest software definition - Didn't include the "preparation" software definition - Had an invalid Gemfile This fixes those issues, getting us a bit closer to the Getting Starting directions working out of the box for people. Fixes #887 Partially addresses #841 Signed-off-by: Steven Danna <[email protected]>

drbrain · 2019-05-01T16:46:24Z

Would you like me to open separate issues for the multiple issues you've identified as combined into this one?

5 is disappointing, I've worked on other software packagers (RubyGems, ruby-packer) and have managed to make use of ./configure features to install into temporary directories that allow for proper linking to occur during the build phase. My understanding is that these features are what OS package managers use to create their packages, also using temporary directories.

stevendanna · 2019-05-03T16:32:12Z

Would you like me to open separate issues for the multiple issues you've identified as combined into this one?

I thinks this is OK. I've fixed 2 of the issues, and another 2 are directly related to the title of the ticket so this ticket is fine to track them.

5 is disappointing, I've worked on other software packagers (RubyGems, ruby-packer) and have managed to make use of ./configure features to install into temporary directories that allow for proper linking to occur during the build phase. My understanding is that these features are what OS package managers use to create their packages, also using temporary directories.

I agree. It isn't impossible to do, but it would require many people to update their software definitions potentially. Perhaps a separate ticket for this could be opened and we could consider it if we ever decide to do another major version of omnibus. Or if someone thinks of a way to do it without breaking compatibility.

drbrain · 2019-05-06T17:40:56Z

I'd hate to leave an issue around to decay in the history and not be noticed if you do get around to a major version bump

Omnibus was previously generating an invalid project by default because it: - Didn't include a version-manifest software definition - Didn't include the "preparation" software definition - Had an invalid Gemfile This fixes those issues, getting us a bit closer to the Getting Starting directions working out of the box for people. Fixes #887 Partially addresses #841 Signed-off-by: Steven Danna <[email protected]>

stevendanna mentioned this issue May 1, 2019

Generate a valid project by default #888

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Building an omnibus package requires write access to the install destination #841

Building an omnibus package requires write access to the install destination #841

drbrain commented Jun 12, 2018

stevendanna commented May 1, 2019 •

edited

Loading

drbrain commented May 1, 2019

stevendanna commented May 3, 2019 •

edited

Loading

drbrain commented May 6, 2019

Building an omnibus package requires write access to the install destination #841

Building an omnibus package requires write access to the install destination #841

Comments

drbrain commented Jun 12, 2018

Description

Omnibus Version

Platform Version

Replication Case

Expected results

Actual results

Build Output

stevendanna commented May 1, 2019 • edited Loading

drbrain commented May 1, 2019

stevendanna commented May 3, 2019 • edited Loading

drbrain commented May 6, 2019

stevendanna commented May 1, 2019 •

edited

Loading

stevendanna commented May 3, 2019 •

edited

Loading