Stove cannot publish to behind the firewall Supermarket #72

rylarson · 2015-03-18T16:49:23Z

I use stove to publish the cygwin cookbook to the community site, and it works fine. But when I try to publish the same cookbook to my behind the firewall community site, I get the following error (with personal info redacted):

$ stove --endpoint https://supermarket.example.com/api/v1 --username user --key /Users/user/user.pem --no-git --log-level=debug
W: No config file found at `/Users/user/.stove'!
                      Stove::Cli | ===> Options: {:endpoint=>"https://supermarket.example.com/api/v1", :username=>"user", :key=>"/Users/user/user.pem", :remote=>"origin", :branch=>"master", :sign=>false, :log_level=>"debug", :path=>"/Users/user/working/chef-cygwin", :no_git=>true}
                      Stove::Cli | ===> ARGV: []
                   Stove::Runner | ===> Skipping plugin `:git'
                   Stove::Runner | ===> Running plugin `:community'
                Stove::Validator | ===> Running validations for `community.username'
                Stove::Validator |      Validation username passed!
                Stove::Validator | ===> Running validations for `community.key'
                Stove::Validator |      Validation key passed!
        Stove::Plugin::Community | ===> Publishing the release to the Chef community site
             ChefAPI::Connection | ===> POST cookbooks...
             ChefAPI::Connection |      Chef flavor: :open_source
             ChefAPI::Connection | ===> Building URI...
             ChefAPI::Connection |      Detected URI is relative
             ChefAPI::Connection |      Appending cookbooks to https://supermarket.example.com/api/v1
             ChefAPI::Connection | ===> Adding request headers...
             ChefAPI::Connection |      Accept: application/json
             ChefAPI::Connection |      Content-Type: application/json
             ChefAPI::Connection |      Connection: keep-alive
             ChefAPI::Connection |      Keep-Alive: 30
             ChefAPI::Connection |      User-Agent: ChefAPI Ruby Gem 0.5.0
             ChefAPI::Connection |      X-Chef-Version: 11.4.0
             ChefAPI::Connection | ===> Detected multipart body
             ChefAPI::Connection |      Content-Type: multipart/form-data; boundary=------ChefAPIMultipartBoundary
             ChefAPI::Connection |      Content-Length: 2809
             ChefAPI::Connection | ===> Adding signed header authentication...
         ChefAPI::Authentication | ===> Parsing private key...
         ChefAPI::Authentication |      Detected private key is the path to a file
             ChefAPI::Connection |      X-Ops-Sign: algorithm=sha1;version=1.0;
             ChefAPI::Connection |      X-Ops-Userid: user
             ChefAPI::Connection |      X-Ops-Timestamp: 2015-03-18T16:01:34Z
             ChefAPI::Connection |      X-Ops-Content-Hash: XXXXXXXXXXXXXXXXXXXXXXXXXXX=
             ChefAPI::Connection |      X-Ops-Authorization-1: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
             ChefAPI::Connection |      X-Ops-Authorization-2: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
             ChefAPI::Connection |      X-Ops-Authorization-3: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
             ChefAPI::Connection |      X-Ops-Authorization-4: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
             ChefAPI::Connection |      X-Ops-Authorization-5: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
             ChefAPI::Connection |      X-Ops-Authorization-6: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
             ChefAPI::Connection | **** Disabling SSL verification...
             ChefAPI::Connection | **** Neither ChefAPI nor the maintainers are responsible for damanges incurred as a result of disabling SSL verification. Please use this with extreme caution, or consider specifying a custom certificate using `config.ssl_pem_file'.
             ChefAPI::Connection |      Raw response:
             ChefAPI::Connection |      {"error_code":"INVALID_DATA","error_messages":["The metadata is formatted incorrectly.","Multipart POST part 'tarball' must contain a non-empty README."]}
             ChefAPI::Connection | ===> Parsing response as error...
             ChefAPI::Connection |      Detected error response as JSON
             ChefAPI::Connection |      Parsing error response as JSON
                      Stove::Cli | >>>> Stove experienced an error!
                      Stove::Cli | >>>> ChefAPI::Error::HTTPBadRequest
                      Stove::Cli | >>>> The Chef Server did not understand the request because it was malformed.

    {"error_code"=>"INVALID_DATA", "error_messages"=>["The metadata is formatted incorrectly.", "Multipart POST part 'tarball' must contain a non-empty README."]}

                      Stove::Cli | >>>> /Users/user/.rvm/gems/ruby-2.0.0-p451/gems/chef-api-0.5.0/lib/chef-api/connection.rb:411:in `error'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/chef-api-0.5.0/lib/chef-api/connection.rb:282:in `block in request'
/Users/user/.rvm/rubies/ruby-2.0.0-p451/lib/ruby/2.0.0/net/http.rb:852:in `start'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/chef-api-0.5.0/lib/chef-api/connection.rb:268:in `request'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/chef-api-0.5.0/lib/chef-api/connection.rb:122:in `post'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/stove-3.2.6/lib/stove/community.rb:54:in `upload'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/stove-3.2.6/lib/stove/mixins/instanceable.rb:20:in `method_missing'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/stove-3.2.6/lib/stove/plugins/community.rb:15:in `block in <class:Community>'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/stove-3.2.6/lib/stove/plugins/base.rb:12:in `instance_eval'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/stove-3.2.6/lib/stove/plugins/base.rb:12:in `block in run'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/stove-3.2.6/lib/stove/plugins/base.rb:44:in `call'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/stove-3.2.6/lib/stove/plugins/base.rb:44:in `block in run_actions'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/stove-3.2.6/lib/stove/plugins/base.rb:43:in `each'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/stove-3.2.6/lib/stove/plugins/base.rb:43:in `run_actions'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/stove-3.2.6/lib/stove/plugins/base.rb:33:in `run'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/stove-3.2.6/lib/stove/runner.rb:26:in `run_plugin'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/stove-3.2.6/lib/stove/runner.rb:15:in `run'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/stove-3.2.6/lib/stove/cli.rb:71:in `execute!'
/Users/user/.rvm/gems/ruby-2.0.0-p451/gems/stove-3.2.6/bin/stove:4:in `<top (required)>'
/Users/user/.rvm/gems/ruby-2.0.0-p451/bin/stove:23:in `load'
/Users/user/.rvm/gems/ruby-2.0.0-p451/bin/stove:23:in `<main>'
/Users/user/.rvm/gems/ruby-2.0.0-p451/bin/ruby_executable_hooks:15:in `eval'
/Users/user/.rvm/gems/ruby-2.0.0-p451/bin/ruby_executable_hooks:15:in `<main>'

However, when I use knife-supermarket, I can upload just fine:

$ knife supermarket share -m https://supermarket.example.com cygwin -o cookbooks/ -k /Users/user/user.pem -u user -V
INFO: Using configuration from /Users/user/chef-repo/knife.rb
INFO: Unable to access cache at /var/chef. Switching cache to /Users/user/.chef
INFO: Validating ruby files
INFO: Validating templates
INFO: Syntax OK
Generating metadata for cygwin from /var/folders/27/6sg87fpd6cq4jy8_510gpxl08wv5x0/T/chef-cygwin-build20150318-28396-bnknz6/cygwin/metadata.rb
Making tarball cygwin.tgz
Upload complete!

The cookbook source is here: https://github.com/rylarson/chef-cygwin and you can see that the README file is anything but empty.

The behind the firewall supermarket is 1.6.0 installed via the chef yum repository on RHEL 6.6 x64

I have access to the servers private key and I can do a wireshark dump of the traffic if you would like.

The text was updated successfully, but these errors were encountered:

sethvargo · 2015-03-18T17:16:41Z

@nathenharvey any ideas on this? Is there a difference between the inhouse and public supermarket?

prometheanfire · 2015-03-18T17:18:56Z

I got this error trying to publish to the normal supermarket as well.

E: Stove experienced an error!
E: ChefAPI::Error::HTTPBadRequest
E: The Chef Server did not understand the request because it was malformed.

    {"error_code"=>"INVALID_DATA", "error_messages"=>["The metadata is formatted incorrectly.", "Multipart POST part 'tarball' must contain a non-empty README."]}

E: /home/mthode/.gem/ruby/2.0.0/gems/chef-api-0.5.0/lib/chef-api/connection.rb:411:in `error'

and here it is working

knife supermarket share phpstack
Generating metadata for phpstack from /tmp/chef-phpstack-build20150318-12259-9s4fnq/phpstack/metadata.rb
Making tarball phpstack.tgz
Upload complete!

nathenharvey · 2015-03-18T17:46:18Z

From the stack traces, it looks like you're using knife supermarket to publish successfully and stove is failing. The error message leads me to believe that your phpstack cookbook does not have a README.md file. Looks like stove requires a README.md file.

Does that help? Does the cookbook contain a README.md? If you add one, does stove do what you expect?

rylarson · 2015-03-18T17:47:55Z

The cookbook has a README.md, and it looks to me like that error is coming from the server, not from Stove.

I was previously using Stove 3.2.3 to publish to the community site and that was working. Using 3.2.3 I got a different error that was just a straight 500 error with no extra messages.

Ryan

prometheanfire · 2015-03-18T17:49:19Z

Ya, I have a README.md as well.

sethvargo · 2015-03-18T17:53:08Z

Can you try removing all the content from your README except a single character? I wonder if it's barfing on some unicode error?

nathenharvey · 2015-03-18T17:53:15Z

yep, I was a bit to quick to respond. It looks like stove might not be packaging up that README.md into the tarball it uses. I'm not sure what's happening here.

@sethvargo is it possible to use stove to generate the tarball and then stop without uploading that tarball.

sethvargo · 2015-03-18T17:57:49Z

Not from the CLI. But if you open up an IRB session, it's something like:

require "stove"
cookbook = Stove::Cookbook.new("/path/to/cookbook/root")
cookbook.tarball #=> "/path/on/disk.tar.gz"

Does that help?

nathenharvey · 2015-03-18T18:07:07Z

Looks like stove is generating a README.md for https://github.com/rylarson/chef-cygwin OK.

Does @sethvargo's suggestion of clearing most of the content from the README give a different result?

rylarson · 2015-03-18T19:12:59Z

A single character in the README yielded the same result. So did renaming README.md => README

rylarson · 2015-03-18T19:25:34Z

FWIW, Stove 3.0 works fine (although I had to change my version constraints in my metadata file)

$ stove --endpoint https://supermarket.example.com/api/v1 --username user --key /Users/user/working/knife-publish/chef-repo/user.pem --no-git --log-level=debug --category OTHER
W: No config file found at `/Users/user/.stove'!
                      Stove::Cli | ===> Options: {:endpoint=>"https://supermarket.example.com/api/v1", :username=>"user", :key=>"/Users/user/working/knife-publish/chef-repo/user.pem", :category=>"OTHER", :remote=>"origin", :branch=>"master", :sign=>false, :log_level=>"debug", :path=>"/Users/user/working/chef-cygwin", :no_git=>true, :version=>nil}
                      Stove::Cli | ===> ARGV: []
                   Stove::Runner | ===> Skipping plugin `:git'
                   Stove::Runner | ===> Running plugin `:community'
                Stove::Validator | ===> Running validations for `community.username'
                Stove::Validator |      Validation username passed!
                Stove::Validator | ===> Running validations for `community.key'
                Stove::Validator |      Validation key passed!
                Stove::Validator | ===> Running validations for `community.category'
                Stove::Validator |      Validation category passed!
        Stove::Plugin::Community | ===> Publishing the release to the Chef community site
             ChefAPI::Connection | ===> POST cookbooks...
             ChefAPI::Connection |      Chef flavor: :open_source
             ChefAPI::Connection | ===> Building URI...
             ChefAPI::Connection |      Detected URI is relative
             ChefAPI::Connection |      Appending cookbooks to https://supermarket.example.com/api/v1
             ChefAPI::Connection | ===> Adding request headers...
             ChefAPI::Connection |      Accept: application/json
             ChefAPI::Connection |      Content-Type: application/json
             ChefAPI::Connection |      Connection: keep-alive
             ChefAPI::Connection |      Keep-Alive: 30
             ChefAPI::Connection |      User-Agent: ChefAPI Ruby Gem 0.5.0
             ChefAPI::Connection |      X-Chef-Version: 11.4.0
             ChefAPI::Connection | ===> Detected multipart body
             ChefAPI::Connection |      Content-Type: multipart/form-data; boundary=------ChefAPIMultipartBoundary
             ChefAPI::Connection |      Content-Length: 2611
             ChefAPI::Connection | ===> Adding signed header authentication...
         ChefAPI::Authentication | ===> Parsing private key...
         ChefAPI::Authentication |      Detected private key is the path to a file
             ChefAPI::Connection |      X-Ops-Sign: algorithm=sha1;version=1.0;
             ChefAPI::Connection |      X-Ops-Userid: user
             ChefAPI::Connection |      X-Ops-Timestamp: 2015-03-18T19:20:53Z
             ChefAPI::Connection | **** Disabling SSL verification...
             ChefAPI::Connection | **** Neither ChefAPI nor the maintainers are responsible for damanges incurred as a result of disabling SSL verification. Please use this with extreme caution, or consider specifying a custom certificate using `config.ssl_pem_file'.
             ChefAPI::Connection |      Raw response:
             ChefAPI::Connection |      {"uri":"https://supermarket.example.com/api/v1/cookbooks/cygwin"}
             ChefAPI::Connection | ===> Parsing response as success...
             ChefAPI::Connection |      Detected response as JSON
             ChefAPI::Connection |      Parsing response body as JSON

Same error with newer version of stove.

svanzoest · 2015-03-30T19:40:59Z

I just ran into this with 3.2.6 a downgrade to 3.2.5 fixed it. Seems odd that the changes in that release would cause it though.

sethvargo · 2015-03-30T19:56:18Z

@nathenharvey does the behind-the-firewall version of Supermarket support the new metadata params? I wonder if we are sending them up and the supermarket is rejecting them because they are not recognized.

guilhem · 2015-04-01T16:27:41Z

Same for me. Downgrading fix the problem :)

prometheanfire · 2015-04-03T18:32:11Z

this issue occurred because stove now requires metadata.rb to include the source_url and issues_url attribute. For instance... https://github.com/rackspace-cookbooks/ohai_public_info/blob/master/metadata.rb

martinb3 · 2015-04-08T19:05:31Z

This is particularly unpleasant as stove only fails after pushing the tag to git, meaning you have to forcibly clean up the tag from remote git before re-uploading. Perhaps stove could verify the git repo is clean, changes are pushed, to the upload, then push the tag? I'd rather the upload be validated by the remote server before a tag is pushed in git.

chef-boneyard/stove#72

rylarson · 2015-04-08T23:00:50Z

I would rather the tag fail since it is easy to remove tags but impossible
to yank a specific cookbook version
On Apr 8, 2015 12:05 PM, "Martin Smith" [email protected] wrote:

This is particularly unpleasant as stove only fails after pushing the
tag to git, meaning you have to forcibly clean up the tag from remote git
before re-uploading. Perhaps stove could verify the git repo is clean,
changes are pushed, to the upload, then push the tag? I'd rather the upload
be validated by the remote server before a tag is pushed in git.

—
Reply to this email directly or view it on GitHub
#72 (comment).

martinb3 · 2015-04-09T10:13:42Z

I'd just prefer to have the Chef-related APIs validate the uploaded tarball before anything else happens, as that seems to be the hardest part to check locally. Maybe we need some sort of way to validate the tarball, then push the tag, then upload the tarball.

sethvargo · 2015-04-09T14:18:51Z

So, I think this is the issue...

@nathenharvey added new metadata in #71. This is important because new cookbook's metadata wasn't being parsed or uploaded by stove. However, supermarket in-house doesn't support those fields yet, so it's rejecting the request. We have a catch-22 now 😦.

nathenharvey · 2015-04-09T15:01:29Z

should we release a 3.2.7 version of stove that reverts those new attributes and release a 3.3.0 version that includes them? And make the necessary updates to the README and such?

In the meantime, I'll work with the Supermarket team to see if the current version of a private supermarket supports those fields and discuss some better way of signaling this to stove.

/cc @nellshamrell @kirtfitzpatrick

sethvargo · 2015-04-09T16:26:34Z

That seems like a reasonable alternative to me. I can work on that later tonight.

rylarson · 2015-04-09T22:03:45Z

Sounds good to me. Thanks @sethvargo and @nathenharvey

jtimberman · 2015-04-09T23:35:30Z

Fwiw, I'm getting this error on a cookbook that does not have source_url or issues_url at all in its metadata.

sethvargo · 2015-04-09T23:47:16Z

@jtimberman right, but stove is sending metadata that looks like this:

{
  "issues_url": ""
}

Since the default value is empty string

jtimberman · 2015-04-10T14:06:39Z

Ah ha! Okay then. 3.2.5 is fine for now :).

martinb3 · 2015-04-16T14:38:57Z

We're also seeing that once you do include these, the cookbooks are no longer usable by chef 11. So you can either use the latest stove and ditch chef 11, or use an older stove.

sethvargo · 2015-04-16T15:18:00Z

@martinb3 ah - that's excellent to know. This really helps identify my pathway moving forward. I am going to allow the fields in the READ of the metadata, but not include them in the WRITE of the metadata. This will ensure BC.

I may add a flag that allows the new fields to be included.

sethvargo · 2015-04-16T22:40:16Z

This should be fixed in 3.2.7!

sethvargo added the Bug label Mar 18, 2015

guilhem added a commit to sous-chefs/nodejs that referenced this issue Apr 8, 2015

Add source_url and issues_url metadata

3524a84

chef-boneyard/stove#72

guilhem mentioned this issue Apr 8, 2015

Add source_url and issues_url metadata sous-chefs/nodejs#75

Merged

sethvargo closed this as completed Apr 16, 2015

tas50 added Type: Bug and removed Bug labels Oct 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stove cannot publish to behind the firewall Supermarket #72

Stove cannot publish to behind the firewall Supermarket #72

rylarson commented Mar 18, 2015

sethvargo commented Mar 18, 2015

prometheanfire commented Mar 18, 2015

nathenharvey commented Mar 18, 2015

rylarson commented Mar 18, 2015

prometheanfire commented Mar 18, 2015

sethvargo commented Mar 18, 2015

nathenharvey commented Mar 18, 2015

sethvargo commented Mar 18, 2015

nathenharvey commented Mar 18, 2015

rylarson commented Mar 18, 2015

rylarson commented Mar 18, 2015

svanzoest commented Mar 30, 2015

sethvargo commented Mar 30, 2015

guilhem commented Apr 1, 2015

prometheanfire commented Apr 3, 2015

martinb3 commented Apr 8, 2015

rylarson commented Apr 8, 2015

martinb3 commented Apr 9, 2015

sethvargo commented Apr 9, 2015

nathenharvey commented Apr 9, 2015

sethvargo commented Apr 9, 2015

rylarson commented Apr 9, 2015

jtimberman commented Apr 9, 2015

sethvargo commented Apr 9, 2015

jtimberman commented Apr 10, 2015

martinb3 commented Apr 16, 2015

sethvargo commented Apr 16, 2015

sethvargo commented Apr 16, 2015

Stove cannot publish to behind the firewall Supermarket #72

Stove cannot publish to behind the firewall Supermarket #72

Comments

rylarson commented Mar 18, 2015

sethvargo commented Mar 18, 2015

prometheanfire commented Mar 18, 2015

nathenharvey commented Mar 18, 2015

rylarson commented Mar 18, 2015

prometheanfire commented Mar 18, 2015

sethvargo commented Mar 18, 2015

nathenharvey commented Mar 18, 2015

sethvargo commented Mar 18, 2015

nathenharvey commented Mar 18, 2015

rylarson commented Mar 18, 2015

rylarson commented Mar 18, 2015

svanzoest commented Mar 30, 2015

sethvargo commented Mar 30, 2015

guilhem commented Apr 1, 2015

prometheanfire commented Apr 3, 2015

martinb3 commented Apr 8, 2015

rylarson commented Apr 8, 2015

martinb3 commented Apr 9, 2015

sethvargo commented Apr 9, 2015

nathenharvey commented Apr 9, 2015

sethvargo commented Apr 9, 2015

rylarson commented Apr 9, 2015

jtimberman commented Apr 9, 2015

sethvargo commented Apr 9, 2015

jtimberman commented Apr 10, 2015

martinb3 commented Apr 16, 2015

sethvargo commented Apr 16, 2015

sethvargo commented Apr 16, 2015