upload-stemcell/checkStemcellUploaded does not detect existing stemcell

[jzinkweg]

Describe the bug

Stemcells downloaded using om download-product are prefixed with [name,version] metadata.

The CheckStemcellAvailability method uses the full filename against the listed stemcells from ops manager diagnostic report:

om/api/stemcell_service.go

Line 78 in b5cc183

if stemcell.Filename == filepath.Base(stemcellFilename) {

On OpsMan 2.10.58, this does not match as the [metadata] prefix is not listed as part of the filename:

  "available_stemcells": [
    {
      "filename": "bosh-stemcell-1.340-vsphere-esxi-ubuntu-jammy-go_agent.tgz",
      "os": "ubuntu-jammy",
      "version": "1.340"
    },
    {
      "filename": "bosh-stemcell-621.813-vsphere-esxi-ubuntu-xenial-go_agent.tgz",
      "os": "ubuntu-xenial",
      "version": "621.813"
    }
  ],

Note that after upgrading to Opsman 3.0.28, the output in the diagnostic report has changed:

  "available_stemcells": [
    {
      "filename": "bosh-vsphere-esxi-ubuntu-jammy-go_agent-1.340.tgz",
      "os": "ubuntu-jammy",
      "version": "1.340"
    },
    {
      "filename": "bosh-vsphere-esxi-ubuntu-xenial-go_agent-621.813.tgz",
      "os": "ubuntu-xenial",
      "version": "621.813"
    }
  ],

As a result, pushing the same stemcell multiple times leads to multiple uploads.
This causes delays in our automation pipeline as one of our sandbox environments is on an unfortunately tiny internet connection.

Reproduction steps

1. Download stemcell using om download-product
2. Upload stemcell for the first time, wait for success
3. Run upload command again and observe second transfer

toolingbox$  om-linux -t https://opsmanager.example.com/ -c ci-concourse -s secret upload-stemcell --stemcell './[stemcells-ubuntu-jammy,1.340]bosh-stemcell-1.340-vsphere-esxi-ubuntu-jammy-go_agent.tgz'
processing stemcell
beginning stemcell upload to Ops Manager
 1.21 GiB / 1.21 GiB [=============================================] 100.00% 10s
3s elapsed, waiting for response from Ops Manager...
finished upload
toolingbox$  om-linux -t https://opsmanager.example.com/ -c ci-concourse -s secret upload-stemcell --stemcell './[stemcells-ubuntu-jammy,1.340]bosh-stemcell-1.340-vsphere-esxi-ubuntu-jammy-go_agent.tgz'
processing stemcell
beginning stemcell upload to Ops Manager
 1.21 GiB / 1.21 GiB [=============================================] 100.00% 10s
3s elapsed, waiting for response from Ops Manager...
finished upload

Expected behavior

Second upload should be skipped as stemcell already exists.

Additional context

Upload-product checks the presence of the product using the metadata name/version instead of file paths.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[vmware-todd-robbins]

I notice there's another PR - #691

I don't understand the logic that re-uploading stemcell all the time is intended behavior? This seems wasteful cycles to automation?

If I rename stemcell file so it matches /api/v0/diagnostic_report then it will not re-upload

% mv bosh-stemcell-1.1091-vsphere-esxi-ubuntu-jammy-go_agent.tgz bosh-vsphere-esxi-ubuntu-jammy-go_agent-1.1091.tgz
% om --env=$OM_ENV upload-stemcell -s  bosh-vsphere-esxi-ubuntu-jammy-go_agent-1.1091.tgz                          
processing stemcell
stemcell has already been uploaded