Openstack image gets deleted after their creation with packer

[hc-github-team-packer]

This issue was originally opened by @unnikrishnan262 in hashicorp/packer#11136 and has been migrated to this repository. The original issue description is below.

---

When filing a bug, please include the following headings if possible. Any
example text in this template can be deleted.

Overview of the Issue

I encounter the same issue as the one described here (7312). (which is closed - i've tested all the workaround shown there)

Every time I create an image on Openstack, it is get deleted at the end of the process. Here is the basic output of the packer command. We are using VMware as OpenStack backend. If I run the packer build with '-debug' option, the image creation is successful.

It looks like from the Nova logs that there is a race condition in which vmware snapshot get deleted before the image upload completes and when we run with debug, there are few secs delay while we hit enter multiple time. Can we introduce a pause before the "Terminating the source server" step or do more check to make sure image is created successfully?

it reads like packer only looks at the image and when it's available, while there's an endpoint in Nova for checking if an asynchronous task is done.
https://docs.openstack.org/api-guide/compute/faults.html#server-actions

openstack: [DEBUG] Request GET https://image-3.ap-ae-1.cloud.sap/v2/images/5404d85a-aef5-42d2-ab5b-5562f86e11e6 200
openstack: [DEBUG] Request GET https://image-3.ap-ae-1.cloud.sap/v2/images/5404d85a-aef5-42d2-ab5b-5562f86e11e6 200

==> openstack: Pausing after run of step 'stepCreateImage'. Press enter to continue.
==> openstack: Pausing after run of step 'stepUpdateImageTags'. Press enter to continue.
==> openstack: Pausing after run of step 'stepUpdateImageVisibility'. Press enter to continue.
==> openstack: Pausing after run of step 'stepAddImageMembers'. Press enter to continue.
==> openstack: Pausing after run of step 'stepUpdateImageMinDisk'. Press enter to continue.
==> openstack: Pausing before cleanup of step 'stepUpdateImageMinDisk'. Press enter to continue.
==> openstack: Pausing before cleanup of step 'stepAddImageMembers'. Press enter to continue.
==> openstack: Pausing before cleanup of step 'stepUpdateImageVisibility'. Press enter to continue.
==> openstack: Pausing before cleanup of step 'stepUpdateImageTags'. Press enter to continue.
==> openstack: Pausing before cleanup of step 'stepCreateImage'. Press enter to continue.
==> openstack: Pausing before cleanup of step 'StepDetachVolume'. Press enter to continue.
==> openstack: Pausing before cleanup of step 'StepStopServer'. Press enter to continue.
==> openstack: Pausing before cleanup of step 'StepCleanupTempKeys'. Press enter to continue.
==> openstack: Pausing before cleanup of step 'StepProvision'. Press enter to continue.
==> openstack: Pausing before cleanup of step 'StepConnect'. Press enter to continue.
==> openstack: Pausing before cleanup of step 'StepAllocateIp'. Press enter to continue.
==> openstack: Pausing before cleanup of step 'StepWaitForRackConnect'. Press enter to continue.
==> openstack: Pausing before cleanup of step 'StepGetPassword'. Press enter to continue.
==> openstack: Pausing before cleanup of step 'StepRunSourceServer'. Press enter to continue.
==> openstack: Terminating the source server: 33e859df-d6c6-4885-b383-4c7fb3a1dd99 ...
openstack: [DEBUG] Request POST https://compute-3.ap-ae-1.cloud.sap:443/v2.1/servers/33e859df-d6c6-4885-b383-4c7fb3a1dd99/action 202
openstack: [DEBUG] Request GET https://compute-3.ap-ae-1.cloud.sap:443/v2.1/servers/33e859df-d6c6-4885-b383-4c7fb3a1dd99 200
openstack: [DEBUG] Request GET https://compute-3.ap-ae-1.cloud.sap:443/v2.1/servers/33e859df-d6c6-4885-b383-4c7fb3a1dd99 200

[root@test100 ~]# /usr/local/bin/packer2 build -on-error=ask -force rhel.json
openstack: output will be in this color.

==> openstack: Loading flavor: 20
openstack: Verified flavor. ID: 20
==> openstack: Creating temporary keypair: packer_60e3fa23-14b7-398b-3c79-61a5899874f6 ...
==> openstack: Created temporary keypair: packer_60e3fa23-14b7-398b-3c79-61a5899874f6
==> openstack: Launching server...
==> openstack: Launching server...
openstack: Server ID: dfd3de80-6f6e-4398-8fbd-c0e83f802b49
==> openstack: Waiting for server to become ready...
openstack: Floating IP not required
==> openstack: Using SSH communicator to connect: 10.209.251.152
==> openstack: Waiting for SSH to become available...
==> openstack: Connected to SSH!
==> openstack: Stopping server: dfd3de80-6f6e-4398-8fbd-c0e83f802b49 ...
openstack: Waiting for server to stop: dfd3de80-6f6e-4398-8fbd-c0e83f802b49 ...
==> openstack: Creating the image: rhel-6-vmware
openstack: Image: f01cbcb0-ea30-4cac-a3bb-f7c2ca747c70
==> openstack: Waiting for image rhel-6-vmware (image id: f01cbcb0-ea30-4cac-a3bb-f7c2ca747c70) to become ready...
==> openstack: Terminating the source server: dfd3de80-6f6e-4398-8fbd-c0e83f802b49 ...
==> openstack: Deleting temporary keypair: packer_60e3fa23-14b7-398b-3c79-61a5899874f6 ...
Build 'openstack' finished after 5 minutes 12 seconds.

==> Wait completed after 5 minutes 12 seconds

==> Builds finished. The artifacts of successful builds are:
--> openstack: An image was created: f01cbcb0-ea30-4cac-a3bb-f7c2ca747c70
[root@test100 ~]#

Packer version

1.6.6

Simplified Packer Buildfile

If the file is longer than a few dozen lines, please include the URL to the
gist of the log or use the Github detailed
format
instead of posting it directly in the issue.

Operating system and Environment details

I tried running packer on my Mac, RHEL7 but same result.

The Openstack nova log shows the below error.

2021-07-01 06:59:32,511.511 6 ERROR oslo_messaging.rpc.server [req-9e3cd3c3-b2cc-471b-9d1d-940ec32ec1e3 greq-522d7578-a4db-f848-f209-788fb4149167 fdda6904f2d786d6092ca89c0e991d96f9e823fac746ed4ea635ab0199cea8ae 184f96ee29874bb488574a0fc130b094 - 4f18f794e1f04441a4607200295b7f2e 4f18f794e1f04441a4607200295b7f2e] Exception during message handling: ManagedObjectNotFoundException: The object 'vim.vm.Snapshot:snapshot-2028228' has already been deleted or has not been completely created

[nywilken]

Hi @unnikrishnan262 thanks for opening up this issue with the detailed debug information. I've moved this issue to the OpenStack plugin repo so that it can be tracked here.

With that said, I see that you are running version 1.6.6 of Packer. There have been some changes to the OpenStack builder since versions 1.7.0 and 1.7.1 of Packer.

Does running with the latest version of Packer result in the same error?

[unnikrishnan262]

Thank you for the update. I will test the same with latest code and update

[unnikrishnan262]

Tested the same code with 1.7.3 and running into the same issue.