Upgrade fails when Volume in CreationError state exist

[thomasdanan]

Component:

'salt', 'upgrade'

What happened:

When upgrading a 6 nodes cluster from 2.8.0 to 2.9.1, with some Volume in CreationError state, the upgrade is failing with the following error:

                            ----------
                                      ID: Create backing storage for myvolume
                                Function: metalk8s_volumes.present
                                    Name: myvolume
                                  Result: False
                                 Comment: Cannot create storage for volume myvolume: block device /dev/sdb does not exists.
                                 Started: 08:10:53.826828
                                Duration: 8.183 ms
                                 Changes:
                            ----------
                                      ID: Prepare backing storage for myvolume
                                Function: metalk8s_volumes.prepared
                                    Name: myvolume
                                  Result: False
                                 Comment: One or more requisite failed: metalk8s.volumes.prepared.Create backing storage for myvolume
                                 Started: 08:10:53.841214
                                Duration: 0.008 ms
                                 Changes:
                            ----------

What was expected:

Either, the upgrade script in dry-run mode or pre-checks should warn the user that those type of Volumes should be cleaned before starting the upgrade. Either the upgrade should be able to succeed, even with those type of Volumes.

Steps to reproduce

Resolution proposal (optional):