CheckSchema: Verify intermediate schema upgrades

When skipping a version for an Icinga DB upgrade, all intermediate
upgrade steps must be taken. While this is already stated in the
documentation, it might be overlooked.

This happened for one community user, upgrading from v1.1.0 to v1.2.0,
skipping the intermediate schema upgrade for v1.1.1.

> https://community.icinga.com/t/icingadb-failing-exactly-5-minutes-after-start/13955

First, the necessity for all upgrades in their release order was made
more prominent in the documentation, hoping that less users would ignore
this when skimming the upgrade docs.

However, the real change here is adding another check to the CheckSchema
function, verifying that all schema upgrades between the lowest known
version and the highest known version in the icingadb_schema table
exists. If an intermediate schema upgrade was skipped, as in the thread
above, this raises a descriptive error.

doc/04-Upgrading.md

@@ -1,7 +1,10 @@
 # Upgrading Icinga DB
 
 Specific version upgrades are described below. Please note that version upgrades are incremental.
-If you are upgrading across multiple versions, make sure to follow the steps for each of them.
+
+If you are upgrading across multiple versions, make sure to follow the steps for each version in order of release.
+For example, when upgrading from 1.1.0 to 1.2.0,
+follow all instructions for upgrading to 1.1.1, then all for 1.2.0, including schema upgrades.
 
 ## Upgrading to Icinga DB v1.2.0
 

pkg/icingadb/schema.go

@@ -25,13 +25,13 @@ func CheckSchema(ctx context.Context, db *database.DB) error {
 		expectedDbSchemaVersion = expectedPostgresSchemaVersion
 	}
 
-	var version uint16
+	var versions []uint16
 
 	err := retry.WithBackoff(
 		ctx,
 		func(ctx context.Context) (err error) {
-			query := "SELECT version FROM icingadb_schema ORDER BY id DESC LIMIT 1"
-			err = db.QueryRowxContext(ctx, query).Scan(&version)
+			query := "SELECT version FROM icingadb_schema ORDER BY version ASC"
+			err = db.SelectContext(ctx, &versions, query)
 			if err != nil {
 				err = database.CantPerformQuery(err, query)
 			}
@@ -44,13 +44,26 @@ func CheckSchema(ctx context.Context, db *database.DB) error {
 		return errors.Wrap(err, "can't check database schema version")
 	}
 
-	if version != expectedDbSchemaVersion {
+	if len(versions) == 0 {
+		return fmt.Errorf("no database schema version is stored in the database")
+	}
+
+	for i := 0; i < len(versions)-1; i++ {
+		if versions[i] != versions[i+1]-1 {
+			return fmt.Errorf(
+				"incomplete database schema upgrade: intermediate version v%d is missing,"+
+					" please make sure you have applied all database migrations after upgrading Icinga DB",
+				versions[i]+1)
+		}
+	}
+
+	if latestVersion := versions[len(versions)-1]; latestVersion != expectedDbSchemaVersion {
 		// Since these error messages are trivial and mostly caused by users, we don't need
 		// to print a stack trace here. However, since errors.Errorf() does this automatically,
 		// we need to use fmt instead.
 		return fmt.Errorf(
 			"unexpected database schema version: v%d (expected v%d), please make sure you have applied all database"+
-				" migrations after upgrading Icinga DB", version, expectedDbSchemaVersion,
+				" migrations after upgrading Icinga DB", latestVersion, expectedDbSchemaVersion,
 		)
 	}