fix(HA): Limit max open database connections to 1

Previously, the HA feature was allowed to open `max_connections`
database connection in parallel to other Icinga DB components. Meaning,
Icinga DB wasn't limited to the configured `max_connections`, but
effectively to `2 * max_connections`. Additionally, this masked a
serious bug in the `HA#realize()` method, where we start a new
transaction after each retry without rolling back in case of an error,
leading to connections never being released.

Author: yhabteab

cmd/icingadb/main.go

@@ -109,6 +109,7 @@ func run() int {
 		heartbeat = icingaredis.NewHeartbeat(ctx, rc, logs.GetChildLogger("heartbeat"))
 
 		db, err := cmd.Database(logs.GetChildLogger("database"))
+		db.SetMaxOpenConns(1)
 		if err != nil {
 			logger.Fatalf("%+v", errors.Wrap(err, "can't create database connection pool from config"))
 		}

doc/03-Configuration.md

@@ -63,6 +63,12 @@ do not need any manual adjustments.
 | max_rows_per_transaction       | **Optional.** Maximum number of rows Icinga DB is allowed to `SELECT`,`DELETE`,`UPDATE` or `INSERT` in a single transaction. Defaults to `8192`. |
 | wsrep_sync_wait                | **Optional.** Enforce [Galera cluster](#galera-cluster) nodes to perform strict cluster-wide causality checks. Defaults to `7`.                  |
 
+!!! info
+
+    The [Icinga DB HA](05-Distributed-Setups.md#high-availability) feature uses its own database connection, separate
+    from the other Icinga DB components, and this configuration cannot be altered by end users. Therefore, expect
+    Icinga DB to establish `max_connections` + `1` database connections.
+
 ## Logging Configuration
 
 Configuration of the logging component used by Icinga DB.