HA: Limit max open database connections to 1

Previously, the HA feature was allowed to open `max_connections`
database connection in parallel to other Icinga DB components. Meaning,
Icinga DB wasn't limited to the configured `max_connections`, but
effectively to `2 * max_connections`. Additionally, this masked a
serious bug in the `HA#realize()` method, where we start a new
transaction after each retry without rolling back in case of an error,
leading to connections not being released before exceeding the ctx
deadline.

Author: yhabteab

cmd/icingadb/main.go

@@ -113,6 +113,7 @@ func run() int {
 			logger.Fatalf("%+v", errors.Wrap(err, "can't create database connection pool from config"))
 		}
 		defer func() { _ = db.Close() }()
+		db.SetMaxOpenConns(1)
 		ha = icingadb.NewHA(ctx, db, heartbeat, logs.GetChildLogger("high-availability"))
 
 		telemetryLogger := logs.GetChildLogger("telemetry")

doc/03-Configuration.md

@@ -62,6 +62,10 @@ do not need any manual adjustments.
 | max_placeholders_per_statement | **Optional.** Maximum number of placeholders Icinga DB is allowed to use for a single SQL statement. Defaults to `8192`.                         |
 | max_rows_per_transaction       | **Optional.** Maximum number of rows Icinga DB is allowed to `SELECT`,`DELETE`,`UPDATE` or `INSERT` in a single transaction. Defaults to `8192`. |
 | wsrep_sync_wait                | **Optional.** Enforce [Galera cluster](#galera-cluster) nodes to perform strict cluster-wide causality checks. Defaults to `7`.                  |
+ 
+The [Icinga DB HA](05-Distributed-Setups.md#high-availability) feature uses its own database connection, separate
+from the other Icinga DB components, and this configuration cannot be altered by end users. Therefore, expect
+Icinga DB to establish `max_connections` + `1` database connections.
 
 ## Logging Configuration