Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HA: Reduce deadlocks via exclusive locking (SELECT ... FOR UPDATE) #830

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

HA: Reduce deadlocks via exclusive locking (SELECT ... FOR UPDATE) #830

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c9deda3
HA: Reduce deadlocks via exclusive locking (`SELECT ... FOR UPDATE`)
Al2Klimov Aug 12, 2024
04dd8ff
HA: Use `... FOR UPDATE` lock clause unconditionally
yhabteab Oct 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions pkg/icingadb/ha.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -290,22 +290,28 @@ func (h *HA) realize(
takeover = ""
otherResponsible = false
isoLvl := sql.LevelSerializable
selectLock := ""

if h.db.DriverName() == database.MySQL {
// The RDBMS may actually be a Percona XtraDB Cluster which doesn't
// support serializable transactions, but only their following equivalent:
// The RDBMS may actually be a Percona XtraDB Cluster which doesn't support serializable
// transactions, but only their equivalent SELECT ... LOCK IN SHARE MODE.
// See https://dev.mysql.com/doc/refman/8.4/en/innodb-transaction-isolation-levels.html#isolevel_serializable
isoLvl = sql.LevelRepeatableRead
selectLock = " LOCK IN SHARE MODE"
}

tx, errBegin := h.db.BeginTxx(ctx, &sql.TxOptions{Isolation: isoLvl})
if errBegin != nil {
return errors.Wrap(errBegin, "can't start transaction")
}

query := h.db.Rebind("SELECT id, heartbeat FROM icingadb_instance "+
"WHERE environment_id = ? AND responsible = ? AND id <> ?") + selectLock
// In order to reduce the deadlocks on both sides, it is necessary to obtain an exclusive lock
// on the selected rows. This can be achieved by utilising the SELECT ... FOR UPDATE command.
// Nevertheless, deadlocks may still occur, when the "icingadb_instance" table is empty, i.e. when
// there's no available row to be locked exclusively.
//
// Note that even without the ... FOR UPDATE lock clause, this shouldn't cause a deadlock on PostgreSQL.
// Instead, it triggers a read/write serialization failure when attempting to commit the transaction.
query := h.db.Rebind("SELECT id, heartbeat FROM icingadb_instance " +
"WHERE environment_id = ? AND responsible = ? AND id <> ? FOR UPDATE")

instance := &v1.IcingadbInstance{}
errQuery := tx.QueryRowxContext(ctx, query, envId, "y", h.instanceId).StructScan(instance)
Expand Down
Loading