Broken identity cache (shelve / bdb) when multiple parallel logins happen: _dbm.error: cannot add item to database

[spaceone]

We have a web service running with multiprocessing and therefore cannot have a in-memory identity cache but must share them between the instances.
The documentation of the used shelve module for the identity cache says:

The shelve module does not support concurrent read/write access to shelved objects. (Multiple simultaneous read accesses are safe.) When a program has a shelf open for writing, no other program should have it open for reading or writing. Unix file locking can be used to solve this, but this differs across Unix versions and requires knowledge about the database implementation used.

But pysaml2 doesn't seem to have some locking around write operations.

The exception we receive from customer systems then are:

    response = self.sp.parse_authn_request_response(message, binding, self.outstanding_queries)
  File "/usr/lib/python3/dist-packages/saml2/client_base.py", line 717, in parse_authn_request_response
    self.users.add_information_about_person(resp.session_info())
  File "/usr/lib/python3/dist-packages/saml2/population.py", line 27, in add_information_about_person
    session_info["not_on_or_after"])
  File "/usr/lib/python3/dist-packages/saml2/cache.py", line 129, in set
    self._db[cni] = data
  File "/usr/lib/python3.7/shelve.py", line 125, in __setitem__
    self.dict[key.encode(self.keyencoding)] = f.getvalue()
_dbm.error: cannot add item to database

The database is then in corrupt state:

# db_verify saml-18202.bdb.db
db_verify: BDB0540 Page 1: invalid next_pgno 8780
db_verify: BDB0540 Page 2: invalid next_pgno 6952
db_verify: saml-18202.bdb.db: BDB0090 DB_VERIFY_BAD: Database verification failed
BDB5105 Verification of saml-18202.bdb.db failed.

related issues: #373 (maybe #741, #740).