CNDB-14460: Fix Nodes test flakiness resulting from unsafe interleaving of async operations in test scenarios

[jkni]

What is the issue

The singleton Nodes instance sequences operations that should not overlap by running them on a single-threaded executor. In some cases, these operations are executed in a synchronous manner, where the caller waits on the future. In other cases, they're executed asynchronously by queuing. In some tests, the singleton Nodes instance is shut down and replaced in an unsafe manner, to test cases where a node is restarted. This shut down does not terminate or wait on the executor, as the asynchronous tasks can safely be recovered on node restart. In the tests, however, these asynchronous operations can interleave with the newly created Nodes instance such that the operations no longer have the expected isolation, resulting in test failures.

Async operations can also interleave with the temporary directories backing a Nodes instance being deleted by Junit.

What does this PR fix and why was it fixed

When unsafely replacing the singleton Nodes instance in tests, trigger a shutdown on the executors and await inflight tasks.

When shutting down at the end of a test, await inflight tasks.

[github-actions]

Checklist before you submit for review

• Make sure there is a PR in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits
• All new files should contain the DataStax copyright header instead of the Apache License one

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[cassci-bot]

✔️ Build ds-cassandra-pr-gate/PR-1812 approved by Butler

---

Approved by Butler
See build details here

[djatnieks]

The changes make sense and look good.

Q: Where are we seeing the flakiness? I may be missing it - I checked Butler and didn't see any NodesTest failures there?

[jkni]

It's a good question! We've had decent luck with it lately, but there are a few failures on PRs: http://butler-stargazer.aws.dsinternal.org/#/ci/upstream/workflow/ds-cassandra-pr-gate/failure/org.apache.cassandra.nodes/NodesTest?test_name=testPeersSerialization. This was also (for whatever timing reason) failing fairly reliably for me as I was doing local development.