Stop large searches blocking small searches

[zabeen]

We already did some work on this last year for match prediction: #944. It worked when we have one large search run before a small search.

However, in production we have encountered cases of multiple large searches being run in parallel, at which point our existing fix seems to no longer be effective, probably as there is an upper limit to how much we can horizontally scale. So we need to consider other ways we can optimise internal queuing of requests/batches to prevent this problem.

• Reduce MatchPredictionProcessingBatchSize on LIVE-WMDA-ATLAS-FUNCTIONS #1283
• Implement "smart" batching in match prediction to maximise benefit of horizontal scaling #736

Possible Ideas

Note: if we improve efficiency of single searches, as covered by #1280, then we may not need to consider further queue management optimisations. Below are just thoughts I want to capture somewhere and not necessarily ideas to be executed. 😄

Limit parallelism of mismatch requests

• We know that mismatch Adult searches bring back significantly more donors than 10/10 searches, and these are the matching requests that tend to fail when the app service plan is under load.
• Users at WMDA are required to run 10/10 search before mismatch search.
  • If 10/10 brings back >40K results, they are not allowed to submit a mismatch search.
• Could we place a limit on how many mismatch Adult searches are run in parallel?
  1. 10/10 search is the first search for a patient and thus more important to complete quickly.
  2. Mismatch searches will result in more load on the system, and so we want to control how many mismatch searches are run in parallel.
• Pros: the most urgent 10/10 searches will be prioritised and not blocked by resource-intensive mismatch searches.
• Cons:
  • If there happen to be many mismatch Adult searches submitted at once, the "mismatch queue" may become too long.
    • Can mitigate this problem by making the parallelism limit configurable
  • I am not sure how to implement this within existing service bus topic architecture in a sensible way - needs HLD