Can batch cold starts be eliminated or mitigated?

[brianraymor]

Per our July 9 discussion on performance, there was an earlier thread on #matrix-dev slack:

Parth Shah [Mar 27th at 10:20 AM]
It can take up to 10 mins just for batch to scale up resources if there are a lot of requests in the queue... so there's gonna be a lot of variability here

Calvin
Is batch the best choice for this job? We’re running a single job per request which can be done with ECS. Using batch, it seems like what we gain over ECS is better job visibility and “smarter” resource allocation at the cost of 10 minute cold starts

Parth Shah
It depends on our file system requirements. Ecs + fargate allocates 10gb. This is probably enough for right now but how large are we anticipating our full hca matrix files to be in the future?

But Marcus expects the matrix service will easily exceed the 10GB allocation. Back of napkin calculation was hundreds of GB(s), so ECS is not a viable solution.

[brianraymor]

See related discussion on AWS forum.

[brianraymor]

Per discussion at the October 1 Matrix meeting - agreed to icebox this issue for future consideration.