Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Batch concurrency bug when c > 5 #417

Open
2 tasks
calvinnhieu opened this issue Oct 29, 2019 · 0 comments
Open
2 tasks

Fix Batch concurrency bug when c > 5 #417

calvinnhieu opened this issue Oct 29, 2019 · 0 comments
Assignees
@calvinnhieu
Milestone
Q4 2019 Milestone 3

Comments

@calvinnhieu
Copy link
Collaborator

When > 5 loom requests for the same (cell id set equality) matrix (cache miss) are submitted in parallel, only 5 matrices are generated successfully and the rest fail during the Batch job. The error thrown during Batch resembles:

TypeError: Can't broadcast (63925, 50) -> (63925, 44) during row writing to the h5py.File/loom file.

Review the concurrency configurations for the ECS Query Runner, Redshift and Batch jobs to determine root cause and fix.

Definition of Done:

  • ensure > 5 identical matrices can be generated in parallel successfully for all formats
  • ensure > 5 different matrices can be generated in parallel successfully for all formats
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@calvinnhieu calvinnhieu

Labels
None yet
Projects
None yet
Milestone
Q4 2019 Milestone 3
Development

No branches or pull requests
2 participants
@brianraymor @calvinnhieu