Skip to content

Batch job submissions timing out #40

@dpwrussell

Description

@dpwrussell

Submissions of batch jobs have been exceeding the 6 second default lambda limit.

[INFO]	2018-09-05T01:17:24.391Z	Found credentials in environment variables.
[INFO]	2018-09-05T01:17:24.638Z	Starting new HTTPS connection (1): ssm.us-east-1.amazonaws.com
START RequestId: e50b2988-6e16-494c-af8e-6e50355d493d Version: $LATEST
Received event: {
"import_uuid": "848e5eea-35e9-412d-adec-a0c023579e96",
"files": [
"ashlar_examples/BP40.ome.tif"
],
"reader": "loci.formats.in.OMETiffReader",
"reader_software": "Bio-Formats",
"reader_version": "(unknown version)",
"fileset_uuid": "e7a6dbc8-a457-4d28-8b3b-65beed085716"
}
Parameters:{
"dir": "848e5eea-35e9-412d-adec-a0c023579e96",
"file": "ashlar_examples/BP40.ome.tif",
"reader": "loci.formats.in.OMETiffReader",
"reader_software": "Bio-Formats",
"reader_version": "(unknown version)",
"fileset_uuid": "e7a6dbc8-a457-4d28-8b3b-65beed085716",
"bucket": "minerva-test-cf-common-tilebucket-1su418jflefem"
}
[INFO]	2018-09-05T01:17:24.875Z	e50b2988-6e16-494c-af8e-6e50355d493d	Starting new HTTPS connection (1): batch.us-east-1.amazonaws.com
END RequestId: e50b2988-6e16-494c-af8e-6e50355d493d
REPORT RequestId: e50b2988-6e16-494c-af8e-6e50355d493d	Duration: 6006.21 ms	Billed Duration: 6000 ms Memory Size: 1024 MB	Max Memory Used: 79 MB	
2018-09-05T01:17:30.840Z e50b2988-6e16-494c-af8e-6e50355d493d Task timed out after 6.01 seconds

[INFO]	2018-09-05T01:17:31.967Z	Found credentials in environment variables.
[INFO]	2018-09-05T01:17:32.37Z	Starting new HTTPS connection (1): ssm.us-east-1.amazonaws.com

There should be no reason for this to time out, it looks like potentially an AWS issue. If this is more than a one time error then we can work around by increasing the lambda timeout, or with retry logic in the step function.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions