Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Input SandBox Files #296

Open
gmolto opened this issue Nov 26, 2018 · 0 comments
Open

Allow Input SandBox Files #296

gmolto opened this issue Nov 26, 2018 · 0 comments

Comments

@gmolto
Copy link
Member

gmolto commented Nov 26, 2018

I hit the 24-KB limit when defining a script to be executed in AWS Batch.

We could workaround this by supporting an input sandbox of files that is specified in the YAML file, is automatically upload to a specific folder in the S3 bucket and is automatically retrieved by the SCAR supervisor before executing the script.

From the top of my head:

functions:
  my-function:
    image: org/repo
    memory: 128
    execution_mode: batch
    s3:
      input_bucket: my-bucket
    init_script: my-script
    input_sandbox:
      - /path/to/local/file-or-folder

This would cause to upload the files in /path/to/local/file-or-folder into s3://my-bucket/my-function/sandbox and define an environment variable when creating the Lambda function that this folder exists.

The supervisor would check this variable (which should be made available to the AWS Batch job as well) and retrieve whatever the contents of that S3 folder contains into the local file system so that the script running in the container can access these files. The local directory in the running container on which the files where retrieved should be made available to the script as an environment variable such as: $SCAR_INPUT_SANDBOX or simply be made available in the same local directory on which

@alpegon alpegon self-assigned this Nov 26, 2018
@alpegon alpegon removed their assignment Sep 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants