Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update EC2 executor to return stdout, stderr, and runtime exceptions #34

Open
5 tasks
cjao opened this issue Nov 10, 2022 · 0 comments
Open
5 tasks

Update EC2 executor to return stdout, stderr, and runtime exceptions #34

cjao opened this issue Nov 10, 2022 · 0 comments

Comments

@cjao
Copy link

cjao commented Nov 10, 2022

The purpose of this issue is to ensure that the EC2 executor

  • returns any stdout and stderr printed by tasks using the mechanism defined in #1380. To do this, the run() implementation needs to
    • retrieves the stdout and stderr from the executor backend -- in the case of AWS Executors, by parsing Cloudwatch logs (see how Braket does this)
    • printing those strings to self.task_stdout and self.task_stderr.
  • distinguish task runtime exceptions -- those raised in the executor backend -- from other exceptions originating from the interaction of Covalent and the executor backend. This is explained in more detail in #1390. When an runtime exception occurs, the run() implementation should:
    1. Retrieve whatever stdout and stderr messages have already been printed by the task
    2. Ensure that the exception traceback is appended to the task's stderr.
    3. Print stdout and stderr to self.task_stdout and self.task_stderr, respectively.
    4. Raise a TaskRuntimeException.
      For examples, see how the dask executor now deals with task runtime exceptions.

Note: These changes should be implemented in a backward-compatible manner -- so that the new AWSExecutors should work with Covalent 0.202.0post1 (the current stable AWSExecutors work with the latest develop).

Acceptance criteria:

  • Any stdout and stderr printed by a task before raising an unhandled exception is retrieved and printed to self.task_stdout and self.task_stderr respectively, where self is the executor plugin instance.
  • If a task raises an exception:
    • The traceback is included in the task’s stderr.
    • The run() method raises a TaskRuntimeError.
  • The executor plugin remains compatible with Covalent Core `0.202.0post1.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant