Failing nodes' error messages are no longer accessible to the UI

[cjao]

When a node fails, the UI only displays the name of the failed node and not any detailed error message.This occurs because a failing node's error message is not propagated correctly to the Result object.

When a node fails during execution, the Result object stores the node's error message in Result._error for display in the UI. Unfortunately, Result._get_node_error() no longer seems to work even though the error message is stored correctly in the node. Consequently, users cannot view the error message in the UI and can only see it by querying the transport graph directly, which also requires them to know the failing node's node_id.

Example:

import covalent as ct

@ct.electron
def failing_task(x):
    assert False
    return x

@ct.lattice
def workflow(x):
    res = failing_task(x)
    return res

dispatch_id = ct.dispatch(workflow)(1)

res = ct.get_result(dispatch_id, wait=True)

print(res.status)

print(res.error)

yields the generic output

FAILED
Node failing_task failed: 

>>> res._get_node_error(0)

''

The error is stored in the node:

>>> res.lattice.transport_graph.get_node_value(0, "error")

'Traceback (most recent call last):\n  File "/var/home/casey/.conda/envs/qa-0.177-3.8/lib/python3.8/site-packages/covalent_dispatcher/_core/execution.py", line 355, in _run_task\n    output, stdout, stderr = await loop.run_in_executor(tasks_pool, execute_callable)\n  File "/var/home/casey/.conda/envs/qa-0.177-3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run\n    result = self.fn(*self.args, **self.kwargs)\n  File "/var/home/casey/.conda/envs/qa-0.177-3.8/lib/python3.8/site-packages/covalent/executor/base.py", line 279, in execute\n    result = self.run(function, args, kwargs, task_metadata)\n  File "/var/home/casey/.conda/envs/qa-0.177-3.8/lib/python3.8/site-packages/covalent/executor/executor_plugins/local.py", line 61, in run\n    return function(*args, **kwargs)\n  File "/var/home/casey/.conda/envs/qa-0.177-3.8/lib/python3.8/site-packages/covalent/executor/base.py", line 95, in wrapper_fn\n    output = fn(*new_args, **new_kwargs)\n  File "/tmp/ipykernel_70281/1978061097.py", line 5, in failing_task\nAssertionError\n'

[Prasy12]

@cjao We read the error information from the error.log file that is generated for the lattices and for nodes we read it from the stdder.log. Hope once this fix is done, these files will be populated with the required information.

cc: @mshkanth

[cjao]

The problem here is that the schema is missing an Electrons.error field. This would contain the string representation of any exception raised by the task. We may add this to V13 or implement some other fix.

[Prasy12]

@cjao We read the error information from the error.log file that is generated for the lattices and for nodes we read it from the stdder.log. Hope once this fix is done, these files will be populated with the required information.

cc: @mshkanth

Sure Casey, I have mentioned the way we read and show the errors on the UI. So if these files are populated properly it should be good.

[cjao]

By the way, the error is stored in the node only because we still pickle the entire transport graph, albeit only $O(1)$ times during a workflow execution. This was a stopgap measure introduced early in the DB implementation. Moving forward we plan to reconstruct the transport graph purely from the Electron and Electron_Dependency tables.

[cjao]

@santoshkumarradha What is the priority on this? Without a fix, errors from electrons would be completely opaque unless the user is well-versed in Covalent internals.

[santoshkumarradha]

@wjcunningham7 can you help me scope out the technicalities ?

[cjao]

We discussed this in the last SW meeting.

To answer the question of how to display exceptions raised by a task, we may wish to distinguish between different classes of exceptions:

• Application Errors: exceptions raised by the task inside the executor and propagated back to the server.

  • Here, streaming to node["stderr"] makes a lot of sense since applications normally log all errors to stderr when run from the console.
  • Is it there any advantage to reporting terminal exceptions separately from non-fatal errors, such as exceptions which are suppressed but logged?

• Task-specific server errors: exceptions raised in the executor but serverside (e.g. ssh login fails or docker image failed to build):

  • Storing the tracebacks in the transport graph node, perhaps in a field like node["executor_errors"].
  • One option discussed ways to stream them to covalent_ui.log. Since the log multiplexes messages from all tasks from all dispatches, we need a way to quickly look up the events for a specific task execution. This would be challenging since covalent_ui.log would be a giant text log not designed for random access. Even if that were possible, we would need to bundle these with the result object when the client calls ct.get_result().

• Other server errors occurring outside task execution.

[cjao]

This is fixed as of v0.209.0