re-run and with-items - execution result/output not updated

[jpavlav]

I have a relatively complex workflow that is doing some provisioning work. Many of the tasks are somewhat long-running and can fail for various reasons. For this reason, all of those tasks use with-items so that the workflow can be re-run using the --tasks flag and the --no-reset flag.

The each subsequent task relies on the published values from the previous tasks to "know" what to do. However, I've found that, when using re-run the execution created by the requested re-run has identical results/output to the original execution. The prior execution's information is visible before the requested re-run execution is even finished.

This makes sense to me, in that, the intention is to re-use the original workflow execution. However, it appears that the result/output of the requested re-run execution does not update the output of the original execution. Essentially in a case with one or more items, where there is a failure, there appears to be no way to actually reference the new executions results:

            ---> task2 (item1) x --->
task1 ---> |                         | ---> check_outcome_task ---> task3 (item2) --> .....
            ---> task2 (item2) √ ---> 

In ☝️ scenario, I would respond by running:

st2 execution re-run <task_id> --tasks task2 --no-reset task2

However, the published values of the now successful task2 for item1 are either inaccessible, resulting in TypeErrors or in a simplified workflow, where the output of the tasks are not consumed, the result of that re-run execution is identical to the original execution. I posted about this in the #community channel in Slack, but I'm adding here, just in case there are more eyes on these discussions. The following are some minimal files that set up the scenario I've described and their output:

dummy_action2.yaml:

---
name: dummy_action2
pack: dummy_pack
description: |
  * Launches the dummy workflow
runner_type: orquesta
entry_point: workflows/dummy_workflow2.yaml
enabled: true
​
parameters:
  hostnames:
    type: array
    description: hostnames

dummy_workflow2.yaml:

---
version: 1.0
description: A dummy workflow
​
input:
  - hostnames
​
output:
  - python_result: "{{ ctx().python_result  }}"
​
tasks:
  dummy_python:
    action: dummy_pack.dummy_python_action
    with:
      items: "{{ ctx().hostnames }}"
    input:
      host: "{{ item() }}"
    next:
      - when: "{{ succeeded() }}"
        publish:
          - python_result: "{{ task() }}"
        do:
          - nothing
      - when: "{{ failed }}'"
        publish:
          - python_result: "{{ task() }}"
        do:
          - nothing
​
  nothing:
    action: core.noop

dummy_python_action.yaml:

---
name: "dummy_python_action"
runner_type: "python-script"
description: "Succeed or Fail"
enabled: true
entry_point: "lib/dummy_action.py"
parameters:
  host:
    type: string
    description: hostnames
    required: true
    secret: false

dummy_action.py

import uuid
​
from st2common.runners.base_action import Action
​
​
class DummyAction(Action):
    def run(self, host):
        fail_or_pass = self.action_service.get_value(name="test_host1", local=False)
        if fail_or_pass == "true":
            fail_or_pass = True
        else:
            fail_or_pass = False
        if host == "host1":
            return (
                fail_or_pass,
                {
                    "host_pass_fail_list": [
                        {
                            "host": host,
                            "status": "failed" if not fail_or_pass else "passed",
                            "uuid": str(uuid.uuid4()),
                        }
                    ]
                },
            )
​
        return (
            True,
            {
                "host_pass_fail_list": [
                    {"host": host, "status": "passed", "uuid": str(uuid.uuid4())}
                ]
            },
        )

☝️ I wanted to verify that I wasn't just missing something, so I created this example for my own edification. The dummy_action.py python Action uses a value in the datastore to determine whether or not a task spun out with-items that includes host1 as input succeeds or fails. The test goes as follows:

I set the datastore value to false ensuring host1 will fail:

st2 key set test_host1 false
+------------------+--------------+
| Property         | Value        |
+------------------+--------------+
| name             | test_host1   |
| value            | false        |
| scope            | st2kv.system |
| expire_timestamp |              |
+------------------+--------------+

And run the action:

st2 run dummy_pack.dummy_action2 hostnames="host1,host2"
.
id: 6488e87c279a956d7fb7786c
action.ref: dummy_pack.dummy_action2
parameters:
  hostnames:
  - host1
  - host2
status: succeeded
start_timestamp: Tue, 13 Jun 2023 22:06:52 UTC
end_timestamp: Tue, 13 Jun 2023 22:06:54 UTC
log:
  - status: requested
    timestamp: '2023-06-13T22:06:52.965000Z'
  - status: scheduled
    timestamp: '2023-06-13T22:06:53.110000Z'
  - status: running
    timestamp: '2023-06-13T22:06:53.138000Z'
  - status: succeeded
    timestamp: '2023-06-13T22:06:54.706000Z'
result:
  output:
    python_result:
      end_timestamp: '2023-06-13 22:06:54.313670+00:00'
      result:
        items:
        - result:
            exit_code: 0
            result:
              host_pass_fail_list:
              - host: host1
                status: failed
                uuid: e35183f3-1355-4190-b457-296add2df3b6
            stderr: 'st2.actions.python.DummyAction: DEBUG    Creating new Client object.
              st2.actions.python.DummyAction: DEBUG    Retrieving value from the datastore (name=test_host1)
              '
            stdout: ''
          status: failed
        - result:
            exit_code: 0
            result:
              host_pass_fail_list:
              - host: host2
                status: passed
                uuid: 93f3c9af-f1ef-4dee-9d68-75d903a125d9
            stderr: 'st2.actions.python.DummyAction: DEBUG    Creating new Client object.
              st2.actions.python.DummyAction: DEBUG    Retrieving value from the datastore (name=test_host1)
              '
            stdout: ''
          status: succeeded
      route: 0
      start_timestamp: '2023-06-13 22:06:53.381685+00:00'
      status: failed
      task_execution_id: 6488e87dcd89d51113579fdf
      task_id: dummy_python
      task_name: dummy_python
      workflow_execution_id: 6488e87df5c87ffc2f94b693
+--------------------------+------------------------+--------------+----------------------------------+-------------------------------+
| id                       | status                 | task         | action                           | start_timestamp               |
+--------------------------+------------------------+--------------+----------------------------------+-------------------------------+
| 6488e87dcd89d51113579fe2 | failed (1s elapsed)    | dummy_python | dummy_pack.dummy_python_action | Tue, 13 Jun 2023 22:06:53 UTC |
| 6488e87dcd89d51113579fe7 | succeeded (1s elapsed) | dummy_python | dummy_pack.dummy_python_action | Tue, 13 Jun 2023 22:06:53 UTC |
| 6488e87ecd89d51113579fff | succeeded (0s elapsed) | nothing      | core.noop                        | Tue, 13 Jun 2023 22:06:54 UTC |
+--------------------------+------------------------+--------------+----------------------------------+-------------------------------+

The output is as expected. The dummy_python task fails for host1. The workflow succeeds because it ends with a noop action that always succeeds. I change the datastore value to true so that upon re-run host1 will succeed:

st2 key set test_host1 true
+------------------+--------------+
| Property         | Value        |
+------------------+--------------+
| name             | test_host1   |
| value            | true         |
| scope            | st2kv.system |
| expire_timestamp |              |
+------------------+--------------+

And execute the re-run:

st2 execution re-run 6488e87c279a956d7fb7786c --tasks dummy_python --no-reset dummy_python
.
id: 6488e9a9279a956d7fb7786f
action.ref: dummy_pack.dummy_action2
parameters:
  hostnames:
  - host1
  - host2
status: succeeded
start_timestamp: Tue, 13 Jun 2023 22:11:52 UTC
end_timestamp: Tue, 13 Jun 2023 22:11:54 UTC
log:
  - status: requested
    timestamp: '2023-06-13T22:11:52.999000Z'
  - status: scheduled
    timestamp: '2023-06-13T22:11:53.144000Z'
  - status: running
    timestamp: '2023-06-13T22:11:53.173000Z'
  - status: succeeded
    timestamp: '2023-06-13T22:11:54.696000Z'
result:
  output:
    python_result:
      end_timestamp: '2023-06-13 22:06:54.313670+00:00'
      result:
        items:
        - result:
            exit_code: 0
            result:
              host_pass_fail_list:
              - host: host1
                status: failed
                uuid: e35183f3-1355-4190-b457-296add2df3b6
            stderr: 'st2.actions.python.DummyAction: DEBUG    Creating new Client object.
              st2.actions.python.DummyAction: DEBUG    Retrieving value from the datastore (name=test_host1)
              '
            stdout: ''
          status: failed
        - result:
            exit_code: 0
            result:
              host_pass_fail_list:
              - host: host2
                status: passed
                uuid: 93f3c9af-f1ef-4dee-9d68-75d903a125d9
            stderr: 'st2.actions.python.DummyAction: DEBUG    Creating new Client object.
              st2.actions.python.DummyAction: DEBUG    Retrieving value from the datastore (name=test_host1)
              '
            stdout: ''
          status: succeeded
      route: 0
      start_timestamp: '2023-06-13 22:06:53.381685+00:00'
      status: failed
      task_execution_id: 6488e87dcd89d51113579fdf
      task_id: dummy_python
      task_name: dummy_python
      workflow_execution_id: 6488e87df5c87ffc2f94b693
+--------------------------+------------------------+--------------+----------------------------------+-------------------------------+
| id                       | status                 | task         | action                           | start_timestamp               |
+--------------------------+------------------------+--------------+----------------------------------+-------------------------------+
| 6488e9a9cd89d5111357a019 | succeeded (1s elapsed) | dummy_python | dummy_pack.dummy_python_action | Tue, 13 Jun 2023 22:11:53 UTC |
| 6488e9aacd89d5111357a02a | succeeded (0s elapsed) | nothing      | core.noop                        | Tue, 13 Jun 2023 22:11:54 UTC |
| 6488e9aacd89d5111357a030 | succeeded (0s elapsed) | nothing      | core.noop                        | Tue, 13 Jun 2023 22:11:54 UTC |
+--------------------------+------------------------+--------------+----------------------------------+-------------------------------+

The output of these to executions, the original and the re-run are identical, save timestamps and the table calling out the actions that were actually run in the workflow. Here is the diff output:

diff <(st2 execution get 6488e87c279a956d7fb7786c) <(st2 execution get 6488e9a9279a956d7fb7786f)
1c1
< id: 6488e87c279a956d7fb7786c
---
> id: 6488e9a9279a956d7fb7786f
8,9c8,9
< start_timestamp: Tue, 13 Jun 2023 22:06:52 UTC
< end_timestamp: Tue, 13 Jun 2023 22:06:54 UTC
---
> start_timestamp: Tue, 13 Jun 2023 22:11:52 UTC
> end_timestamp: Tue, 13 Jun 2023 22:11:54 UTC
12c12
<     timestamp: '2023-06-13T22:06:52.965000Z'
---
>     timestamp: '2023-06-13T22:11:52.999000Z'
14c14
<     timestamp: '2023-06-13T22:06:53.110000Z'
---
>     timestamp: '2023-06-13T22:11:53.144000Z'
16c16
<     timestamp: '2023-06-13T22:06:53.138000Z'
---
>     timestamp: '2023-06-13T22:11:53.173000Z'
18c18
<     timestamp: '2023-06-13T22:06:54.706000Z'
---
>     timestamp: '2023-06-13T22:11:54.696000Z'
59,61c59,61
< | 6488e87dcd89d51113579fe2 | failed (1s elapsed)    | dummy_python | dummy_pack.dummy_python_action | Tue, 13 Jun 2023 22:06:53 UTC |
< | 6488e87dcd89d51113579fe7 | succeeded (1s elapsed) | dummy_python | dummy_pack.dummy_python_action | Tue, 13 Jun 2023 22:06:53 UTC |
< | 6488e87ecd89d51113579fff | succeeded (0s elapsed) | nothing      | core.noop                        | Tue, 13 Jun 2023 22:06:54 UTC |
---
> | 6488e9a9cd89d5111357a019 | succeeded (1s elapsed) | dummy_python | dummy_pack.dummy_python_action | Tue, 13 Jun 2023 22:11:53 UTC |
> | 6488e9aacd89d5111357a02a | succeeded (0s elapsed) | nothing      | core.noop                        | Tue, 13 Jun 2023 22:11:54 UTC |
> | 6488e9aacd89d5111357a030 | succeeded (0s elapsed) | nothing      | core.noop                        | Tue, 13 Jun 2023 22:11:54 UTC |

There are two things that prick my senses. The biggest thing by far, is that, the new execution result/output is no where to be found. The now succeeded second execution of dummy_python_action against host1 has a result/output, but it is no where to be found (unless I specifically pull it using st2). The fact that the re-run execution doesn't have it's output/result updated means that it isn't possible to consume that output to resume a workflow that has tasks beyond these, that rely on that output.

I'm uncertain if this is intentional. It is a hinderance for my use-case so I'll need to figure out a way to work around it, but if it is in fact unintentional, I'd be happy to file a bug (and hopefully find some time to work on it). The scenario I've created above, is a best-case for this particular situation. As I mentioned, in cases where subsequent actions actually try to make use of the output from the re-run task, it often results in various TypeErrors:

    {
      "type": "error",
      "message": "TypeError: cannot pickle 'StrictUndefined' object",
      "task_id": "dummy_python",
      "route": 0
    }

With all of this said, the TLDR:

1. Should re-run executions be updated with the outcomes produced by the tasks encapsulated by it? (I think so)
2. Is there just something wrong with my invocation?

[guzzijones]

re-run using a specific task never worked for me either.
I also submitted an issue to enhance rerun as well for child workflows.

[guzzijones]

a possible workaround is to put the retry in your task in the workflow if you expect network gremlins to cause failures.