[backend] Terminating a run causes missing artifacts (logs) on the Kubeflow UI

[ameya-parab]

Environment

• How did you deploy Kubeflow Pipelines (KFP)?

Deployed using the raw 1.8 manifest on AWS EKS 1.29 cluster

• KFP version: v1.8.1
• KFP SDK version: 1.8.22

Steps to reproduce

When a running workflow is terminated from the UI, the main.log Output Artifact is not displayed on the UI, even though the Pod that runs the task pushed the local file to the Log repository (S3 in our case).

import kfp
from kfp import dsl
from kfp.components import create_component_from_func


def sleep_and_print_time_func():

    import time
    from datetime import datetime

    total_time = 300  # 5 minutes in seconds
    interval = 5  # print every 5 seconds

    print(
        "Starting sleep component. Will run for 5 minutes, printing time every 5 seconds."
    )

    for i in range(0, total_time, interval):
        current_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
        print(f"Current time: {current_time}")

        # Don't sleep after the last iteration
        if i < total_time - interval:
            time.sleep(interval)

    print("Sleep component completed.")

sleep_and_print_component = create_component_from_func(
    func=sleep_and_print_time_func, base_image="python:3.7"
)


# Define the pipeline
@kfp.dsl.pipeline(
    name="Sleep",
)
def sleep_print_pipeline():
    sleep_task = (
        sleep_and_print_component().set_cpu_request("0.5").set_memory_limit("64Mi")
    )

# Trigger the pipeline
client = kfp.Client()
client.create_run_from_pipeline_func(sleep_print_pipeline, arguments={})

When the above workflow is 'Running' and the user terminates it from the UI, the workflow is marked as failed, as expected, but the main-log is not shown in the 'Input\Output' tab.

Expected result

Even after the workflow is terminated, the main.log file should remain in the task's 'Input/Output' tab.

Materials and Reference

---

Impacted by this bug? Give it a 👍.