Skip to content

[BUG] Workflow deadlocks indefinitely when task fails and blocks dependent tasks #325

Description

@mcosicaws

Checks

  • I have updated to the lastest minor and patch version of Strands
  • I have checked the documentation and this is not expected behavior
  • I have searched ./issues and there are no duplicates of my issue

Strands Version

1.18.0

Tools Package Version

0.2.16

Tools used

Workflow

Python Version

3.13.2

Operating System

macOS 15.7

Installation Method

pip

Steps to Reproduce

The workflow tool has a critical deadlock bug when a task exhausts all retry attempts and fails, blocking dependent tasks from running. The workflow enters an infinite loop waiting for tasks that can never complete.

from strands import Agent, tool
from strands_tools import workflow
from strands.models import BedrockModel

@tool
def task_that_succeeds():
    """Task that always succeeds."""
    return "Success!"

model = BedrockModel(model_id="us.anthropic.claude-sonnet-4-5-20250929-v1:0")
agent = Agent(model=model, tools=[workflow, task_that_succeeds])

# Create workflow with dependencies
agent.tool.workflow(
    action="create",
    workflow_id="deadlock_test",
    agent=agent,
    tasks=[
        {
            "task_id": "research",
            "description": "Call task_that_succeeds",
            "tools": ["task_that_succeeds"]
        },
        {
            "task_id": "analysis1",
            "description": "Call task_that_succeeds",
            "tools": ["task_that_succeeds"],
            "dependencies": ["research"]
        },
        {
            "task_id": "analysis2",
            "description": "Call task_that_succeeds",
            "tools": ["task_that_succeeds"],
            "dependencies": ["research"],
            "system_prompt": "TRIGGER_FAILURE_FOR_TESTING"  # Special marker to trigger API failures
        },
        {
            "task_id": "report",
            "description": "Call task_that_succeeds",
            "tools": ["task_that_succeeds"],
            "dependencies": ["analysis1", "analysis2"]  # Blocked by analysis2 failure
        }
    ]
)

# This will deadlock indefinitely when analysis2 exhausts retries due to API throttling
agent.tool.workflow(action="start", workflow_id="deadlock_test", agent=agent)

The task fails when the Bedrock API calls are throttled repeatedly, exhausting the @Retry decorator's 5 attempts with exponential backoff. This simulates real-world scenarios where:

  • API rate limits are exceeded
  • Service throttling occurs
  • Any transient error that persists beyond retry attempts

Expected Behavior

When a task fails and blocks dependent tasks:

  1. The workflow should detect the deadlock condition (no tasks running, no tasks can run)
  2. Mark blocked tasks as "skipped" or "blocked"
  3. Return a partial success result with status information
  4. Complete execution gracefully

Example expected output:
Workflow completed with partial success:

  • research: completed ✅
  • analysis1: completed ✅
  • analysis2: error ❌
  • report: skipped (dependency failed) ⏸️
    Success rate: 50% (2/4 tasks)

Actual Behavior

The workflow enters an infinite loop:

  1. research: status="completed" → added to completed_tasks (count: 1)
  2. analysis1: status="completed" → added to completed_tasks (count: 2)
  3. analysis2: status="error" → added to completed_tasks (count: 3)
  4. report: never runs (dependency check fails: workflow["task_results"]["analysis2"]["status"] != "completed")
  5. Loop continues: len(completed_tasks) = 3 < total_tasks = 4
  6. No ready tasks, no active futures
  7. Infinite loop with time.sleep(0.1) until external timeout

The workflow never returns a result and blocks the execution environment until an external timeout (e.g., AgentCore 60s timeout, Lambda timeout) kills the process.

In the log, we can see an error message coming from this line:
logger.error(f"❌ Task '{task_id}' failed: {str(e)}")

Additional Context

File: strands_tools/workflow.py

Line 628: Loop condition

while len(completed_tasks) < total_tasks:

Line 593-595: Dependency check

for dep_id in task["dependencies"]:
    if workflow["task_results"][dep_id]["status"] != "completed":
        dependencies_met = False

The problem:

  • Failed tasks have status "error" (not "completed")
  • Dependent tasks check for status == "completed", so they never become "ready"
  • Failed tasks ARE added to completed_tasks set
  • But blocked tasks are NOT added to completed_tasks
  • Loop waits for completed_tasks == total_tasks, which can never be true

Line 656-659: No deadlock detection

if active_futures:
    done, _ = wait(active_futures.values(), return_when=FIRST_COMPLETED)

When active_futures is empty and no tasks are ready, the loop just spins with time.sleep(0.1).

Tested with:

  • AWS Bedrock Claude Sonnet 4.5
  • Multi-task workflows with parallel execution
  • Simulated task failures via API throttling (100% rate to exhaust retries)
  • Confirmed deadlock with 5-20 minute timeouts

Possible Solution

Add deadlock detection in the workflow loop:

while len(completed_tasks) < total_tasks:
    ready_tasks = self.get_ready_tasks(workflow)

    # Deadlock detection
    if not active_futures and not ready_tasks:
        # No tasks running and no tasks can run - deadlock detected
        logger.warning(f"Deadlock detected: {len(completed_tasks)}/{total_tasks} tasks completed, "
                      f"{total_tasks - len(completed_tasks)} tasks blocked")

        # Mark blocked tasks as skipped
        for task in workflow["tasks"]:
            task_id = task["task_id"]
            if task_id not in completed_tasks:
                workflow["task_results"][task_id] = {
                    **workflow["task_results"][task_id],
                    "status": "skipped",
                    "result": [{"text": "Task skipped due to failed dependency"}],
                    "completed_at": datetime.now(timezone.utc).isoformat(),
                }
                completed_tasks.add(task_id)
        break

    # ... rest of loop

Related Issues

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    Language

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions