Skip to content

add CI tests for openhands-getting-started playbook - #615

Open
harkgill-amd wants to merge 11 commits into
mainfrom
users/harkgill/openhands-ci-tests
Open

add CI tests for openhands-getting-started playbook#615
harkgill-amd wants to merge 11 commits into
mainfrom
users/harkgill/openhands-ci-tests

Conversation

@harkgill-amd

@harkgill-amd harkgill-amd commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Addresses #601.

Added tests for openhands CI following test patterns from existing playbooks (most notable n8ns and other lemonade based playbooks).

Temporarily forcing uv install through CI tests to ensure consistency across all runners.

  ┌──────────────────────────┬───────────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Test ID (×linux/windows) │           Step            │                                                                  What it validates                                                                   │
  ├──────────────────────────┼───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ prereq-clis-*            │ Prerequisites             │ lemonade, node, npm, uv all present                                                                                                                  │
  ├──────────────────────────┼───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ lemonade-chat-*          │    Verify the Local Model │ Lemonade /api/v1/models responds; model is present + downloaded; deterministic "Reply with exactly: OK" chat (temperature:0, max_tokens:32) succeeds │
  ├──────────────────────────┼───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ agent-canvas-install-*   │    Install                │ npm install -g @openhands/agent-canvas succeeds (user-owned npm prefix on Linux, no root needed)                                                     │
  ├──────────────────────────┼───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ agent-canvas-version-*   │    Install                │ agent-canvas CLI is on PATH (--version, falling back to --help)                                                                                      │
  ├──────────────────────────┼───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ agent-canvas-server-*    │    Launch                 │ Server starts, http://127.0.0.1:8000/ returns HTTP 200, then the process is killed cleanly (frees the shared runner)                                 │
  ├──────────────────────────┼───────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ lemonade-unload-*        │ Cleanup                   │ lemonade unload frees the GPU pool for the next job                                                                                                  │
  └──────────────────────────┴───────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

@harkgill-amd
harkgill-amd requested a review from sreeram-11 July 21, 2026 20:44
@harkgill-amd

Copy link
Copy Markdown
Collaborator Author

Hey @neubig, our Windows CI runners are failing to execute the playbook commands with the below logs from the agent-canvas step,

  [1/2] Checking prerequisites... 
  [2/2] Starting services...
  [agent-server] Starting on port 18000...
  [agent-server] Using PyPI (1.36.1, default)
  [agent-server] The system cannot find the file specified.
  [agent-server] Exited with code 1                                  
  [agent-server] Timeout waiting for http://localhost:18000/server_info after 60s
  [agent-server] Last error: fetch failed
  [secrets] Skipping secret seeding - agent-server not ready
  [automation] Building litellm==1.93.0
  [automation] × Failed to build `litellm==1.93.0`
  [automation]   ╰─▶ Call to `maturin.build_wheel` failed (exit code: 1)
  [automation]       Rust not found, installing into a temporary directory

The frontend comes up but the agent-server backend never starts due to a missing litellm dependency. Noticing the documentation at https://github.com/OpenHands/openhands#quickstart points to a docker based setup for Windows systems. Do we want to go this route for the playbook as well or is there a recommended solution for the environment setup issues seen on a native Windows install?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant