iframes

[seanmcguire12]

why

• addresses stagehand's longest standing issue: Support iframes #37

What changed

This PR adds full support for interacting with nested iframes in
Stagehand—across extract, observe, and act by:

• Building a combined accessibility tree (and URL/XPath mappings)
  spanning the main document and any nested iframes.
• Tracking frame‑scoped element IDs so that element identifiers are
  globally unique across frames. This is done because backendNodeId's are not guaranteed to be unique across OOPIF's (out of process iframes).
• Extending CDP session management to correctly target Out‑Of‑Process
  iframes (OOPIFs) and fallback to same‑process frames.
• Introducing a “deep” XPath‑based locator that can step into <iframe>
  elements when performing Playwright actions.
• Updating Zod schema transforms to expect and handle the new
  “frameId-backendId” string format for element IDs.
• Adding new error types for improved diagnostics when iframe resolution
  or XPath lookups fail.
• Updating internal types (context.ts, stagehand.ts,
  stagehandErrors.ts) and utility modules (a11y/utils.ts, utils.ts,
  handlers) to accommodate frame‑aware operations.
• Adding three end‑to‑end eval tasks testing iframe support:
  • iframe_hn (extract)
  • iframe_same_proc (act)
  • iframe_form_filling (act).

Details

1. Frame‑Scoped Element IDs (EncodedId)

• lib/StagehandPage.ts
  • Introduced encodeWithFrameId(…), ordinalForFrameId(…), and
    resetFrameOrdinals() to assign and track per‑frame ordinals.
  • CDP session caching moved to a WeakMap<Page|Frame, CDPSession> so we
    can open sessions against arbitrary frames.
• types/context.ts
  • Defined EncodedId = ${number}-${number} for
    “frameOrdinal-backendNodeId” IDs.
  • Updated TreeResult to key xpathMap/idToUrl by EncodedId.

2. Combined Accessibility Tree Across Frames

• lib/a11y/utils.ts
  • Added getAccessibilityTreeWithFrames() which walks the CDP frame
    tree, captures accessibility sub‑trees for each frame, and concatenates them
    into a single “combinedTree” string plus combined URL/XPath maps keyed by
    EncodedId.
  • Updated formatSimplifiedTree() to emit the new encodedId in tree
    lines.
  • Updated buildBackendIdMaps() to traverse nested frame DOM nodes
    (OOPIF and same‑process iframes) and include the frame’s frameId when encoding
    backend IDs.

3. Deep XPath Locator for Frame Actions

• lib/handlers/handlerUtils/actHandlerUtils.ts
  Added deepLocator(root, rawXPath) which splits an XPath on <iframe>
  steps to descend into FrameLocators automatically before applying the
  remainder of the path.
• lib/handlers/actHandler.ts
  Uses deepLocator() instead of a flat page.locator(...) so that
  Playwright actions can target elements inside iframes when options.iframes is
  set.

4. Frame‑Aware Extract & Observe Handlers

• lib/handlers/extractHandler.ts
  • Imports and uses getAccessibilityTreeWithFrames() when iframes: true; otherwise falls back to the legacy single‑frame tree.
  • Passes through the iframes flag into its internal calls.
• lib/handlers/observeHandler.ts
  • Similarly leverages getAccessibilityTreeWithFrames() and builds its
    element‑to‑XPath mapping from the combined tree.
  • Removes the old ad‑hoc iframe injection logic in favor of the unified
    combined tree approach.

5. Inference Schema & Element ID Changes

• lib/inference.ts
  • Changed the Observe LLM output schema so elementId is now a
    string matching the regex /^\d+-\d+$/ (frame‑ID plus backend‑ID) instead
    of a raw number.
  • Updated parsing logic to no longer coerce to Number.

6. Zod URL‑Field Transformation

• lib/utils.ts
  • Renamed makeIdNumberSchema() → makeIdStringSchema() to emit
    z.string().regex(/^\d+-\d+$/) for fields that were formerly string().url(),
    so extracted URL placeholders match the new EncodedId format.
  • Updated injectUrls() to map both numeric and frame‑ID strings back
    into real URLs once extraction is complete.

to do:

• clean up code
• add JSDoc
• add comprehensive PR desc
• parameterize iframe traversal. ie, need to set iframes: true
• ignore iframes without content
• add evals

[changeset-bot]

🦋 Changeset detected

Latest commit: c17a209

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[greptile-apps]

PR Summary

This PR implements comprehensive iframe support in Stagehand, enabling interaction with nested iframes across extract, observe, and act operations. The changes introduce frame-scoped element IDs, combined accessibility trees, and CDP session management for both same-process and out-of-process iframes.

• Added frame-scoped element IDs using new EncodedId format (${frameOrdinal}-${backendNodeId}) for unique element identification across frames
• Implemented getAccessibilityTreeWithFrames() to build combined accessibility trees spanning main document and nested iframes
• Added deepLocator function in actHandlerUtils.ts to handle XPath selectors traversing through iframes
• Added three comprehensive eval tasks (iframe_hn, iframe_same_proc, iframe_form_filling) to test iframe functionality
• Fixed redundant error check in iframe_hn.ts where the same condition is checked twice

_{16 file(s) reviewed, 16 comment(s)
Edit PR Review Bot Settings | Greptile}

[miguelg719]

working through it, another note before approving is adding this behind experimental:true

[miguelg719]

CDP Goat status