Skip to content

Add MCP workflow telemetry for AI plugins#2979

Open
glendaviesnz wants to merge 1 commit intotrunkfrom
add-agent-plugin-telemetry
Open

Add MCP workflow telemetry for AI plugins#2979
glendaviesnz wants to merge 1 commit intotrunkfrom
add-agent-plugin-telemetry

Conversation

@glendaviesnz
Copy link
Copy Markdown

@glendaviesnz glendaviesnz commented Apr 6, 2026

Summary

This adds plugin-aware MCP workflow telemetry for Studio's AI integrations. This is needed so we can track usage of the Studio MCP by external agents

Changes include:

  • adding a hidden studio mcp --telemetry-group <group> option so different AI plugin integrations can identify themselves when they launch the Studio MCP server
  • extending the Studio CLI bump-stat types to allow plugin telemetry groups and workflow-stage stats
  • adding a new record_workflow_event MCP tool that records workflow milestones like theme-build-started, theme-build-completed, and theme-build-failed
  • threading the configured telemetry group through the MCP startup path so the tool can record events without the caller having to pass the plugin identity every time
  • updating the Studio AI UI/system prompt metadata so the new tool is surfaced cleanly in the built-in assistant
  • adding targeted MCP tool tests for configured and unconfigured telemetry behavior

This is designed to support plugin-grouped telemetry such as:

  • codex-plugin + theme-build-started
  • claude-code-plugin + theme-build-completed

Testing

Automated

  • npm test -- apps/cli/ai/tests/tools.test.ts

Manual

  1. Launch Studio MCP with a telemetry group, for example:
    • studio mcp --telemetry-group codex-plugin
  2. Connect an MCP client and confirm the record_workflow_event tool is available.
  3. Call the tool with:
    • workflow: theme-build
    • stage: started
  4. Confirm the tool returns a success message showing the plugin group and stat name.
  5. Repeat with completed and failed stages and with another workflow such as site-build.
  6. Launch studio mcp without --telemetry-group and confirm the tool reports that workflow telemetry was skipped because no plugin group was configured.

Notes

  • I also ran npx tsc -p apps/cli/tsconfig.json --noEmit, but it still reports an existing unrelated memfs test typing issue in apps/cli/lib/tests/snapshots.test.ts.

@glendaviesnz glendaviesnz self-assigned this Apr 6, 2026
@wpmobilebot
Copy link
Copy Markdown
Collaborator

📊 Performance Test Results

Comparing f424ab3 vs trunk

app-size

Metric trunk f424ab3 Diff Change
App Size (Mac) 1263.82 MB 1263.82 MB +0.00 MB ⚪ 0.0%

site-editor

Metric trunk f424ab3 Diff Change
load 1881 ms 1559 ms 322 ms 🟢 -17.1%

site-startup

Metric trunk f424ab3 Diff Change
siteCreation 8194 ms 9184 ms +990 ms 🔴 12.1%
siteStartup 4932 ms 4952 ms +20 ms ⚪ 0.0%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants