[Tracking]: Storybook MCP Docs Validation

[valentinpalkovic]

Problem statement

In the previous cycle, we successfully built the Docs Toolset—a feature that provides component metadata (the manifest) to LLMs, enabling design system reuse. Initial benchmarks showed improved speed, cost, and quality compared to agents scanning raw code.

Goal: Stabilize the solution, implement crucial optimizations (token efficiency), validate performance against a large cohort, and streamline the EAP experience.

Milestones

M1: Token Efficiency - Markdown Output

Convert the Docs Toolset output from verbose XML to a Markdown-based format to significantly reduce token count and cost.

Owner: @JReinhold
Complete by: @date

• Implement and unit test the Markdown formatter, deprecating the XML output logic.
• Update the get-documentation tool to use the new Markdown formatter by default, ensuring the old XML formatter is removed or flagged for deprecation.

M2: Token Efficiency - Single-Component Retrieval

Refactor the get-component-documentation tool to only return docs for one component ID per request to prevent exceeding the $25,000$ token limit observed in LLMs like Claude

Owner: @JReinhold
Complete by: @date

• Modify the API contract and tool description for get-component-documentation to strictly accept only a single componentId instead of an array of IDs.
• Update the MCP server's exposed tool schema to correctly reflect the new, single-ID input requirement, ensuring agents like Claude are aware of the new constraint.

M3: Robust Import Resolution

Finalize and test the implementation for resolving package/path aliases (monorepos, barrel files) to ensure react-docgen successfully extracts prop data from component source files. (A known major blocker in many popular OSS Design Systems).

Owner: @valentinpalkovic
Complete by: @date

• Implement or extend the logic that resolves component source files, specifically handling tsconfig path aliases (e.g., imports starting with @/, ~, or package names in a monorepo).
• Questions for @kasperpeulen: Clarify how package name lookup works (per component vs per manifest) and discuss barrel file handling approach.
• Document what cases are NOT supported for barrel files (e.g., specific import patterns, re-export scenarios that fail).
• Ensure the logic correctly handles components imported via barrel files (index.ts files that re-export components) without erroring or pointing to transpiled files (which lack docgen data).
• Use the manifest debugger (components.html) to re-test known failing OSS design systems and confirm that $\ge 90%$ of components that previously failed due to unresolved imports/docgen issues are now successfully parsed.

M4: MDX Docs as Context

Implement the extraction and serving of MDX documentation content as context for the agent. (Requested multiple times in pre-EAP, as MDX often contains crucial setup and usage information).

Owner: @JReinhold
Complete by: @date

• Develop logic to locate and extract relevant raw Markdown/MDX content associated with components and general documentation pages during manifest generation.
• Establish a convention for identifying crucial setup documentation. Implement prioritization logic to include content from: a) The main component's dedicated MDX file (if any). b) Explicitly tagged "Setup" or "Getting Started" documentation pages (see next point).
• Suggestion: Explicit Tagging/Naming Convention. Update the manifest generator to recognize specific files or stories that contain critical setup instructions, based on one of the following convention: File Naming: Prioritize content from files named Introduction.mdx, GettingStarted.mdx, or Setup.mdx.
• Augment the get-component-documentation tool's output to include two separate, clearly delimited fields in the MCP payload: a) Component-specific Markdown b) System Setup Markdown. (or should we introduce even another tools endpoint to gather setup instructions for a particular Design System?)
• Prepare a sample EAP documentation guide instructing maintainers on how to use the convention (e.g., "Tag your main setup page with setup (do mdx files support 'tags'?!) to ensure the agent understands how to initialize your DS").

M5: EAP Onboarding & Support

Provide direct engineering support (bug fixing and diagnosis) for the EAP cohort to capture real-world friction points and stabilize the platform.

Owner: @JReinhold
Complete by: @date

• Set up a centralized database/system for issues and feedback collection to avoid being pinged across multiple Slack channels. Establish a clear process for EAP users to report issues.
• Prioritize incoming critical manifest/addon errors reported by EAP users and reproduce them on an internal test setup to isolate the root cause.
• Implement and test validated code fixes. Coordinate patch/prerelease distribution of @storybook/addon-mcp and Storybook core to EAP participants to minimize bug-related friction
• Guide EAP users to use the Manifest Debugger (components.html). Collect failing manifest logs and initial evaluation output files for diagnosis

M6: Tool & Prompt Optimization

Pressure test and optimize the tool descriptions and system prompts to maximize tool usage efficiency and quality of results across agents (Claude, Copilot).

Owner: @JReinhold
Complete by: @date

• Implement feedback collection mechanism during evaluation runs for repetitive test scenarios to capture agent behavior patterns and tool usage efficiency.
• Pressure test and optimize tool descriptions and system prompts based on collected feedback.

M7: Best Practices Documentation

DX: Document the best practices derived from the EAP and evaluations (e.g., "how to use JSDocs for LLMs," "common monorepo fix") for maintainers.

Owner: @mention
Complete by: @date

• Draft the main sections of the "Best Practices for LLMs" guide, focusing on common EAP errors (e.g., "The Case Against Barrel Files," "Why You Need @import").
• Develop clear, copy-pasteable code snippets demonstrating correct JSDoc usage (@import, @summary), component path configuration, and recommended CSF structure for maximum MCP reliability.
• Coordinate with the DX team for final technical, editorial, and design review. Prepare the guide for official publication, ensuring it clearly addresses maintainer friction points.

========= Everything below this line is strictly nice-to-have =========

Open Questions (Requiring a clear decision)

• Manifest Structure: Should we optimize the structure by splitting the single components.json into a listing file and manifest-per-component files? (A potential future optimization to reduce cold-start time and load only necessary data).

• Local/Remote API Merge: Should we attempt to merge the remote DS MCP and local Dev MCP into a single, unified API endpoint? (A complex DX improvement for agents that need to call both).

• Documentation Location: Where should the primary EAP/launch documentation live (e.g., dedicated Notion page, new docs/mcp section, blog post)?

• Do images provided by e.g. Chromatic provide some additional value for the LLM/User?

Nice-to-haves (Future Scope)

• Extend Renderer Support: Implement manifest generation for other renderers. E.g. web components based on the Custom Element Manifest format.
• DS Documentation Prompt: Develop a prompt that helps maintainers use an LLM to generate or refine their component documentation and JSDoc tags.
• Optimization Research: Conduct in-depth research on the difference between react-docgen and react-docgen-typescript in the context of manifest generation.