Eval: Add Storybook Dev context, rename Storybook MCP to Storybook Docs #109

JReinhold · 2025-12-09T12:14:58Z

This PR introduces another context type: Storybook MCP - Dev.
When using that, Storybook is set up in the experiment, its dev server is started and the dev server's MCP server is configured for the agent.

The existing Storybook MCP context has been renamed to Storybook MCP - Docs, to signify it's the docs-only (ie "remote") Storybook MCP version.

changeset-bot · 2025-12-09T12:15:02Z

⚠️ No Changeset found

Latest commit: 88a9cba

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

💥 An error occurred when fetching the changed packages and changesets in this PR

Some errors occurred when validating the changesets config:
The package or glob expression "@storybook/mcp-eval*" is specified in the `ignore` option but it is not found in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.

…-eval

pkg-pr-new · 2025-12-09T12:20:42Z

npm i https://pkg.pr.new/storybookjs/mcp/@storybook/addon-mcp@109

npm i https://pkg.pr.new/storybookjs/mcp/@storybook/mcp@109

commit: 88a9cba

codecov · 2025-12-09T12:20:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.82%. Comparing base (4648a4b) to head (88a9cba).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #109   +/-   ##
=======================================
  Coverage   87.82%   87.82%           
=======================================
  Files          19       19           
  Lines         427      427           
  Branches      122      122           
=======================================
  Hits          375      375           
  Misses          8        8           
  Partials       44       44

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

JReinhold · 2025-12-09T12:29:54Z

eval/lib/agents/claude-code-cli.ts

+			// Check for MCP server status in init message
+			if (
+				parsed.type === 'system' &&
+				parsed.subtype === 'init' &&
+				parsed.mcp_servers &&
+				parsed.mcp_servers.length > 0
+			) {
+				for (const server of parsed.mcp_servers) {
+					if (server.status === 'connected') {
+						clackLog.success(`MCP server "${server.name}" connected`);
+					} else {
+						clackLog.error(
+							`MCP server "${server.name}" failed to connect (status: ${server.status})`,
+						);
+						process.exit(1);
+					}
+				}
+			}
+


This is a small QoL improvement, logging which MCP server actually successfully connected. If any MCP server is configured but failed to connect, it bails the whole eval early. I doubt that is ever the intention anyway.

Copilot

Pull request overview

This PR introduces a new "Storybook MCP - Dev" context mode that sets up and runs a local Storybook development server with the MCP addon, while renaming the existing remote manifest-based context to "Storybook MCP - Docs" for clarity. This enables testing agents in a live development environment versus documentation-only scenarios.

Key Changes

New storybook-mcp-dev context type that installs Storybook, starts a dev server on a random port, and provides HTTP MCP endpoint
Added dev server lifecycle management with automatic startup during experiment preparation and teardown after agent execution
Renamed existing Storybook MCP context to "Storybook MCP - Docs" throughout UI, documentation, and code

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
eval/types.ts	Adds `storybook-mcp-dev` context type definition
eval/lib/storybook-dev-server.ts	New module for managing Storybook dev server lifecycle (port allocation, process management, readiness detection)
eval/lib/prepare-experiment.ts	Enhanced to install Storybook packages and start dev server for the new context, returns MCP config
eval/lib/teardown-experiment.ts	New cleanup module to stop dev server after experiment
eval/lib/agents/claude-code-cli.ts	Added MCP server connection validation and logging
eval/lib/collect-args.ts	Updated CLI parsing and help text to support new context mode
eval/eval.ts	Integrates teardown step and uses prepared MCP config from experiment setup
eval/lib/save/google-sheet.ts	Handles new context type in reporting
eval/lib/evaluations/prepare-evaluations.ts	Adds `@storybook/addon-mcp` to evaluation dependencies
eval/templates/evaluation/.storybook/main.ts	Simplified config, removed `getAbsolutePath` helper, added `@storybook/addon-mcp`
eval/README.md	Documentation updates for five context modes instead of four
.github/instructions/eval.instructions.md	Comprehensive documentation of new context mode and architecture
eval/evals/100-flight-booking-plain/prompt.md	Minor formatting fix (extra blank line)
eval/evals/100-flight-booking-plain/explicit-stories.md	New file with story-writing reminder

eval/lib/storybook-dev-server.ts

eval/eval.ts

eval/lib/prepare-experiment.ts

eval/eval.ts

eval/lib/teardown-experiment.ts

eval/lib/agents/claude-code-cli.ts

eval/lib/prepare-experiment.ts

valentinpalkovic · 2025-12-09T12:39:33Z

eval/lib/agents/claude-code-cli.ts

+				parsed.mcp_servers.length > 0
+			) {
+				for (const server of parsed.mcp_servers) {
+					if (server.status === 'connected') {


At this stage, could the MCP server be pending? Could you let us know how we know it has successfully connected at this stage?

good question. I added a comment clarifying what this connection status means.

this is the status as reported by the agent, so this is the source of truth. The agent initialises connections with all servers when the session starts, and if any servers are not ready by this point, they won't be used AFAIK.

valentinpalkovic · 2025-12-09T12:59:29Z

eval/lib/save/google-sheet.ts

 };

 function getContextDetails(context: Context): string {
 	switch (context.type) {


Technical improvement:
I personally like to have exhausting type checks for the default fall-through case, like demonstrated here: https://www.typescriptlang.org/play/?#code/KYOwrgtgBAwg9gGzgJygbwFBSgJWAEwBosoBxZYUY7AIQTGAwF8MMAzMEAYwBcBLOCCgBzYD3hJkAOQCGEYAAouALliIUASlUBnHsj4hh6EtoDufHlwAWUJRuPZsXGduBrJAOjz5lJR1AoeMGQhACIKfFCAbj8oZ1d3FA9yShBff2xA4LDhClBo2IB6QqgAMRRhOB4oGQAjODBqugZY-GA2GTAEHnSMrkFdKGAADytO3T4AN2AYK2AuAGtVEGBp1ABeOJiMqB4rZDhTKBWjgFFkA+QFAAMAVRAxkHwEAjj1VHjgVQASNBGxsATaazeYLJjXDTbKAsJhAA

So whenever the union context.type is extended, TypeScript will error to signal that adjustments are necessary here as well

That's a good idea. However it's already covered without the default case, because the function's return type errors in that scenario too, without a default: https://www.typescriptlang.org/play/?#code/KYOwrgtgBAwg9gGzgJygbwFBSgJWAEwBosoBxZYUY7AIQTGAwF8MMAzMEAYwBcBLOCCgBzYD3hJkAOQCGEYAAouALliIUASlUBnHsj4hh6EtoDufHlwAWUJRuPZsXGduBrJAOjz5lJR1AoeMGQhACIKfFCAbj8oZ1d3FA9yShBff2xA4LDhClBo2IB6QqgAMRRhOB4oGQAjODBqugYSFiYgA

which is also the case for this code specifically.

however I did something that I like even better, I configured the lint rule to handle this:

https://github.com/storybookjs/mcp/pull/109/files#diff-2113aef2ff3d50c021fca1fb6ecf4d4553a0e3f5f51ded7302ad499e43bc2b1eR3

https://oxc.rs/docs/guide/usage/linter/rules/typescript/switch-exhaustiveness-check.html

So we don't have to remember to add the runtime code to ensure switches are exhaustive.

eval/lib/collect-args.ts

valentinpalkovic · 2025-12-09T14:56:31Z

eval/lib/storybook-dev-server.ts

+	process: Result,
+	timeout: number = 10_000,
+): Promise<void> {
+	return await new Promise<void>((resolve, reject) => {


Why are you returning a new Promise inside of an async function, instead of using await inside and throwing errors directly? Is it because of the onExit callback function, which is passed to the off,on event listeners?

it's the basic pattern of turning callbacks into promises - onData, onExit and setTimeout. I don't think there's any other way of doing it, except for Promise.withResolvers which I'm fine with too.

Even though the function is async, there's nothing we could await here.

Co-authored-by: Copilot <[email protected]>

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

eval/lib/storybook-dev-server.ts

eval/lib/prepare-experiment.ts

eval/evals/100-flight-booking-plain/explicit-stories.md

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

… sb-dev-eval

JReinhold added 14 commits December 4, 2025 13:52

add uploadId to args and sheets

09626d9

add support for .env file

f332234

fixes

ff71a96

fix changeset ignore id

929aabd

simplify cli -> env arg parsing

5a9b39b

use commander.js for simpler args collection

97de539

cleanup

36291f1

clean experiments

dac5d16

cleanup

8da5a4d

cleanup

e7a1083

add initial storybook dev based eval

115ed30

log connected MCP servers, exit early if mcp servers failed to connect

d3b4a76

override the mcp config if Storybook MCP is connected

5d50e28

add Storybook Dev context, rename Storybook MCP to Storybook Docs

d729375

Merge branch 'main' of https://github.com/storybookjs/mcp into sb-dev…

8f673e2

…-eval

JReinhold added 2 commits December 9, 2025 13:24

cleanup

f48ae60

update docs

a616195

JReinhold commented Dec 9, 2025

View reviewed changes

JReinhold marked this pull request as ready for review December 9, 2025 12:33

Copilot AI review requested due to automatic review settings December 9, 2025 12:33

Copilot started reviewing on behalf of JReinhold December 9, 2025 12:34 View session

Copilot AI reviewed Dec 9, 2025

View reviewed changes

valentinpalkovic reviewed Dec 9, 2025

View reviewed changes

JReinhold changed the title ~~add Storybook Dev context, rename Storybook MCP to Storybook Docs~~ Eval: Add Storybook Dev context, rename Storybook MCP to Storybook Doc Dec 9, 2025

JReinhold changed the title ~~Eval: Add Storybook Dev context, rename Storybook MCP to Storybook Doc~~ Eval: Add Storybook Dev context, rename Storybook MCP to Storybook Docs Dec 9, 2025

Update eval/eval.ts

659c89d

Co-authored-by: Copilot <[email protected]>

Copilot AI review requested due to automatic review settings December 9, 2025 19:07

Copilot started reviewing on behalf of JReinhold December 9, 2025 19:07 View session

Copilot AI reviewed Dec 9, 2025

View reviewed changes

eval/lib/storybook-dev-server.ts Show resolved Hide resolved

eval/lib/prepare-experiment.ts Show resolved Hide resolved

eval/evals/100-flight-booking-plain/explicit-stories.md Show resolved Hide resolved

Copilot AI reviewed Dec 9, 2025

View reviewed changes

JReinhold added 2 commits December 9, 2025 20:26

address feedback

030bcd1

Merge branch 'sb-dev-eval' of https://github.com/storybookjs/mcp into…

88a9cba

… sb-dev-eval

JReinhold requested a review from valentinpalkovic December 9, 2025 19:35

valentinpalkovic approved these changes Dec 9, 2025

View reviewed changes

JReinhold merged commit c2c7920 into main Dec 10, 2025
11 checks passed

JReinhold deleted the sb-dev-eval branch December 10, 2025 10:32

Eval: Add Storybook Dev context, rename Storybook MCP to Storybook Docs #109

Eval: Add Storybook Dev context, rename Storybook MCP to Storybook Docs #109

Uh oh!

Conversation

JReinhold commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot bot commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

pkg-pr-new bot commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JReinhold Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

valentinpalkovic Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

JReinhold Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

valentinpalkovic Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

JReinhold Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

valentinpalkovic Dec 9, 2025

Choose a reason for hiding this comment

Uh oh!

JReinhold Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JReinhold commented Dec 9, 2025 •

edited

Loading

changeset-bot bot commented Dec 9, 2025 •

edited

Loading

pkg-pr-new bot commented Dec 9, 2025 •

edited

Loading

codecov bot commented Dec 9, 2025 •

edited

Loading

JReinhold Dec 9, 2025 •

edited

Loading