Make it possible to use webserial on Huggingchat

[haixuanTao]

This PR makes it possible to use webserial with Huggingchat making it possible to connect to things like the Huggingface Reachy Mini or the SO100 :)

[gary149]

I'm not very familiar with this, but did you test whether the browser's serial port permission prompt actually fires from within the sandboxed iframe, or does the sandbox block it silently?

[haixuanTao]

I'm not very familiar with this, but did you test whether the browser's serial port permission prompt actually fires from within the sandboxed iframe, or does the sandbox block it silently?

Yup: https://huggingface.co/chat/r/V7M5LZ0?leafId=ae8941e7-ab69-41cc-be2d-dce1c33db5bd

So basically without the PR, you can't access serial within the iframe.

[julien-c]

Not super familiar with this, but maybe better to ship an MCP adapter for webserial?

[haixuanTao]

So basically, webserial enables llm to generate code to connect to a robot through the web. The idea is that you can then have a webpage that control the robot as you may like with interaction, buttons and logic.

You could for example build a metronome app or a robot singer, directly from the chat and directly running from huggingchat ( so without having to install the map server, git clone, pip ... )

We could also use MCP calls to interact with the robot but it would make interaction limited to the function exposed by the MCP servers. It may also be less easy to create full-on apps as it will be mainly driven by the llm conversation which mean less room for button interactivity like choosing the speed of the metronome or which note to emphasize within a metronome app.

It should also be easier to run than having a local mcp server or local python app as everything is within the browser not requiring python installation or terminal that can be trickier for kids, parent or teachers...

It's also more secure as nothing run on the actual system and is sandboxed within a webpage. For example, less risk for the generated code or llm to do: rm -rf ~/ if a children say something crazy like delete everything.

We have enabled access of webserial within Huggingface Spaces and it might some cool demos already for the SO100 and this PR is in the continuation of those work.

I have an idea of a demo in mind of like huggingchat doing a zero-shot webpage generation that control Reachy for something simple like remote controlling Reachy mini with a mouse and I think it is pretty empowering for people who never programmed before.

Reference tweet: https://x.com/HaixuanT/status/1933519704800247902