Add websearch controls for assistants

[nsarrazin]

Added 4 options to assistant for RAG

• Disabled (will never use websearch)
• Enabled (will always use websearch)
• Links (user specifies up to 3 URLs that will be parsed directly)
• Domains (lets the user specify domains that will be used for searching the web)

Links

This is super useful if you have a website that contains maybe your restaurant's menu, or the daily country in your "guess a country game" or any information that you want to udpate easily from a website you own. (could even be a markdown file on github or a pastebin)

Domains

Useful if you want to still use a search engine but restrict it to maybe your company website, or your docs, or a specific subreddit for example. It just adds site:[domainhere] to the query before asking the search engine.

[nsarrazin]

cc @mishig25 maybe have a look at 59066f2 for websearch related changes if you want! (I added plain text support to the parser)

[mishig25]

@nsarrazin I didn't follow the reason why the websearch was turned off for the assistants from the beginning ?

[nsarrazin]

Don't remember exactly 😅 , cc @gary149

[mishig25]

If assistant has rag

rag?: {
	allowAll: boolean;
	allowList: string[];
	links: string[];
};

I think it would be useful to show the rag on the assitant page as well

[mishig25]

we can also add max n here

[nsarrazin]

Started a conversation internally about the security aspects of letting the user fetch an arbitrary URL server-side. Maybe this should be feature flagged?

[gururise]

Started a conversation internally about the security aspects of letting the user fetch an arbitrary URL server-side. Maybe this should be feature flagged?

Any resolution? This feature would be useful for agents.

[mishig25]

lgtm ! made some nits (please accept/disregard any)