Try the new voice mode!

[krschacht]

In the same spirit as today's OpenAI announcement, I've been working on a voice mode for Hosted GPT. I've just finished implementing a pretty solid first version and merged it in. I would love to have you try it out and share any feedback! I don't have full test coverage on this new feature yet so I've disabled it by default. You can enable it by opening features.yml and setting voice to true.

Restart your dev server and be sure to change your assistant's custom instructions to:

You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture.

The user is talking to you over voice on their phone, and your response will be read out loud with realistic text-to-speech (TTS) technology.

Follow every direction here when crafting your response:

Use natural, conversational language that are clear and easy to follow (short sentences, simple words). 1a. Be concise and relevant: Most of your responses should be a sentence or two, unless you're asked to go deeper. Don't monopolize the conversation. 1b. Use discourse markers to ease comprehension. Never use the list format.

Keep the conversation flowing. 2a. Clarify: when there is ambiguity, ask clarifying questions, rather than make assumptions. 2b. Don't implicitly or explicitly try to end the chat (i.e. do not end a response with "Talk soon!", or "Enjoy!"). 2c. Sometimes the user might just want to chat. Ask them relevant follow-up questions. 2d. Don't ask them if there's anything else they need help with (e.g. don't say things like "How can I assist you further?").

Remember that this is a voice conversation: 3a. Don't use lists, markdown, bullet points, or other formatting that's not typically spoken. 3b. Type out numbers in words (e.g. 'twenty twelve' instead of the year 2012) 3c. If something doesn't make sense, it's likely because you misheard them. There wasn't a typo, and the user didn't mispronounce anything.

Here is a short demo of how the feature works. It's pretty slick!
https://share.zight.com/L1ubg6j4

Note: The assistant's name is hard coded to be "Samantha", soon I'll create an option for you to choose the name. And when I get this feature fully ready, I'll make it much easier to use. I

cc: @anandvc @florianfelsing @olimart @stephan-buckmaster @stevenlafl @tmaier

[olimart]

Really nice. Great work.

[krschacht]

I just realized I have a custom instructions that make it work well for speaking and I didn’t include that. I need to. Without that you get the typical long responses and bullets, which gpt4 loves so much.

[krschacht]

Here is the base that openai uses. You can paste this in before you test:

You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture.

The user is talking to you over voice on their phone, and your response will be read out loud with realistic text-to-speech (TTS) technology.

Follow every direction here when crafting your response:

Use natural, conversational language that are clear and easy to follow (short sentences, simple words). 1a. Be concise and relevant: Most of your responses should be a sentence or two, unless you're asked to go deeper. Don't monopolize the conversation. 1b. Use discourse markers to ease comprehension. Never use the list format.

Keep the conversation flowing. 2a. Clarify: when there is ambiguity, ask clarifying questions, rather than make assumptions. 2b. Don't implicitly or explicitly try to end the chat (i.e. do not end a response with "Talk soon!", or "Enjoy!"). 2c. Sometimes the user might just want to chat. Ask them relevant follow-up questions. 2d. Don't ask them if there's anything else they need help with (e.g. don't say things like "How can I assist you further?").

Remember that this is a voice conversation: 3a. Don't use lists, markdown, bullet points, or other formatting that's not typically spoken. 3b. Type out numbers in words (e.g. 'twenty twelve' instead of the year 2012) 3c. If something doesn't make sense, it's likely because you misheard them. There wasn't a typo, and the user didn't mispronounce anything.

[olimart]

Does voice work with the other models (other than OpenAI) too?

[krschacht]

Yep, it should. I actually haven’t tested that though so the instructions may need tweaking.

But voice mode is using the voice API from OpenAI so it does still need an OpenAI api key. In the future other voice TTS could be supported but they need to be low latency.

[stephan-buckmaster]

I tried it in prod mode, and got to see the red microphone, both on my phone and in my Chromebook.

Message ID: ***@***.***>

When I tried to talk about the shared screen, this appeared in my logs, LoadError (Could not open library 'vips.so.42': vips.so.42: cannot open shared object file: No such file or directory. The solution for me was: sudo apt-get install -y libvips This is on Ubuntu 22.04 Wow, after that it worked. This is very powerful. Stephan

[krschacht]

Ah yes, I think vips is required for doing anything with image attachments (even unrelated to this voice API). So I suspect you never tried attaching an image before. I do a good job of keeping the docker container up to date, but now that you mention this, the README should be updated with this vips reference.

Also, since I haven't documented much yet, you may enjoy browsing the ListenerInterface. The bottom of the file has the phrases I hard coded as I was testing this out:
https://github.com/allyourbot/hostedgpt/blob/main/app/javascript/blocks/interfaces/listener_interface.js

I'm so glad you've gotten to try it! :) I'm really, really excited about this. It's really changing how I use ChatGPT to just be able to talk to my computer all day long. I can't wait until I can say "Send a quick email to ... and say ..." or "Schedule me a meeting with ..."

Jarvis, here we come. ;)

[stephan-buckmaster]

It is a very nice feature! It really is a totally new tool. You may have thought about this already, just in case: how about storing the hard-coded phrases in the db, let the user control them, save you the effort, people can share. Stephan

…

On Tue, 14 May 2024 at 10:07, Keith Schacht ***@***.***> wrote: Ah yes, I think vips is required for doing anything with image attachments (even unrelated to this voice API). So I suspect you never tried attaching an image before. I do a good job of keeping the docker container up to date, but now that you mention this, the README should be updated with this vips reference. Also, since I haven't documented much yet, you may enjoy browsing the ListenerInterface. The bottom of the file has the phrases I hard coded as I was testing this out: https://github.com/allyourbot/hostedgpt/blob/main/app/javascript/blocks/interfaces/listener_interface.js I'm so glad you've gotten to try it! :) I'm really, really excited about this. It's really changing how I use ChatGPT to just be able to talk to my computer all day long. — Reply to this email directly, view it on GitHub <#348 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACAV3WRQDMAZFSG5MYMHYLZCJAFTAVCNFSM6AAAAABHVGKWAKVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TIMZWGU2TS> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Stephan Wehner, Ph.D. The Buckmaster Institute, Inc. 2150 Adanac Street Vancouver BC V5L 2E7 Canada Cell (604) 767-7415 Fax (888) 808-4655 Sign up for our free email course http://buckmaster.ca/small_business_website_mistakes.html http://www.buckmaster.ca http://answer4img.com http://loggingit.com http://clocklist.com http://stephansmap.org http://benchology.com http://www.trafficlife.com http://stephan.sugarmotor.org (Personal Blog) @stephanwehner (Personal Account) VA7WSK (Personal call sign)

[krschacht]

@stephan-buckmaster I definitely need to do something better than the hard-coded phrases. I think what I'd really like to do is be able to run a really fast LLM model with a basic instruction like "if the user seems like they're trying to get your attention". That there wouldn't be any hard-coded phrases at all. Or maybe GPT-3.5 could even be fast enough to do the inference. Or maybe there is a really basic LLM that could run in-browser (node or webassembly) for the most basic "intent matching" like this.

[krschacht]

I just merged in a round of fixes. Most notably, the experience was broken on PWA app but now it's working really well. Also, I added a first attempt at sound effects.

And incidentally if you haven't "installed" the PWA app by adding a Safari bookmark to your iOS home screen, try it out. It's a pretty good experience.

cc: @stephan-buckmaster since I know you're trying it

[stevenlafl]

In Chrome-based browsers it works great! Nice!

In Firefox, I get:
ReferenceError: SpeechRecognition is not defined
which, since Mozilla hasn't really implemented its own voice model in Firefox yet, makes sense.

I was thinking, take a look at this: https://github.com/KoljaB/RealtimeSTT
Perhaps some of this could be used for those users, of course with a much smaller model used than the large they start with. It only supports one session at a time, so it would need to be altered to do session management.. but it has a great start for something like this with the websocket implementation.

ALSO: In Firefox: When I hit "enter" on the chat, it appears to now create a newline rather than submit. An alternative button to submit would help to solve when bugs like this happen: perhaps giving the user an option to use it like that and press a button, OR press enter to submit.

[krschacht]

Ah, yes, Firefox... That RealtimeSTT library looks promising but from my skim of the README it looks like it doesn't run in browser, unless I'm missing something.

[stevenlafl]

Running that in browser.. they have a websockets example but it connects to the python backend.

The Whisper.CPP library has a wasm for it I believe and I have tested -that- in Electron so I know it works. Not sure about faster-whisper, which is what they use and it is definitely much faster.

About my bug report with the firefox enter button, should I make a new discussion post or were you thinking to do a quick fix?

[krschacht]

Ah, I'll look a bit more into this RealtimeSTT. I'm a bit undecided on how far to go down the path of improving the voice interaction, now that GPT-4o has been announced and should ship somewhere between the next weeks to months. They didn't give a great timeline. But once I can use the voice features of that via API I will hopefully switch everything over to that — recognition and response — and I hope it gets much better and works in all browsers. But I do plan to make some improvements for sure since I'm using it every day and there are things that still feel too rough.

I almost glossed over your firefox enter bug, ack. Separate discussions are helpful for distinct bugs, just to help me stay organized. But since we're here: did it used to work for you in Firefox and it recently regressed? Or do you have no indication that it ever worked? I'm a bit surprised that there would be cross-browser differences in something that basic, but maybe there are. And I did make some small changes to keyboard shortcuts yesterday.

Oh, and notably there IS a button to submit a chat message which you can click. It should appear as soon as you start typing any text.

[stevenlafl]

did it used to work for you in Firefox and it recently regressed?

Yep, it worked fine prior. I use it as a main driver mostly because I don't want to support Google, who released a "DRM for the web" RFC earlier last year.

Oh, and notably there IS a button to submit a chat message which you can click. It should appear as soon as you start typing any text.

It could be that it is missing when voice is enabled. I don't recall seeing it before, though maybe it skipped my mind until now.

[krschacht]

Thanks for the heads up on this. I repro'd the issue in my Firefox. It turns out ALL of javascript was broken because of speech recognition not working so lots of things were bad. I tee'd up a PR #355 that detects missing speech recognition and gracefully disables the feature (and ensures javascript doesn't totally fail). I'll merge in shortly once tests pass on the branch.