L2D Assistant

Disclaimer: The TTS voice produced by this application is AI-generated and not a human voice.

L2D Assistant is a Live2D virtual assistant that lives on your desktop. Powered by OpenAI's GPT, STT (Speech-to-Text), and TTS (Text-to-Speech) models, your assistant will respond to your spoken commands in near-realtime.

Note: This application has only been tested on Windows. It may not work on Linux or macOS.

Window fades-in on mouse hover:

Settings menu:

Features

• Animated Live2D character that responds to your voice
• Voice commands (STT), AI-generated responses (GPT), and AI-generated voice replies (TTS)
• Function calling system allows L2D Assistant to:
  • Get live weather information
  • Search the internet
  • More coming in the future!
• Settings menu to customize you model, activation hotword, AI instructions and manage your OpenAI API key

Requirements

• Node.js (version 14 or higher)
• npm (usually included with Node.js)
• A Windows operating system (optimal experience; not tested on macOS or Linux)
• A free OpenAI developer account (get an API key at OpenAI Platform)
• (Optional) It is recomended to use Nvidia Broadcast or a similar software for background noise removal.

Installation

1. Clone or download this repository to your local machine.

2. Navigate to /src/ in the project folder using your terminal or type cmd in the path bar in File Explorer while in /src/.

3. Install dependencies:

   npm install
   npm update
   

4. Start the application:

   npm start
   

5. Upon first launch, enter your OpenAI API key into the settings menu. You can obtain a key by signing up for a free developer account at https://platform.openai.com/.

   Please note that using your API key costs money. Using default settings my bill over the course of a week was only $0.12. It is recomended to use Nvidia Broadcast or a similer noise canceling software to prevent sending empty background noise to the API, as this will not only increase your bill but also make STT more unrelyable. You can also play with the VOLUME_THRESHOLD in preload.js until I add a slider to the settings menu.

Usage

1. Ensure your microphone is properly set up and recognized by your computer. This app uses your default system microphone: How To Set Default Microphone in Windows
2. Launch the application "npm start". Eventually I'll upload some builds here.
3. Configure your OpenAI settings (model, TTS voice, etc.) from the app’s settings menu. MAKE SURE TO SET YOUR API KEY!
4. You can find some pre-configured Live2D models here: https://github.com/evrstr/live2d-widget-models?tab=readme-ov-file Just drop the link to the model.json into the settings menu and hit apply.
5. Interact with the desktop assistant by speaking commands such as "Hey Assistant, what is the weather like today?" or "Assistant, what are the best JavaScript for dummies courses online?". You can also change your sleeper agent activation word into any word or phrase you like. Just be mindful of the limitations of STT.
6. Enjoy your personalized Live2D AI assistant!

Roadmap

• Additional AI function integrations (e.g., media constrols, reminders/alarms, and community requested features).
• Support for macOS and Linux (feedback needed).

---

Disclaimer: This README was partially(mostly) generated by AI.