This repository demonstrates an ultra-low latency architecture that orchestrates Rasa Pro (Agent), Rime (TTS), and Deepgram (ASR). It features a custom Python orchestrator, a real-time rich CLI dashboard, and CALM-based banking logic to achieve sub-second, human-like conversational turns.
We achieve fluid, human-like voice interaction by orchestrating best-in-class specialized services rather than using a monolithic black box.
- Input: User Audio (Simulated via Rime "Abbie" voice).
- ASR (Ears): Deepgram Nova-2 for lightning-fast speech-to-text.
- Brain: Rasa Pro executes deterministic business logic (CALM) to ensure secure money transfers.
- TTS (Voice): Rime Mist v2 generates high-fidelity agent audio with <300ms latency.
- Orchestrator: A custom Python client manages the real-time traffic to ensure immediate playback.
- Python 3.10+
uv(for fast dependency management)
Install all dependencies into a strictly versioned virtual environment:
make install
Create a .env file with your credentials:
# Required for Rasa Pro
RASA_LICENSE=your-rasa-pro-license-key-here
# Required for NLU/Reasoning
OPENAI_API_KEY=your-openai-api-key-here
# Required for the Voice Demo
DEEPGRAM_API_KEY=your-deepgram-api-key-here
RIME_API_KEY=your-rime-api-key-here
Instead of using a live microphone (which is risky for demos), we pre-generate high-quality user prompts using Rime:
make generate-audio
Train the Rasa CALM model to handle the banking logic:
make train
To run the full orchestration, open 3 separate terminal tabs:
Tab 1: The Action Server (Handles custom business logic)
make run-actions
Tab 2: The Agent (Rasa Pro Core)
make run-rasa
Tab 3: The Live Client (The Orchestrator & Visual Dashboard)
make demo
You will see a rich CLI dashboard and hear the conversation flow between the User and the Agent in real-time.
| 🤖 Start Building | 🧠 Join the Conversation |
|---|---|
| Try Hello Rasa | Agent Engineering Community |
| The fastest way to prototype. An interactive playground to build CALM (Conversational AI with Language Models) agents in your browser. ✨ No setup required ✨ No NLU training needed ✨ Built-in Copilot |
The home for people building real-world AI agents. A vendor-neutral space to discuss architectures, memory, and orchestration. 🤝 Meet other builders 🛠️ Share agent patterns 🎓 Learn form the best |
