CSM-1B TTS API

An OpenAI-compatible Text-to-Speech API that harnesses the power of Sesame's Conversational Speech Model (CSM-1B). This API allows you to generate high-quality speech from text using a variety of consistent voices, compatible with systems like OpenWebUI, ChatBot UI, and any platform that supports the OpenAI TTS API format.

Features

• OpenAI API Compatibility: Drop-in replacement for OpenAI's TTS API
• Multiple Voices: Six distinct voices (alloy, echo, fable, onyx, nova, shimmer)
• Voice Consistency: Maintains consistent voice characteristics across multiple requests
• Voice Cloning: Clone your own voice from audio samples
• Conversational Context: Supports conversational context for improved naturalness
• Multiple Audio Formats: Supports MP3, OPUS, AAC, FLAC, and WAV
• Speed Control: Adjustable speech speed
• CUDA Acceleration: GPU support for faster generation
• Web UI: Simple interface for voice cloning and speech generation

Getting Started

Prerequisites

• Docker and Docker Compose
• NVIDIA GPU with CUDA support (recommended)
• Hugging Face account with access to sesame/csm-1b model

Installation

1. Clone this repository:

git clone https://github.com/phildougherty/sesame_csm_openai
cd sesame_csm_openai

2. Create a .env file in the /app folder with your Hugging Face token:

HF_TOKEN=your_hugging_face_token_here

3. Build and start the container:

docker compose up -d --build

The server will start on port 8000. First startup may take some time as it downloads the model files.

Hugging Face Configuration

This API requires access to the sesame/csm-1b model on Hugging Face:

1. Create a Hugging Face account if you don't have one: https://huggingface.co/join
2. Accept the model license at https://huggingface.co/sesame/csm-1b
3. Generate an access token at https://huggingface.co/settings/tokens
4. Use this token in your .env file or pass it directly when building the container:

HF_TOKEN=your_token docker compose up -d --build

Required Models

The API uses the following models which are downloaded automatically:

• CSM-1B: The main speech generation model from Sesame
• Mimi: Audio codec for high-quality audio generation
• Llama Tokenizer: Uses the unsloth/Llama-3.2-1B tokenizer for text processing

Voice Cloning Guide

The CSM-1B TTS API comes with powerful voice cloning capabilities that allow you to create custom voices from audio samples. Here's how to use this feature:

Method 1: Using the Web Interface

1. Access the voice cloning UI by navigating to http://your-server-ip:8000/voice-cloning in your browser.

2. Clone a Voice:

   • Go to the "Clone Voice" tab
   • Enter a name for your voice
   • Upload an audio sample (2-3 minutes of clear speech works best)
   • Optionally provide a transcript of the audio for better results
   • Click "Clone Voice"

3. View Your Voices:

   • Navigate to the "My Voices" tab to see all your cloned voices
   • You can preview or delete voices from this tab

4. Generate Speech:

   • Go to the "Generate Speech" tab
   • Select one of your cloned voices
   • Enter the text you want to synthesize
   • Adjust the temperature slider if needed (lower for more consistent results)
   • Click "Generate Speech" and listen to the result

Method 2: Using the API

1. Clone a Voice:

curl -X POST http://localhost:8000/v1/voice-cloning/clone \
  -F "name=My Voice" \
  -F "audio_file=@path/to/your/voice_sample.mp3" \
  -F "transcript=Optional transcript of the audio sample" \
  -F "description=A description of this voice"

2. List Available Cloned Voices:

curl -X GET http://localhost:8000/v1/voice-cloning/voices

3. Generate Speech with a Cloned Voice:

curl -X POST http://localhost:8000/v1/voice-cloning/generate \
  -H "Content-Type: application/json" \
  -d '{
    "voice_id": "1234567890_my_voice",
    "text": "This is my cloned voice speaking.",
    "temperature": 0.7
  }' \
  --output cloned_speech.mp3

4. Generate a Voice Preview:

curl -X POST http://localhost:8000/v1/voice-cloning/voices/1234567890_my_voice/preview \
  --output voice_preview.mp3

5. Delete a Cloned Voice:

curl -X DELETE http://localhost:8000/v1/voice-cloning/voices/1234567890_my_voice

Voice Cloning Best Practices

For the best voice cloning results:

1. Use High-Quality Audio: Record in a quiet environment with minimal background noise and echo.

2. Provide Sufficient Length: 2-3 minutes of speech provides better results than shorter samples.

3. Clear, Natural Speech: Speak naturally at a moderate pace with clear pronunciation.

4. Include Various Intonations: Sample should contain different sentence types (statements, questions) for better expressiveness.

5. Add a Transcript: While optional, providing an accurate transcript of your recording helps the model better capture your voice characteristics.

6. Adjust Temperature: For more consistent results, use lower temperature values (0.6-0.7). For more expressiveness, use higher values (0.7-0.9).

7. Try Multiple Samples: If you're not satisfied with the results, try recording a different sample or adjusting the speaking style.

Using Cloned Voices with the Standard TTS Endpoint

Cloned voices are automatically available through the standard OpenAI-compatible endpoint. Simply use the voice ID or name as the voice parameter:

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "csm-1b",
    "input": "This is my cloned voice speaking through the standard endpoint.",
    "voice": "1234567890_my_voice",
    "response_format": "mp3"
  }' \
  --output cloned_speech.mp3

How the Voices Work

Unlike traditional TTS systems with pre-trained voice models, CSM-1B works differently:

• The base CSM-1B model is capable of producing a wide variety of voices but doesn't have fixed voice identities
• This API creates consistent voices by using acoustic "seed" samples for each named voice
• When you specify a voice (e.g., "alloy"), the API uses a consistent acoustic seed and speaker ID
• The most recent generated audio becomes the new reference for that voice, maintaining voice consistency
• Each voice has unique tonal qualities:
  • alloy: Balanced mid-tones with natural inflection
  • echo: Resonant with slight reverberance
  • fable: Brighter with higher pitch
  • onyx: Deep and resonant
  • nova: Warm and smooth
  • shimmer: Light and airy with higher frequencies

The voice system can be extended with your own voice samples by using the voice cloning feature.

API Usage

Basic Usage

Generate speech with a POST request to /v1/audio/speech:

curl -X POST http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "csm-1b",
    "input": "Hello, this is a test of the CSM text to speech system.",
    "voice": "alloy",
    "response_format": "mp3"
  }' \
  --output speech.mp3

Available Endpoints

Standard TTS Endpoints

• GET /v1/audio/models - List available models
• GET /v1/audio/voices - List available voices (including cloned voices)
• GET /v1/audio/speech/response-formats - List available response formats
• POST /v1/audio/speech - Generate speech from text
• POST /api/v1/audio/conversation - Advanced endpoint for conversational speech

Voice Cloning Endpoints

• POST /v1/voice-cloning/clone - Clone a new voice from an audio sample
• GET /v1/voice-cloning/voices - List all cloned voices
• POST /v1/voice-cloning/generate - Generate speech with a cloned voice
• POST /v1/voice-cloning/voices/{voice_id}/preview - Generate a preview of a cloned voice
• DELETE /v1/voice-cloning/voices/{voice_id} - Delete a cloned voice

Request Parameters

Standard TTS

Parameter	Description	Type	Default
model	Model ID to use	string	"csm-1b"
input	The text to convert to speech	string	Required
voice	The voice to use (standard or cloned voice ID)	string	"alloy"
response_format	Audio format	string	"mp3"
speed	Speech speed multiplier	float	1.0
temperature	Sampling temperature	float	0.8
max_audio_length_ms	Maximum audio length in ms	integer	90000

Voice Cloning

Parameter	Description	Type	Default
name	Name for the cloned voice	string	Required
audio_file	Audio sample file	file	Required
transcript	Transcript of the audio	string	Optional
description	Description of the voice	string	Optional

Available Voices

• alloy - Balanced and natural
• echo - Resonant
• fable - Bright and higher-pitched
• onyx - Deep and resonant
• nova - Warm and smooth
• shimmer - Light and airy
• [cloned voice ID] - Any voice you've cloned using the voice cloning feature

Response Formats

• mp3 - MP3 audio format
• opus - Opus audio format
• aac - AAC audio format
• flac - FLAC audio format
• wav - WAV audio format

Integration with OpenWebUI

OpenWebUI is a popular open-source UI for AI models that supports custom TTS endpoints. Here's how to integrate the CSM-1B TTS API:

1. Access your OpenWebUI settings
2. Navigate to the TTS settings section
3. Select "Custom TTS Endpoint"
4. Enter your CSM-1B TTS API URL: http://your-server-ip:8000/v1/audio/speech
5. Use the API Key field to add any authentication if you've configured it (not required by default)
6. Test the connection
7. Save your settings

Once configured, OpenWebUI will use your CSM-1B TTS API for all text-to-speech conversion, producing high-quality speech with the selected voice.

Using Cloned Voices with OpenWebUI

Your cloned voices will automatically appear in OpenWebUI's voice selector. Simply choose your cloned voice from the dropdown menu in the TTS settings or chat interface.

Advanced Usage

Conversational Context

For more natural-sounding speech in a conversation, you can use the conversation endpoint:

curl -X POST http://localhost:8000/api/v1/audio/conversation \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Nice to meet you too!",
    "speaker_id": 0,
    "context": [
      {
        "speaker": 1,
        "text": "Hello, nice to meet you.",
        "audio": "BASE64_ENCODED_AUDIO"
      }
    ]
  }' \
  --output response.wav

This allows the model to take into account the previous utterances for more contextually appropriate speech.

Model Parameters

For fine-grained control, you can adjust:

• temperature (0.0-1.0): Higher values produce more variation but may be less stable
• topk (1-100): Controls diversity of generated speech
• max_audio_length_ms: Maximum length of generated audio in milliseconds
• voice_consistency (0.0-1.0): How strongly to maintain voice characteristics across segments

Troubleshooting

API Returns 503 Service Unavailable

• Verify your Hugging Face token has access to sesame/csm-1b
• Check if the model downloaded successfully in the logs
• Ensure you have enough GPU memory (at least 8GB recommended)

Audio Quality Issues

• Try different voices - some may work better for your specific text
• Adjust temperature (lower for more stable output)
• For longer texts, the API automatically splits into smaller chunks for better quality
• For cloned voices, try recording a cleaner audio sample

Voice Cloning Issues

• Poor Voice Quality: Try recording in a quieter environment with less background noise
• Inconsistent Voice: Provide a longer and more varied audio sample (2-3 minutes)
• Accent Issues: Make sure your sample contains similar words/sounds to what you'll be generating
• Low Volume: The sample is normalized automatically, but ensure it's not too quiet or distorted

Voice Inconsistency

• The API maintains voice consistency across separate requests
• However, very long pauses between requests may result in voice drift
• For critical applications, consider using the same seed audio

License

This project is released under the MIT License. The CSM-1B model is subject to its own license terms defined by Sesame.

Acknowledgments

• Sesame for releasing the CSM-1B model
• This project is not affiliated with or endorsed by Sesame or OpenAI

---

Happy speech generating!