Skip to content

FastAPIChat is a powerful application for evaluating large language model based chatbots.

License

Notifications You must be signed in to change notification settings

yuting1214/FastAPIChat

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

10 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

Deploy on Railway

FastAPIChat is a powerful application for testing and evaluating large language model based chatbots.

  • FastAPIChat powers LLM Arena, serving over 10+ LLMs, for diverse and dynamic chatbot interactions
  • FastAPIChat enables seamlessly rate LLM generated responses, ensuring continuous improvement based on user feedback.

Demo of FastAPIChat

News

  • [2024/06] πŸ”₯ We released FastAPIChat v1.0.0
More
  • [2024/06] We released FastAPIChat v1.0.0 with ChatBot and ChatBot Arena

πŸŽ‰ Key Features and Integrations

Key features:

  • πŸ’¬ LLM Chats(10+ LLMs)
  • βš”οΈ LLM Arena
  • πŸ—³οΈ Chat Feedback collection
  • ⚑ FastAPI API Documentation and Authentication

FastAPIChat integrates seamlessly with Python programs and libraries and includes out-of-the-box integrations for:

Quick-Start πŸš€

Note: To begin, you'll need an API key from OpenRouter.

Development Mode

  • Note: In Dev mode, the default config with backend is SQLite.
  1. Clone Repo:
git clone https://github.com/yuting1214/FastAPIChat.git
  1. Configure Virtual Environment (Recommended):
  • Create Virtual Environment

    # macOS/Linux
    # You may need to run `sudo apt-get install python3-venv` first on Debian-based OSs
    python3 -m venv .venv
    
    # Windows
    # You can also use `py -3 -m venv .venv`
    python -m venv .venv
    
  • Activate the virtual environment:

    # macOS/Linux
    source .venv/bin/activate
    
    # Windows
    .venv\Scripts\activate
    
  1. Install Dependencies:
pip install -r requirements.txt
  1. Update the .env file with
USER_NAME=<API_Doc_Username>
PASSWORD=min=<API_Doc_PASSWORD>
OPENROUTER_API_KEY=<Your_API_KEY>
  1. Run the application:
python -m backend.app.main --mode dev 

Production Mode

  1. Add extra envs in with your environment configurations:
DB_ENGINE=<Your_DB_config>
DB_USERNAME=<Your_DB_config>
DB_PASS=<Your_DB_config>
DB_HOST=<Your_DB_config>
DB_PORT=<Your_DB_config>
DB_NAME=<Your_DB_config>
API_BASE_URL=<Your_Host_URL>
  1. Run the application:
python -m backend.app.main --mode prod
  • Alternatively, use Dockerfile for deployment.

App Settings πŸ“‹

Managing Quota

  • Control quotas by adjusting limits in backend/app/core/constants.py:
TEXT_API_QUOTA_LIMIT = <int>

Adding New LLM Models (OpenRouter)

  • To add a new LLM model, specify its details in backend/data/llm_models:
    {
        'llm_model_name': 'Mixtral 8x22B',
        'llm_vendor': 'Mistral AI',
        'llm_type': 'text',
        'api_provider': 'OpenRouter',
        'api_endpoint': 'mistralai/mixtral-8x22b-instruct',
    }

Features 🌟

  1. Utilize Gradio and FastAPI Integration: Seamlessly combine Gradio for the frontend display and FastAPI for robust backend functionality, offering a smooth user experience.

  2. Well-Crafted API Management: Our meticulously designed API ensures efficient data transportation and effective management of API quota usage, providing users with a reliable and hassle-free experience.

  3. Integration of Leading Language Models: Integrate more than 10+ popular language models from various vendors, including OpenAI, Mistral, Meta and Anthropic, enabling users to leverage cutting-edge AI capabilities for text generation.

  4. Comprehensive Testing and Documentation: Benefit from comprehensive testing suites and detailed documentation, empowering developers to understand and contribute to the project with ease, ensuring robustness and maintainability. Third-Party Services: Integrate with third-party APIs for enhanced functionality.

  5. Third-Party Services Integration: Seamlessly integrates with third-party APIs for enhanced functionality.

Project Structure πŸ“

More Project Details
FastAPIChat/
β”œβ”€β”€ backend/                      # Backend directory for the FastAPI application
β”‚   β”œβ”€β”€ app/                      # Main application directory
β”‚   β”‚   β”œβ”€β”€ __init__.py           # Initialization file for the app package
β”‚   β”‚   β”œβ”€β”€ api/                  # Directory for API related code
β”‚   β”‚   β”‚   β”œβ”€β”€ __init__.py       # Initialization file for the API package
β”‚   β”‚   β”‚   β”œβ”€β”€ v1/               # Version 1 of the API
β”‚   β”‚   β”‚   β”‚   β”œβ”€β”€ __init__.py   # Initialization file for the v1 API package
β”‚   β”‚   β”‚   β”‚   β”œβ”€β”€ endpoints/    # Directory for API endpoint definitions
β”‚   β”‚   β”‚   β”‚   β”‚   β”œβ”€β”€ __init__.py          # Initialization file for endpoints package
β”‚   β”‚   β”‚   β”‚   β”‚   β”œβ”€β”€ text_generation.py   # Endpoints for text generation
β”‚   β”‚   β”‚   β”‚   β”‚   β”œβ”€β”€ llm_management.py    # Endpoints for LLM model management
β”‚   β”‚   β”‚   β”‚   β”‚   β”œβ”€β”€ api_usage.py         # Endpoints for API usage management
β”‚   β”‚   β”‚   β”‚   β”‚   β”œβ”€β”€ api_calldetail.py    # Endpoints for API Detail management
β”‚   β”‚   β”‚   β”‚   β”‚   β”œβ”€β”€ message.py           # Endpoints for messasge management
β”‚   β”‚   β”‚   β”‚   β”‚   β”œβ”€β”€ chat.py              # Endpoints for chat management
β”‚   β”‚   β”‚   β”‚   β”‚   β”œβ”€β”€ rating.py            # Endpoints for rating management
β”‚   β”‚   β”‚   β”‚   β”‚   β”œβ”€β”€ user.py              # Endpoints for user management
β”‚   β”‚   β”‚   β”‚   β”‚   β”œβ”€β”€ quota.py             # Endpoints for quota management
β”‚   β”‚   β”‚   β”‚   β”‚   β”œβ”€β”€ doc.py               # Endpoints for OpenAPI doc management
β”‚   β”‚   β”œβ”€β”€ dependencies/         # Directory for dependency management
β”‚   β”‚   β”‚   β”œβ”€β”€ __init__.py       # Initialization file for dependencies package
β”‚   β”‚   β”‚   β”œβ”€β”€ database.py       # Database connection and session management
β”‚   β”‚   β”‚   β”œβ”€β”€ rate_limiter.py   # Rate limiting logic
β”‚   β”‚   β”œβ”€β”€ core/                 # Core application logic
β”‚   β”‚   β”‚   β”œβ”€β”€ __init__.py       # Initialization file for core package
β”‚   β”‚   β”‚   β”œβ”€β”€ constant.py       # Constant settings
β”‚   β”‚   β”‚   β”œβ”€β”€ config.py         # Configuration settings
β”‚   β”‚   β”‚   β”œβ”€β”€ init_setting.py   # Init settings with user's input
β”‚   β”‚   β”œβ”€β”€ models/               # Directory for SQLAlchemy models
β”‚   β”‚   β”‚   β”œβ”€β”€ __init__.py       # Initialization file for models package
β”‚   β”‚   β”‚   β”œβ”€β”€ user.py           # User model
β”‚   β”‚   β”‚   β”œβ”€β”€ quota.py          # Quota model
β”‚   β”‚   β”‚   β”œβ”€β”€ message.py        # Message model
β”‚   β”‚   β”‚   β”œβ”€β”€ api_usage.py      # API usage model
β”‚   β”‚   β”‚   β”œβ”€β”€ llm.py            # LLM model
β”‚   β”‚   β”‚   β”œβ”€β”€ chat.py           # Chat model
β”‚   β”‚   β”‚   β”œβ”€β”€ rating.py         # rating model
β”‚   β”‚   β”œβ”€β”€ schemas/              # Directory for Pydantic schemas
β”‚   β”‚   β”‚   β”œβ”€β”€ __init__.py       # Initialization file for schemas package
β”‚   β”‚   β”‚   β”œβ”€β”€ user.py           # Schemas for user data
β”‚   β”‚   β”‚   β”œβ”€β”€ quota.py          # Schemas for quota data
β”‚   β”‚   β”‚   β”œβ”€β”€ message.py        # Schemas for message data
β”‚   β”‚   β”‚   β”œβ”€β”€ api_usage.py      # Schemas for API usage data
β”‚   β”‚   β”‚   β”œβ”€β”€ llm.py            # Schemas for LLM model data
β”‚   β”‚   β”‚   β”œβ”€β”€ llm_message.py    # Schemas for LLM generated output data
β”‚   β”‚   β”‚   β”œβ”€β”€ chat.py           # Schemas for chat data
β”‚   β”‚   β”‚   β”œβ”€β”€ rating.py         # Schemas for rating data
β”‚   β”‚   β”œβ”€β”€ crud/                 # Directory for CRUD operations
β”‚   β”‚   β”‚   β”œβ”€β”€ __init__.py       # Initialization file for crud package
β”‚   β”‚   β”‚   β”œβ”€β”€ user.py           # CRUD for user management
β”‚   β”‚   β”‚   β”œβ”€β”€ quota.py          # CRUD for quota management
β”‚   β”‚   β”‚   β”œβ”€β”€ message.py        # CRUD for message management
β”‚   β”‚   β”‚   β”œβ”€β”€ api_usage.py      # CRUD for API usage management
β”‚   β”‚   β”‚   β”œβ”€β”€ llm.py            # CRUD for LLM model management
β”‚   β”‚   β”‚   β”œβ”€β”€ llm_message.py    # CRUD for LLM generated output management
β”‚   β”‚   β”‚   β”œβ”€β”€ chat.py           # CRUD for chat management
β”‚   β”‚   β”‚   β”œβ”€β”€ rating.py         # CRUD for rating management
β”‚   β”‚   β”œβ”€β”€ main.py               # Main FastAPI application file
β”‚   β”œβ”€β”€ data/                     # Directory for data when initiating db
β”‚   β”‚   β”œβ”€β”€ __init__.py           # Initialization file for data package
β”‚   β”‚   β”œβ”€β”€ llm_models.py         # LLM models infomation
β”‚   β”œβ”€β”€ security/                  # New directory for authentication and authorization
β”‚   β”‚   β”œβ”€β”€ __init__.py            # Initialization file for security package
β”‚   β”‚   β”œβ”€β”€ authentication.py      # Authentication logic
β”‚   β”‚   β”œβ”€β”€ authorization.py       # Authorization logic
β”‚   β”œβ”€β”€ tests/                    # Directory for test files
β”‚   β”‚   β”œβ”€β”€ __init__.py           # Initialization file for tests package
β”‚   β”‚   β”œβ”€β”€ test_text_generation.py  # Tests for text generation endpoints
β”œβ”€β”€ frontend/                     # Frontend directory for the Gradio app
β”‚   β”œβ”€β”€ __init__.py               # Initialization file for frontend package
β”‚   β”œβ”€β”€ gradio/                   # Main gradio UI directory
β”‚   β”‚   β”œβ”€β”€ __init__.py           # Initialization file for the gradio package
β”‚   β”‚   β”œβ”€β”€ text/                 # Directory for ChatBot UI related code
β”‚   β”‚   β”‚   β”œβ”€β”€ __init__.py       # Initialization file for the text package
β”‚   β”‚   β”‚   β”œβ”€β”€ event_listeners.py #      
β”‚   β”‚   β”‚   β”œβ”€β”€ text_generation.py # 
β”‚   β”‚   β”‚   β”œβ”€β”€ text_generation_arena.py               
β”‚   β”œβ”€β”€ login/                     # Main login UI directory
β”‚   β”‚   β”œβ”€β”€ __init__.py           # Initialization file for the gradio package
β”‚   β”‚   β”œβ”€β”€ static/               # Directory for static files
β”‚   β”‚   β”‚   β”œβ”€β”€ style.css         # CSS for login UI               
β”‚   β”‚   β”œβ”€β”€ templates/            # Directory for HTML templates
β”‚   β”‚   β”‚   β”œβ”€β”€ base.html         # HTML base template
β”‚   β”‚   β”‚   β”œβ”€β”€ login.html        # HTML login template   
β”œβ”€β”€ llm/
β”‚   β”œβ”€β”€ __init__.py               # Makes llm a Python package
β”‚   β”œβ”€β”€ llm_text_chain.py         # Module for LLM text generation integration
β”‚   └── prompt/                   # Folder for prompt handling
β”‚       β”œβ”€β”€ __init__.py           # Initializes the prompt package
β”‚       β”œβ”€β”€ base_text_templates.py# Stores base prompt templates for text generation
β”‚       └── examples/             # Directory for few-shot examples used by the chain
β”‚       └── deprecated/           # Directory for deprecated prompts
β”‚   └── vendors/                  # Directory for vendor-specific LLM configurations
β”‚       β”œβ”€β”€ __init__.py           # Makes vendors a Python package
β”‚       └── openrouter.py         # Configurations and usage for OpenRouter as LLM provider
β”œβ”€β”€ .env                          # Environment variables file
β”œβ”€β”€ .gitignore                    # Git ignore file
β”œβ”€β”€ requirements.txt              # Python dependencies file
β”œβ”€β”€ Dockerfile                    # Docker configuration
β”œβ”€β”€ README.md                     # Project README file

Future Direction πŸ“

  1. Integraion of local LLM: Integrate with Ollama to enable users to run LLMs locally and battle with LLMs from API.

  2. Integraion of RAG LLM: Enable users to compare different configuration of RAG LLM in LLM Arena.

  3. Integration of Redis or other NoSQL Databases: Incorporate Redis or other NoSQL databases to efficiently track and store user history data, enabling personalized experiences and insights for users.

  4. History Message Management: Allow user to fetch back the previous messages from database and continue the chat.

  5. Rate Limiter: Integrate rate limiter for API protection.

Contributing 🀝

We welcome contributions from the community! Whether it's bug fixes, feature enhancements, or documentation improvements, feel free to open a pull request.


Designed with ❀️ by Mark Chen