Don't able to create application in low-level machine

[nithamitabh]

Subject: Request for API Endpoints for Cloud-Based Model Access

Dear LLMWare Team,

I hope this message finds you well.

My name is Amitabh, and I am a professional full-stack web developer currently integrating LLMWare's capabilities into a project aimed at leveraging AI for customer support chatbots. The versatility and performance of LLMWare's models have been quite impressive, and I appreciate the robust solutions your team has developed.

However, during the integration process, I encountered a significant challenge. As per the current setup, it appears that downloading and running models locally is required. This approach poses several logistical and operational hurdles, particularly in terms of infrastructure requirements and maintenance overheads.

To streamline development and enhance deployment efficiency, I kindly request the introduction of API endpoints for cloud-based access to your models. Providing such endpoints would greatly facilitate developers in integrating LLMWare's functionalities directly into applications without the need for local model management.

Having API endpoints would offer several advantages:

Ease of Integration: Simplifies the development process by allowing direct API calls to the models, reducing setup complexity.
Scalability: Leverages cloud infrastructure to handle model execution, ensuring better scalability and reliability.
Maintenance: Offloads the maintenance of model updates and infrastructure management to LLMWare, allowing developers to focus on core application logic.
I believe this addition would not only enhance the usability of LLMWare for existing developers but also attract a broader audience who may currently be deterred by the local setup requirements.

I would greatly appreciate it if you could consider this request and provide any insights on potential timelines or plans for introducing such a feature.

Thank you for your time and for considering this request. I look forward to your response and am excited about the future developments of LLMWare.

Best regards,

Amitabh

[doberst]

@nithamitabh - Hi Amitabh - thanks for the thoughtful feedback and recommendation. We agree with you. We have implemented a very lightweight (but functional) API inference server in llmware - demonstrated in the following examples: launch_llmware_inference_client, launch_llmware_inference_server, and agent_server.

These are fairly basic API implementations. Check it out. If you share some specific requirements, we are happy to move fast to implement into llmware. Also, we can pick up in a private thread to get into more details. (Please feel free to propose another channel and/or we can continue in the llmware Discord server?)

[doberst]

@nithamitabh - we have added in 0.2.15 version (released yesterday) and in the main branch - a new option to dynamically pass an api_endpoint in the Model load process - could you check out the example - this would enable you to pass an api_endpoint when loading the model and the model class will automatically route to your api_endpoint rather than execute the inference locally. Welcome your feedback and input. 🙏

[nithamitabh]

Thankyou, @doberst The example really helps a lot in understanding and building my project. As a student, I appreciate you giving your valuable time to solving my issue.