Merge pull request #347 from LlmKira/dev

Fix Everything

.github/workflows/docker-ci-dev.yml

@@ -2,6 +2,11 @@ name: Docker Image CI (dev)
 
 on:
   workflow_dispatch:
+  push:
+    branches:
+      - docker
+      # - develop
+      # - '**-develop'
 
 jobs:
   build-and-push-image:
@@ -35,7 +40,6 @@ jobs:
             sudoskys/llmbot
           tags: |
             # set latest tag for default branch
-            type=raw,value=latest,enable={{is_default_branch}}
             type=ref,event=branch
 
           flavors: |

.github/workflows/docker-ci.yaml

@@ -2,9 +2,11 @@ name: Docker Image CI (stable)
 
 on:
   workflow_dispatch:
-  pull_request:
+  push:
     branches:
       - main
+    tags:
+      - '**'
   release:
     types: [ published ]
 
@@ -40,7 +42,7 @@ jobs:
             sudoskys/llmbot
           tags: |
             # set latest tag for default branch
-            type=raw,value=latest,enable={{is_default_branch}}
+            type=raw,value=latest
             type=ref,event=tag
             type=ref,event=branch
 

.github/workflows/python_test.yml

@@ -0,0 +1,37 @@
+name: Run tests
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - main
+      - dev
+      - develop
+      - '**-develop'
+
+jobs:
+  test:
+
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: [ 3.8.x, 3.9.x,3.10.x ,3.11.x ]
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v3
+        with:
+          python-version: ${{ matrix.python-version }}
+          architecture: "x64"
+      - name: Install dependencies
+        run: |
+          pip install poetry==1.6.1
+          poetry install --all-extras
+          python -m pip install pytest
+
+      - name: Run tests
+        run: |
+          poetry install --all-extras
+          cd test
+          poetry run pytest

CONTRIBUTING.md

@@ -0,0 +1,133 @@
+[EN](#en) | [CN](#cn)
+-------------------
+
+# [How to contribute](#en)
+
+We welcome everyone to contribute to the project. If you would like to contribute to the project, please read the
+following.
+
+## Community Contact
+
+- Ask any issues directly on Github.
+- Join our Telegram group: https://t.me/Openai_LLM
+
+## Branch description
+
+- Our `main` branch is the default release branch, please do not submit code directly to this branch.
+- Our `dev` branch is the development branch, if you want to contribute to the project, please submit code to this
+  branch, we will merge the `dev` branch into the `main` branch regularly.
+- Our documentation is published at https://github.com/LlmKira/Docs and any form of contribution is accepted.
+- Our framework packages are published on pypi, changes to the sdk will only trigger Release CI when a new OpenAPI is
+  released.
+
+## CI/CD
+
+Our CI/CD service is run by `GitHub Actions`, and every commit of `dev` triggers the CI/CD process or manually triggered
+by the Release Manager. `main` branch consists of
+Release
+Manager or Manager triggered manually.
+
+## Content specifications
+
+- Do not submit personal information.
+- Please use the PEP8 specification for naming.
+- The formatting operation is completed by Reviewer, so there is no need to worry about formatting issues.
+- Make sure all commits are atomic (one feature at a time).
+- We use pydantic>2.0.0 for data verification. You can submit the 1.0.0 version of the code (this is highly
+  discouraged), but we will upgrade it to when released.
+  2.0.0.
+- Fixed Logger needs to be printed at the head or tail of the function of this layer, not at the calling statement
+  level. Loggers that do not conform to the specifications need to be deleted after debugging.
+- The printed content of Logger is concise and clear. It starts with English capital letters and does not require
+  punctuation at the end. Do not use `:` to separate, use `--`
+  Separate parameters.
+- It is recommended to use `assert` in the function header for parameter verification, and do not use `if` for parameter
+  verification.
+- If it is not a reading function, please throw an exception if the execution fails and do not return `None`.
+- Issues must be marked with `# TODO` or `# FIXME`, and add `# [Issue Number]` after `# TODO` or `# FIXME`, if there is
+  no Issue
+  Number, please create an Issue when submitting.
+- Please do not use `str | None` `:=` `list[dict]` and other new features.
+
+## Compatibility instructions
+
+Our code needs to be compatible with Python 3.8+, please do not use new features such as `str | None` `:=` `list[dict]`
+and so on.
+Of course, if you want to use new features, we also welcome your contributions. Release Manager will do compatibility
+checks when releasing new versions.
+
+## Add plugins to the registry
+
+If you want to add your plugin to the registry, please submit the `llmkira/external/plugin.py` file update, we will
+automatically
+synchronize to the registry and test through CI.
+
+---------------
+
+# [如何贡献](#cn)
+
+我们欢迎每一个人为项目做出贡献。如果你想要为项目做出贡献，请阅读以下内容。
+
+## 社区联系
+
+- 直接在Github上提出任何问题。
+- 加入我们的Telegram群组：https://t.me/Openai_LLM
+
+## 分支说明
+
+- 我们的 `main` 分支为默认发布分支，请勿向此分支直接提交代码。
+- 我们的 `dev` 分支是开发分支，如果你想要为项目做出贡献，请向此分支提交代码，我们会定期将 `dev` 分支合并到 `main` 分支。
+- 我们的文档发布在 https://github.com/LlmKira/Docs ，接受任意形式的贡献。
+- 我们的框架包发布于 pypi，对 sdk 的更改只会在发布新的 OpenAPI 时才会触发 Release CI。
+
+## CI/CD
+
+我们的 CI/CD 服务由 `GitHub Actions` 运行，`dev` 的每一次提交都会触发 CI/CD 流程或 Release Manager 手动触发。`main` 分支由
+Release
+Manager 或 Manager 手动触发。
+
+## 内容规范
+
+- 不要提交个人信息。
+- 命名请使用 PEP8 规范。
+- 格式化操作由 Reviewer 完成，不需要担心格式化问题。
+- 确保所有提交都是原子的（每次提交一个功能）。
+- 我们使用 pydantic>2.0.0 来进行数据校验，你可以提交 1.0.0 版本的代码（非常不建议这样做），但是我们会在发布时将其升级到
+  2.0.0。
+- 固定式 Logger 需要在本层函数的头部或尾部进行打印，不要在调用语句层打印。调试后不合规范的 Logger 需要删除。
+- Logger 的打印内容简洁明了，使用英文大写字母开头，结尾不需要标点符号，不要使用 `:` 分隔，使用 `--`
+  分隔参数。
+- 推荐在函数头部使用 `assert` 进行参数校验，不要使用 `if` 进行参数校验。
+- 如果不是读取函数，执行失败请抛出异常，不要返回 `None`。
+- 问题必须注明 `# TODO` 或 `# FIXME`，并且在 `# TODO` 或 `# FIXME` 后面加上 `# [Issue Number]`，如果没有 Issue
+  Number，请在提交时创建 Issue。
+- 请不要使用 `str | None` `:=` `list[dict]` 等新特性。
+
+- 错误示例
+
+```python
+cache = global_cache_runtime.get_redis()
+if not cache:
+    raise Exception("Redis not connected")
+```
+
+- 正确示例
+
+```python
+cache = global_cache_runtime.get_redis()
+# cache 由函数 get_redis() 返回，raise Exception("Redis not connected") 会抛出异常，不需要返回 None
+```
+
+```python
+cache = dict().get("cache")
+assert cache, "Redis not connected"
+```
+
+## 兼容说明
+
+我们的代码需要兼容 Python 3.8+，请不要使用 `str | None` `:=` `list[dict]` 等新特性。
+当然如果你想要使用新特性，我们也欢迎你的贡献，Release Manager 会在发布新版本时做兼容性检查。
+
+## 添加插件到注册表
+
+如果你想要将你的插件添加到注册表，请提交 `llmkira/external/plugin.py` 文件更新，我们会自动通过 CI 同步到注册表并测试。

Dockerfile

@@ -24,7 +24,8 @@ FROM python:3.11-slim-buster as runtime
 RUN apt update &&  \
     apt install -y npm &&  \
     npm install pm2 -g && \
-    pip install poetry==1.6.1
+    pip install poetry==1.6.1 && \
+    apt install ffmpeg -y
 
 ENV VIRTUAL_ENV=/app/.venv \
     PATH="/app/.venv/bin:$PATH"

NOTICE.MD

@@ -1,27 +1,35 @@
 # NOTICE OF THIS PROJECT
 
 ```
-Copyright 2022 sudoskys
+Copyright (C) 2023  sudoskys
 
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
 
-    http://www.apache.org/licenses/LICENSE-2.0
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
 
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
 ```
 
+## GPL2 License
+
+The GPL2 license applies to the denpendencies of this project:
+
+- pyTelegramBotAPI
+
 ## MIT License
 
 The MIT license applies to the files in:
 
 - folder: "llmkira/sdk/func_calling" from https://github.com/nonebot/nonebot2
 - folder: "llmkira/sdk/memory/redis" from https://github.com/langchain-ai/langchain
+- code: "llmkira/sdk/schema.py@parse_from_pydantic" from https://github.com/jxnl/instructor
 
 ## 仅对部署者的安全指南(For Deployer Only)
 

README.md

@@ -31,6 +31,8 @@
   <a href="https://llmkira.github.io/Docs/dev/basic">🧀 开发文档</a>
   &
   <a href="README_EN.md">📝 English Readme</a>
+  &
+  <a href="CONTRIBUTING.md">🤝 贡献必看</a>
 </p>
 
 
@@ -139,12 +141,10 @@ docker-compose -f docker-compose.yml up -d
 
 ```
 
-
 更新镜像使用 `docker-compose pull`。
 
 在 docker 中查看 Shell，使用 `docker exec -it llmbot /bin/bash`，输入 `exit` 退出。
 
-
 ### 🍔 Shell
 
 人工使用Pm2启动，需要自行安装 `redis` ，`rabbitmq` ，`mongodb` 。

README_EN.md

@@ -31,6 +31,8 @@
   <a href="https://llmkira.github.io/Docs/en/dev/basic">🧀 Dev Docs</a>
   &
   <a href="README.md">📝 中文Readme</a>
+  &
+  <a href="CONTRIBUTING.md">🤝 How to contribute</a>
 </p>
 
 > Any issues with deployment? Submit an Issue to help us fix SLA

docs/app.md

@@ -228,6 +228,14 @@ https://github.com/LlmKira/Openaibot/issues/284
 
 这里被动态消息组构建器实现了。
 
+## 多后端必要性的论证
+
+在非 Function Call 场景下，可以使用其他的 LLM 回应消息。
+
+但是目前本项目已经和 Openai 深度绑定，所以这是一个低优先级事项。
+
+> 这里被多后端兼容性实现了。根据模型决定后端。
+
 # 问题
 
 ### 产品定位
@@ -293,12 +301,6 @@ provider 不为空时，则启用此模式。
 
 ![ex](SeriveProvider.svg)
 
-## 多后端必要性的论证
-
-在非 Function Call 场景下，可以使用其他的 LLM 回应消息。
-
-但是目前本项目已经和 Openai 深度绑定，所以这是一个低优先级事项。
-
 ### RAG
 
 Langchain 的 RAG 模型是一个很好的参考。
@@ -348,4 +350,25 @@ Openai 的业务逻辑 Api 终于开始开放了。更多的业务逻辑参与
 
 ## 动态配对系统
 
-既然要求每个 tool call 对应一个回复。我们可以预填补空位。然后动态构建更新新的结果。
+既然要求每个 tool call 对应一个回复。我们可以预填补空位。然后动态构建更新新的结果。
+
+## self recognize
+
+自我识别，自我认知。
+
+读自身状态。
+
+## 插件的禁用系统
+
+需要更智能。
+
+## 系统的维护
+
+schema 必须独立。
+
+## 大模型的剑走偏锋
+
+AGI：情绪不拟人，反而要用人的方式操作物理GUI。
+
+审查：语言模型最大的优势本来就是情绪价值。
+

docs/refer.md

@@ -34,4 +34,14 @@ https://mp.weixin.qq.com/s?__biz=Mzg2OTY0MDk0NQ==&mid=2247502626&idx=1&sn=17603e
 
 LangChain？
 https://twitter.com/AravSrinivas/status/1677884199183994881
-https://minimaxir.com/2023/07/langchain-problem/
+https://minimaxir.com/2023/07/langchain-problem/
+
+"""
+1 lock
+2 ...
+1 ...
+2 lock
+1 ...
+2 ...
+3 lock
+"""

docs/test_script/database/note_redis_lpush_usage.py

@@ -5,10 +5,11 @@
 # @Software: PyCharm
 import time
 
-from llmkira.sdk.cache.redis import cache
+from llmkira.sdk.cache import RedisRuntime
 
 
 async def redis():
+    cache = RedisRuntime().get_redis()
     await cache.lpush_data("test", int(time.time()))
     await cache.lpush_data("test", int(time.time()) + 1)
     _data = await cache.lrange_data("test")