Skip to content

Commit

Permalink
Merge develop into stable for v4.2.0 (#23)
Browse files Browse the repository at this point in the history
* A few optimizations

- Reduce Docker image size
- Migrate requirements.txt to Pipfile
- Remove unused dependencies
- Add a few unnecessary files in .dockerignore

* Pull modmail-dev#3281

* Pull modmail-dev#3279

* Pull modmail-dev#3278

* Pull modmail-dev#3277 & proper indentation

* Squashed commit of the following:

commit 2e32d6e
Merge: 51a023c b21037c
Author: Martin <[email protected]>
Date:   Sun Apr 9 21:22:08 2023 +0200

    Merge branch 'modmail-dev:master' into master

commit 51a023c
Author: Martin <[email protected]>
Date:   Mon Apr 3 13:35:56 2023 +0200

    Changelog+Small fixes

commit f5f25b5
Author: Martin <[email protected]>
Date:   Mon Apr 3 13:10:35 2023 +0200

    confirm_thread_creation Buttons instead of reactions

commit 11e094c
Merge: ebd38d3 e919304
Author: Taku <[email protected]>
Date:   Tue Mar 14 05:19:54 2023 -0700

    Merge branch 'master' into master

commit ebd38d3
Author: Taku <[email protected]>
Date:   Tue Mar 14 05:18:17 2023 -0700

    Fix indentations

commit b77e435
Author: Martin <[email protected]>
Date:   Sun Mar 12 17:22:42 2023 +0100

    Reminder Plugin

    Created a reminder plugin

* Pull modmail-dev/modmail#modmail-dev#3271

* Squashed commit of the following:

commit 13437cd
Author: baptiste0928 <[email protected]>
Date:   Tue Mar 21 22:29:24 2023 +0100

    Format with black

commit 8a6ecd6
Merge: d9e364e 02250e8
Author: Taku <[email protected]>
Date:   Tue Mar 21 13:30:40 2023 -0700

    Merge branch 'development' into feat/guild-icon-size

    Signed-off-by: Taku <[email protected]>

commit d9e364e
Author: baptiste0928 <[email protected]>
Date:   Sun Mar 19 20:17:35 2023 +0100

    Update changelog

commit d0bfeb4
Author: baptiste0928 <[email protected]>
Date:   Sun Mar 19 20:06:20 2023 +0100

    Set size for guild icons on embeds

* Format with black

* Pull modmail-dev/modmail#modmail-dev#3233

* Squashed commit of the following:

commit cf3444a
Author: Khakers <[email protected]>
Date:   Tue Jan 24 14:13:15 2023 -0800

    Add dm_channel_id field to log entry

* Update changelog, workflow & remove bandit

* Pull modmail-dev#3272 + lock dependencies

* Add docs boilerplate + bumppython version

* Dpy bump, add versions for ?about

* Format with black

* Update README.md

* Fix permission error when installing plugin

* Add new mention_message config, black + ruff

* Switch from root to app user for Docker

* Add support for trailing space in prefix

* Add logviewer as built-in plugin

* logviewer plugin enhancements

* Minor logviewer changes

* Update changelog + logviewer pomelo change

* add pomelo for logbase

* Change logviewer favicon

* Refactor loglist object with proper serialization

* Remove favicon to untrack later

* fix log entry key error

* Replace local favicon with remote avatar

* Add markdown header support

* Switch repo link and branch

* Change more repo links

* Handle properly when loglist is the index page

* toggle off internal message checkbox on load

* Remove trailing slash, fix broken log url on index

* Fix broken default avatar url

* Fix pathing issue on displaying log list in index

* Add placeholder for not logged in user

* Fix blockquote parsing emote ending

* Whoops, remove comment

* Fix blockquote parsing + oauth navbar indicator

* Update docs

* config and docs change

* Update FUNDING.yml

* Update docs

* Update ignores + workflow

* Tweak loadcheck

* tweak loadcheck (2)

* blegh

* ✨

* Attempt to fix invalid guild_id on test

* Docs update

* Terminate process more gracefully

* Update dependencies for Python 3.11

* Refactor logviewer pomelo changes

* Minify docker image size

* Remove unused files

* bump black to fix checks

* Update dev requirements

* Update workflow

* More pomelo changes

* Replace avatar with local asset

* Fix broken author avatar on `logs` command

* Implement pomelo on channel name

* Format with black

* Add pre-commit hook

* Update requirements

* Overhaul block functionality

* Add plugin uninstall alias

* Update python badge

* Update README.md

* Update README.md

* Fix block datetime not being timezone aware

* Remove cairosvg

* Fix copy paste mistake

* Another copy paste issue sigh

* Fix NoneType on parse

* Improve block check and add message cooldown

* bump dpy

* Update internal dpy version

* Update ?info command for pomelo

* Bump version

* Fix missing package error

* Fix index compile error

* Fix space in filepath failing to load local plugin

* Escape codeblock language in class name

* Fix genesis message retrieval failure

* Update docs

* Don't build image on PR

* Blocklist rewrite (#11)

* Add blocklist

blocklist.py contains the initial classes and code for the new blocklist system.
The intention of this system is to unify blocking to one new mongodb collection and vastly simplify bot code required to manage it. Additionally, most simply methods should be side effect free.

* Add migrations.py

migrations.py contains functions to migrate legacy blocklists to the new blocklist. contains handling for both legacy modmail string and the newer dict storage

* Add blocklist bot.py integration

* rename BlockedUser to BlocklistItem & add more blocklist methods

Adds add_block, unblock_id, and is_id_blocked to blocklist class

* migrate block and unblock commands to new blocklist

* fix overzealous ide rename

* add get_all_blocks method and documentation + cleanup to blocklist.py

* refactor blocked command to use new block system

* cleanup block command code

now with more DRY

* migrate bot.py is_blocked method to new blocklist system

* migrate whitelist command and refactor is_id_blocked to return tuple

is_id_blocked now returns a tuple[bool, Optional[BlocklistItem]] so the calling code can get information about the block easily

* rename BlocklistItem to BlocklistEntry

* Add deprecation warning logs to legacy block code and docstrings to bot.py block related code

* clean up migrations.py code

* Add [p]migration command

* add final deprecation docstrings and refactor contact command to use blocklist.is_user_blocked() instead of core is_blocked()

* remove blocklist print statements

* Format with black

---------

Co-authored-by: Raiden Sakura <[email protected]>

* fix note code

upgrade signature of note code state store_user method to match expected discord.py method

use a typed checked dict for author instead of making a declaring a class in a for loop that replicates a different type. hopefully a type checker would raise warnings the next time one of these changes

* Fix failing tests and docker build

* fix blocklist bugs (#13)

* fix missing await statements in new block method usage

* remove Self type importing and usage

* rename placeholder variable name

* StrEnum is only in 3.11...

---------

Co-authored-by: Raiden Sakura <[email protected]>

* oops

* Fix object is not subscriptable for 3.8

* Pull 2 changes from upstream (#14)

* Fixed unset registry_plugins_only causing non-registry plugins to fail to install

* To avoid confusion, config.get() only accepts one pos argument now

---------

Co-authored-by: Taku <[email protected]>

* Format with black

* More blocklist fixes (#15)

* update requirements.txt

* fix blocklist migration detection

* fix migrations.py type hints for python 3.8

* Add index for channel_id

Channel_id is the main query field for nearly every log related database operation and for some reason it isn't indexed. This should speed up all log related database reliant operations especially on large collections

* Update CHANGELOG.md

Update changelog with my changes

* Remove CairoSVG dependencies from Dockerfile

* Improve is_image detection to actually detect images (excluding videos) and add content_type value

* Fix logviewer positional arg error

* partially reverts 2f76202 which added lottie sticker support and updates dependency lock (#20)

* partially reverts 2f76202 which added lottie sticker support and updates dependency lock

* Lint

---------

Co-authored-by: Raiden Sakura <[email protected]>

* Synchronize thread NSFW status and title with the database (#18)

* Changing the thread title or nsfw status now immediately updates in the database

!Changes signature of thread.set_title()

adds update_nsfw and update_title methods to the mongodb client

* Lint

---------

Co-authored-by: Raiden Sakura <[email protected]>

* Refactor docker workflow to multiplatform build (#21)

Upgrades out of date action & adds dependency actions (QEMU & BuildX) for multiplatform build. Sets platforms to Linux/amd64 and Linux/arm64

* Update funding

* Remove internal logviewer plugin

* Add .gitignore for local plugin

* Migrate to poetry

* Update workflow for poetry

* Update workflow

* Oops, forgot to install deps

* Update launch command

* Attempt to solve image not building arm64

* Add missing libffi

* Refactor Dockerfile

* Resolve deprecated pkg_resources warning

* Update changelog

* Bump workflow versions

* oops

* Add [speed] extra for dpy

* Bump dpy, don't exit when dpy version mismatch

* Fix sticker replies not sending

* Update changelog

* Improved join/leave message for multiple servers.

From modmail-dev@0086e04

* Fix code scanning alert #5

* Fixed compat with MissingRequiredArgument requiring additional param

modmail-dev@9464c5d

* Fix plugin update bug on broken plugin

https://github.com/modmail-dev/Modmail/pull/3295/files

* Update CHANGELOG.md

* Fix rate limit issue on raw reaction add/remove events

https://github.com/modmail-dev/Modmail/pull/3306/files

* Remove Heroku Reference from Debug Command

modmail-dev@d891ad1

* Update CHANGELOG.md

* Remove unused files

* Update pip

* Bump aiohttp and use [speedups]

* Update .dockerignore

* Rename project

* Add shell script for pm2

* Update docs

* Update env example

* Remove app.json

* Update dependencies

* Update readme

* Add config to manage how long logs are stored (modmail-dev#3257)

* Add presence intent into .env.example

* Formating

* Fix missing guild icon url

* Revert gyazo image embedding

* Fix overly permissive regex

* Specify fork on metadata

* Update changelog

* Bump version

---------

Co-authored-by: khakers <[email protected]>
Co-authored-by: Taku <[email protected]>
  • Loading branch information
3 people authored Dec 27, 2023
1 parent e46fbb1 commit aba4834
Show file tree
Hide file tree
Showing 116 changed files with 2,761 additions and 14,861 deletions.
7 changes: 4 additions & 3 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -130,26 +130,27 @@ dmypy.json
package-lock.json
node_modules/

# Modmail
# Unused Modmail files
config.json
plugins/
!plugins/registry.json
!plugins/@local/
temp/
test.py

# Other stuff
# Unused repository files
.env.example
.gitignore
.dockerignore
.github/
.pre-commit-config.yaml
app.json
Procfile
pyproject.toml
*.md
.*.json
Dockerfile
docker-compose.yml
requirements.txt
LICENSE
PRIVACY.md

Expand Down
50 changes: 46 additions & 4 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,5 +1,47 @@
# Your Discord bot token
TOKEN=MyBotToken
LOG_URL=https://logviewername.herokuapp.com/
GUILD_ID=1234567890
OWNERS=Owner1ID,Owner2ID,Owner3ID
CONNECTION_URI=mongodb+srv://mongodburi

# ID of your Discord server
GUILD_ID=123456

# ID of the server the thread channels will be created in
# MODMAIL_GUILD_ID=123456

# Discord ID of the owners
OWNERS=123456,123456

# Connection URI for your mongodb
CONNECTION_URI=mongod+srv://mongo.net

# The url of logviewer for viewing self-hosted logs
# LOG_URL=https://logviewername.domain.com/

# The path to your log viewer extending from your `LOG_URL`, defaults to "/logs"
# LOG_URL_PREFIX=/logs

# Enable presence intent for the bot, defaults to false
# ENABLE_PRESENCE_INTENT=false

# A github personal access token with the repo scope to access private repository
# GITHUB_TOKEN=abcdef

# If set to true, only plugins that are in the registry can be loaded
# REGISTRY_PLUGINS_ONLY=false

# Client ID for your application, used by logviewer`
# OAUTH2_CLIENT_ID=12345

# Client Secret for your application, used by logviewer
# OAUTH2_CLIENT_SECRET=abcdef

# The redirect URL for logviewer OAuth. You must also add this URL in your application's Redirects section
# OAUTH2_REDIRECT_URI=https://logviewername.herokuapp.com/callback

# The host address logviewer will bind itself to. Defaults to 0.0.0.0
# HOST=0.0.0.0

# The port logviewer plugin will listen to. Defaults to 8000
# PORT=8000

# The number of logs to display per page on logviewer. Defaults to 25
# LOGVIEWER_PAGINATION=25
1 change: 1 addition & 0 deletions .github/FUNDING.yml
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
ko_fi: raidensakura
github: raidensakura
22 changes: 14 additions & 8 deletions .github/workflows/docker-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,6 @@ on:
- 'develop'
tags:
- 'v*'
pull_request:
branches:
- 'develop'

env:
REGISTRY: ghcr.io
Expand All @@ -25,18 +22,24 @@ jobs:

steps:
- name: Checkout repository
uses: actions/checkout@v3

uses: actions/checkout@v4

- name: Set up QEMU
uses: docker/setup-qemu-action@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Log in to the Container registry
uses: docker/login-action@v2
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v4
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
Expand All @@ -51,9 +54,12 @@ jobs:
type=raw,value=latest,enable=${{ github.event.release.prerelease == false }}
- name: Build and push Docker image
uses: docker/build-push-action@v3
uses: docker/build-push-action@v5
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
6 changes: 3 additions & 3 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8"]
python-version: ["3.11"]
name: Python ${{ matrix.python-version }} on ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
architecture: x64
Expand Down
25 changes: 12 additions & 13 deletions .github/workflows/load-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,40 +16,39 @@ jobs:

name: Load Modmail - Python ${{ matrix.python-version }}
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

- name: Cache venv
id: cache-venv
- uses: Gr1N/setup-poetry@v8
- name: Cache poetry
id: cache-poetry
uses: actions/cache@v3
with:
path: .venv
key: ${{ matrix.python-version }}-${{ hashFiles('Pipfile') }}-${{ secrets.CACHE_V }}
path: ~/.cache/pypoetry/virtualenvs
key: ${{ runner.os }}-poetry-${{ hashFiles('poetry.lock') }}

- name: Maybe make pipenv
- name: Maybe install dependencies
if: steps.cache-venv.outputs.cache-hit != 'true'
run: |
pip install pipenv && PIPENV_VENV_IN_PROJECT=1
pipenv install --deploy
run: poetry install --no-root

- name: Run script loadcheck.py
run: |
pipenv run python .github/workflows/scripts/loadcheck.py
poetry run python .github/workflows/scripts/loadcheck.py
env:
TOKEN: ${{ secrets.DISCORD_TEST_BOT }}
CONNECTION_URI: ${{ secrets.CONNECTION_URI }}
LOG_URL: https://domain.example
GUILD_ID: "616969119685935162"
MODMAIL_GUILD_ID: "1100011945207271494"
OWNERS: "243316261264556032,505386507440488458"
DATA_COLLECTION: off
LOG_LEVEL: DEBUG

- name: Save Modmail output as Artifact
if: always() # still run if prev step failed
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: "Modmail log - Python ${{ matrix.python-version }}"
path: modmail.log
2 changes: 1 addition & 1 deletion .github/workflows/scripts/loadcheck.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@

file = open("modmail.log", "w")
proc = subprocess.Popen(
"pipenv run bot",
"poetry run python bot.py",
stdout=file,
stderr=subprocess.STDOUT,
shell=True,
Expand Down
43 changes: 43 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,46 @@ however, insignificant breaking changes do not guarantee a major version bump, s

# [UNRELEASED]

# v4.2.0

### Breaking
- Completely rewritten blocklist system. Blocklisting now runs off its own mongoDB collection. This once again introduces backwards incompatible schema changes, so a manual migration is required. You may upgrade from both v4.0 and v4.1 using the `[p]migrate blocklist` command. This removes any need to perform the previous migration steps in v4.1.0, you may upgrade directly to this version. After running the command, blocklist functionality will return and legacy config based blocks will have been deleted. You should always back up your config before migration.
- Migrated package manager from pipenv to Poetry. Dockerfile and GitHub Actions are also now set up to install using Poetry.

### Deprecated
- Legacy blocklist properties are deprecated and no longer function. They now log a warning when used and provide no functionality. They have been replaced with methods in blocklist.py

### Added
- Added `content_type` to attachments stored in the database.
- Introduced multiplatform Docker workflow for GitHub Actions.
- Improved join/leave message for multiple servers.
- Added `log_expiration` config to auto-delete old logs on a schedule.

### Changed
- Changing a thread's title or NSFW status immediately updates the status in the database.

### Removed
- The logviewer plugin is no longer included locally.
- Removed lottie sticker support and its dependencies due to unnecessary bulk.
- Removed gyazo image embedding.

### Fixed
- Persistent notes have been fixed after the previous discord.py update.
- `is_image` now is true only if the image is actually an image.
- Resolved deprecated pkg_resources warning in certain Python environment.
- Improved genesis message retrieval checks where it would fail under certain circumstances.
- Fixed failure on installing local plugins when plugin path contains whitespace.
- Fixed sticker replies not being sent in a Modmail thread.
- Fixed plugin update command where it would fail to remove the plugin when the plugin is invalid.
- Fixed MissingRequiredArgument requiring additional param error on certain commands.
- Fixed rate limit issue on raw reaction add/remove events.

### Internal
- Add `update_title` and `update_nsfw` methods to `ApiClient` to update thread title and nsfw status in the database.
- `thread.set_title` now requires `channel_id` to be passed as keyword arguments.
- New `thread.set_nsfw_status` method to set nsfw status of a thread.
- Updated discord.py to 2.3.2, with [speed] extra.

# v4.1.0

### Breaking
Expand Down Expand Up @@ -46,6 +86,9 @@ however, insignificant breaking changes do not guarantee a major version bump, s
- Fixed blocked roles improperly saving in `blocked_users` config.
- Fixed `?block` command improperly parsing reason as timestamp.

### Internal
- `ConfigManager.get` no longer accepts two positional arguments: the `convert` argument is now keyword-only.

# v4.0.2

### Breaking
Expand Down
40 changes: 25 additions & 15 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,27 +1,37 @@
FROM python:3.11-alpine as base

RUN apk update && apk add git \
# cairosvg dependencies
cairo-dev cairo cairo-tools \
# pillow dependencies
jpeg-dev zlib-dev
jpeg-dev zlib-dev && \
pip install --upgrade pip && \
adduser -D -h /home/modmail -g 'Modmail' modmail

FROM base as python-deps
WORKDIR /home/modmail

RUN apk add --virtual build-deps build-base gcc libffi-dev
COPY requirements.txt /
RUN pip install --prefix=/inst -U -r /requirements.txt
FROM base as builder

ENV POETRY_NO_INTERACTION=1 \
POETRY_VIRTUALENVS_IN_PROJECT=1 \
POETRY_VIRTUALENVS_CREATE=1 \
POETRY_CACHE_DIR=/tmp/poetry_cache

RUN apk add build-base libffi-dev && \
pip install -U poetry

COPY --chown=modmail:modmail poetry.lock pyproject.toml /home/modmail/

RUN --mount=type=cache,target=$POETRY_CACHE_DIR poetry install --without dev --no-root

FROM base as runtime

ENV USING_DOCKER yes
COPY --from=python-deps /inst /usr/local
ENV VIRTUAL_ENV=/home/modmail/.venv \
PATH="/home/modmail/.venv/bin:$PATH" \
USING_DOCKER=yes

COPY . /modmail
WORKDIR /modmail
COPY --from=builder --chown=modmail:modmail ${VIRTUAL_ENV} ${VIRTUAL_ENV}

CMD ["python", "bot.py"]
COPY --chown=modmail:modmail . .

USER modmail

RUN adduser --disabled-password --gecos '' app && \
chown -R app /modmail
USER app
CMD ["python", "bot.py"]
35 changes: 0 additions & 35 deletions Pipfile

This file was deleted.

Loading

0 comments on commit aba4834

Please sign in to comment.