ofelia eating up my memory, system running out of memory

[rpasing]

Contribution guidelines

• I've read the contribution guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... I understand that not following the below instructions will result in immediate closure and/or deletion of my issue.
• ... I have understood that this bug report is dedicated for bugs, and not for support-related inquiries.
• ... I have understood that answers are voluntary and community-driven, and not commercial support.
• ... I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description

Recently, ofelia seems to be eating more and more memory on my machine over time, eventually leading to an out-of-memory situation on my machine. It looks like a memory leak with ~20-30MByte/10minutes (2-3MByte/minute), which is of course not a lot, but with a long runtime I run into trouble on my 16G memory machine.

I am running mailcow on this machine in the same setup since a couple of years already. This issue though seems to be new - I cannot exactly tell since when, but not longer than a couple of weeks for sure. I see there are no changes done on mailcow since Feb 15th, so I would suspect this is an issue within ofelia or docker itself, but I don't know where to start digging, and hope to find other affected users here.

Take a look at the VSZ and RSS columns in the below output, which has been taken from within the ofelia-mailcow container. Basically, we can see that the memory footprint of the ofelia process is increasing over time.

I also checked other logs (e.g. docker-compose log of the ofelia container), but I didn't find anything suspicious (i.e. errors or something like that).

Logs:

# while [ true ] ; do date; ps aux --sort=-%mem | head -n 2 ; sleep 600; done
Wed Mar 27 02:12:52 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 20.6 4703396 3295552 ?     Ssl  Mar26   3:52 /usr/bin/ofelia daemon --docker
Wed Mar 27 02:22:52 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 20.6 4703396 3295808 ?     Ssl  Mar26   3:54 /usr/bin/ofelia daemon --docker
Wed Mar 27 02:32:52 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 20.9 4770040 3348032 ?     Ssl  Mar26   3:55 /usr/bin/ofelia daemon --docker
Wed Mar 27 02:42:53 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 21.0 4770040 3368512 ?     Ssl  Mar26   3:57 /usr/bin/ofelia daemon --docker
Wed Mar 27 02:52:53 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 21.3 4836684 3410496 ?     Ssl  Mar26   3:59 /usr/bin/ofelia daemon --docker
Wed Mar 27 03:02:53 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 21.4 4836684 3431488 ?     Ssl  Mar26   4:00 /usr/bin/ofelia daemon --docker
Wed Mar 27 03:12:53 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 21.6 4903328 3463104 ?     Ssl  Mar26   4:02 /usr/bin/ofelia daemon --docker
Wed Mar 27 03:22:53 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 21.7 4903328 3483328 ?     Ssl  Mar26   4:03 /usr/bin/ofelia daemon --docker
Wed Mar 27 03:32:53 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 21.9 4969972 3504576 ?     Ssl  Mar26   4:05 /usr/bin/ofelia daemon --docker
Wed Mar 27 03:42:53 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 21.9 4969972 3504576 ?     Ssl  Mar26   4:07 /usr/bin/ofelia daemon --docker
# Unfortunately had to switch consoles, so a 20 minute gap to the next one:
Wed Mar 27 04:00:47 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 22.1 4969972 3545920 ?     Ssl  Mar26   4:10 /usr/bin/ofelia daemon --docker
Wed Mar 27 04:10:47 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 22.3 5036616 3577280 ?     Ssl  Mar26   4:11 /usr/bin/ofelia daemon --docker
Wed Mar 27 04:20:47 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 22.3 5036616 3577280 ?     Ssl  Mar26   4:13 /usr/bin/ofelia daemon --docker
Wed Mar 27 04:30:48 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 22.4 5036616 3598144 ?     Ssl  Mar26   4:14 /usr/bin/ofelia daemon --docker
Wed Mar 27 04:40:48 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 22.4 5036616 3598144 ?     Ssl  Mar26   4:16 /usr/bin/ofelia daemon --docker
Wed Mar 27 04:50:48 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 22.6 5036616 3618624 ?     Ssl  Mar26   4:18 /usr/bin/ofelia daemon --docker
Wed Mar 27 05:00:48 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 22.6 5036616 3618880 ?     Ssl  Mar26   4:19 /usr/bin/ofelia daemon --docker
Wed Mar 27 05:10:48 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 22.8 5103260 3660608 ?     Ssl  Mar26   4:21 /usr/bin/ofelia daemon --docker
Wed Mar 27 05:20:48 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 23.0 5103260 3681088 ?     Ssl  Mar26   4:23 /usr/bin/ofelia daemon --docker
Wed Mar 27 05:30:48 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 23.0 5169904 3681856 ?     Ssl  Mar26   4:24 /usr/bin/ofelia daemon --docker
Wed Mar 27 05:40:48 PM CET 2024
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        2919  0.2 23.1 5169904 3702336 ?     Ssl  Mar26   4:26 /usr/bin/ofelia daemon --docker

Steps to reproduce:

1. docker-compose up -d
1. docker-compose exec ofelia-mailcow sh
1. while [ true ] ; do date; ps -o pid,vsz,rss,args | head -n 2 ; sleep 600; done

Which branch are you using?

master

Which architecture are you using?

x86

Operating System:

Arch Linux

Server/VM specifications:

16G memory, 4 cores

Is Apparmor, SELinux or similar active?

no

Virtualization technology:

KVM (Hetzner Cloud Server)

Docker version:

26.0.0 (also happened with 25.0.4, upgraded 2 days ago)

docker-compose version or docker compose version:

latest

mailcow version:

2024-02

Reverse proxy:

Nginx

Logs of git diff:

Nothing related to ofelia, only data/conf/nginx, data/conf/sogo and data/web

Logs of iptables -L -vn:

Doesn't seem relevant

Logs of ip6tables -L -vn:

Doesn't seem relevant

Logs of iptables -L -vn -t nat:

Doesn't seem relevant

Logs of ip6tables -L -vn -t nat:

Doesn't seem relevant

DNS check:

104.18.32.7
172.64.155.249

[FedorAdo]

Same story, it started a week ago, out of nowhere. In a day ofelia reaches 3 GB. I end up restarting.
Solr, clam, ipv6 are disabled in the config. I didn’t make any other changes to the configs or the web muzzle settings. 1 domen 13 mailboxes.

[testos77]

Can confirm the bug. System is running in Arch Linux. Since mailcow docker installation hasn't changed before the bug ocurred, maybe a package dependency outside of the container environment causes this faulty behavior?

[mkuron]

This is a known problem (mcuadros/ofelia#157, #4272] that ofelia never fixed. It was suggested a while ago to replace ofelia with its fork chadburn (#4362), but that fork isn‘t receiving much maintenance anymore either.

[sadorowo]

can confirm this issue, however, it's a big one for me, because I have only 2 GBs of RAM. I can only shutdown ofelia container or increase swap, which isn't ideal solution.

edit: now i managed to avoid this issue by increasing vm.swappiness in sysctl config

[craeckor]

@sadorowo
May I ask how you managed to run Mailcow with 2GB RAM?
Even on my system with Alpine 3.17 with the absolute minimum, it still eats almost 4GB of RAM.
Do you have like 2GB of swap or something?

[sadorowo]

@sadorowo May I ask how you managed to run Mailcow with 2GB RAM? Even on my system with Alpine 3.17 with the absolute minimum, it still eats almost 4GB of RAM. Do you have like 2GB of swap or something?

oh yeah i recently added swap too, because other services that I'm running needed more

[testos77]

After one week my 16 GB are eaten up by this known problem (mcuadros/ofelia#157, https://github.com/mailcow/mailcow-dockerized/issues/4272]

I really hope this bug is taken seriously and will be fixed soon.

[Pingger]

Same issue here.

ofelia a "crontab for docker" is one of the worst docker tools I have ever seen. A crontab should at most use a few MiB of memory and not multiple GiB.

As this apparently a known issue for years with, I petition the mailcow team to switch to something else. Literally anything else, because it can't possibly get worse.

My Systems:
Ubuntu 22.04 LTS
Alpine 3.17

The screenshot from the alpine instance after 20h of uptime. In ubuntu the memory consumption rises even faster...

My current workaround is a script, that I run hourly, that restarts the ofelia container if ofelia uses more than 500MB:

#!/usr/bin/fish
cd /opt/mailcow-dockerized/
# Get usage using ps
set usage (ps -o comm,rss | grep ofelia | grep -oPe "[0-9]+(k|m|g)")

# convert lowercase unit to upper-case
set usage (string upper $usage)

# convert human-readable value to bytes
set usage (numfmt --from=iec $usage)

echo $usage
if test $usage -gt 500000000;
        docker-compose restart ofelia-mailcow;
else
        echo "usage ok"
end

(updated script with input from #5807 (comment))

[xenorio]

Just to further showcase the severity of this: I was led here by ofelia munching on 27G of memory, maxing out all available memory space it could. Server has 64G total.

Somewhat ironically, I guess I'll set up a cron job to regularly restart the ofelia container

[hydrandt]

I'm temporarily resolving it with while true; do sleep 1000; docker compose restart ofelia-mailcow; done (in tmux:)

[milkmaker]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[hydrandt]

Unless this has been resolved, I believe this ticket should stay open.

[longz3r]

We need a way to disable it

[rushowr]

I'm experiencing this issue as well

[rushowr]

Quick note, after implementing the ofelia container restart, my system was still locking up. Creating a similar restart crontab job for the watchdog container did the trick

[milkmaker]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[timwhite]

Still an issue

[rushowr]

still an issue, occasionally reinstalling mailcow for testing, still having the same problem

[milkmaker]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[rushowr]

still an issue