Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System slowdown after more than 1 day of operation #147

Open
3 tasks done
instantdreams opened this issue Jul 12, 2023 · 0 comments
Open
3 tasks done

System slowdown after more than 1 day of operation #147

instantdreams opened this issue Jul 12, 2023 · 0 comments
Labels
kind/bug status/triage

Comments

@instantdreams
Copy link

Support guidelines

I've found a bug and checked that ...

  • ... the documentation does not mention anything about my problem
  • ... there are no open or closed issues that are related to my problem

Description

When running fail2ban as a docker container on the same host as my nginx proxy manager docker container the system slows down over a 24 hour period

Expected behaviour

For the system to not slow down even if left alone for a week or more

Actual behaviour

I have two servers that are functionally identical - two Raspberry Pi 4B 2GB boards with 120GB SSD via USB3 connectors. They run edge services using docker compose files.

edge1: Linux edge1 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux
edge2: Linux edge2 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux

edge1 containers ```BASH $ docker container ls --all --format "table {{.Names}}\t{{.Size}}\t{{.Networks}}\t{{.Status}}\t{{.RunningFor}}\t{{.State}}" | (sed --unbuffered 1q;sort --key 1) NAMES SIZE NETWORKS STATUS CREATED STATE acme-sh-1 2B (virtual 40.3MB) edge1 Up 3 minutes 4 days ago running diun-1 0B (virtual 54.6MB) edge1 Up 3 minutes 4 days ago running fail2ban 3.12kB (virtual 90.9MB) host Up 3 minutes (healthy) 4 days ago running netdata-1 707kB (virtual 395MB) edge1 Up 3 minutes (healthy) 23 hours ago running npm 42.4MB (virtual 858MB) edge1 Up 3 minutes (healthy) 4 days ago running pihole-1 74.2MB (virtual 368MB) host Up 3 minutes (healthy) 23 hours ago running promtail-1 0B (virtual 185MB) edge1 Up 3 minutes 3 days ago running scrutiny-collector-1 9.37kB (virtual 189MB) edge1 Up 3 minutes 4 days ago running wireguard-1 23kB (virtual 167MB) edge1 Up 3 minutes 4 days ago running ```
edge2 containers ```BASH $ docker container ls --all --format "table {{.Names}}\t{{.Size}}\t{{.Networks}}\t{{.Status}}\t{{.RunningFor}}\t{{.State}}" | (sed --unbuffered 1q;sort --key 1) NAMES SIZE NETWORKS STATUS CREATED STATE acme-sh-2 2B (virtual 40.3MB) edge2 Up 23 hours 4 days ago running diun-2 0B (virtual 54.6MB) edge2 Up 23 hours 4 days ago running duckdns 21.7kB (virtual 40.8MB) edge2 Up 22 hours 22 hours ago running netdata-2 26.6kB (virtual 394MB) edge2 Up About a minute (healthy) About a minute ago running pihole-2 128MB (virtual 422MB) host Up 23 hours (healthy) 23 hours ago running promtail-2 0B (virtual 185MB) edge2 Up 23 hours 3 days ago running scrutiny-collector-2 9.37kB (virtual 189MB) edge2 Up 23 hours 4 days ago running wireguard-2 23kB (virtual 167MB) edge2 Up 23 hours 4 days ago running ```

Generally, both servers need no daily maintenance and the Diun jobs run on Sundays to let me know what containers need updating (thanks for that, btw!).

I wanted to implement fail2ban in a docker container and used this version based on the good work done with Diun. I found a tutorial and followed it, amending the configuration to suit my needs. It works, and I sometimes get ban notifications.

However, after a day or so of usage the following will occur:

  • System load increases
  • Disk I/O increases
  • Zombie processes spike
  • Threads increase

According to my netdata metrics, this is due to fail2ban. I'm just not sure how to see what it was doing at the time.

Steps to reproduce

  1. Start server
  2. Check all docker services are running
  3. Wait 24+ hours
  4. Monitor system performance

Docker info

Client: Docker Engine - Community
 Version:    24.0.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.19.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 9
  Running: 9
  Paused: 0
  Stopped: 0
 Images: 9
 Server Version: 24.0.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.21-v8+
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 1.855GiB
 Name: id-edge1
 ID: 7HKR:XS7T:KP55:WXII:JDYT:SCR5:EF3N:PHZA:XOAK:EWEP:3J7W:NWFN
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Docker Compose config

name: fail2ban
services:
  fail2ban:
    cap_add:
    - NET_ADMIN
    - NET_RAW
    container_name: fail2ban
    dns:
    - 192.168.1.1
    environment:
      F2B_DB_PURGE_AGE: 14d
      F2B_LOG_LEVEL: WARNING
      F2B_LOG_TARGET: STDOUT
      TZ: America/Edmonton
    hostname: edge1
    image: ghcr.io/crazy-max/fail2ban:latest
    network_mode: host
    restart: unless-stopped
    volumes:
    - type: bind
      source: /srv/fail2ban/data
      target: /data
      bind:
        create_host_path: true
    - type: bind
      source: /srv/nginx-proxy-manager/data/logs
      target: /var/log/npm
      read_only: true
      bind:
        create_host_path: true
    - type: bind
      source: /etc/timezone
      target: /etc/timezone
      read_only: true
      bind:
        create_host_path: true
    - type: bind
      source: /etc/localtime
      target: /etc/localtime
      read_only: true
      bind:
        create_host_path: true

Logs

fail2ban  | Setting timezone to America/Edmonton...
fail2ban  | ln: /etc/localtime: File exists
fail2ban  | /entrypoint.sh: line 40: /etc/timezone: Read-only file system
fail2ban  | Setting SSMTP configuration...
fail2ban  | WARNING: SSMTP_HOST must be defined if you want fail2ban to send emails
fail2ban  | Initializing files and folders...
fail2ban  | Setting Fail2ban configuration...
fail2ban  | Checking for custom actions in /data/action.d...
fail2ban  |   Add custom action action-ban-docker-f-b.conf...
fail2ban  |   Add custom action telegram_notif.sh...
fail2ban  | Checking for custom filters in /data/filter.d...
fail2ban  |   Add custom filter npm-general-f-b.conf...
fail2ban  | 2023-07-07 15:49:10,951 fail2ban.configreader   [1]: INFO    Loading configs for fail2ban under /etc/fail2ban
fail2ban  | 2023-07-07 15:49:10,956 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban  | 2023-07-07 15:49:10,958 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban  | 2023-07-07 15:49:10,959 fail2ban                [1]: INFO    Using socket file /var/run/fail2ban/fail2ban.sock
fail2ban  | 2023-07-07 15:49:10,959 fail2ban                [1]: INFO    Using pid file /var/run/fail2ban/fail2ban.pid, [WARNING] logging to STDOUT
fail2ban  | 2023-07-07 15:49:10,967 fail2ban.configreader   [1]: INFO    Loading configs for jail under /etc/fail2ban
fail2ban  | 2023-07-07 15:49:10,969 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/jail.conf']
fail2ban  | 2023-07-07 15:49:10,998 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/paths-debian.conf']
fail2ban  | 2023-07-07 15:49:11,002 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/paths-common.conf']
fail2ban  | 2023-07-07 15:49:11,008 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/paths-overrides.local']
fail2ban  | 2023-07-07 15:49:11,011 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/jail.d/jail.local']
fail2ban  | 2023-07-07 15:49:11,014 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/paths-common.conf', '/etc/fail2ban/paths-debian.conf', '/etc/fail2ban/jail.conf', '/etc/fail2ban/jail.d/jail.local']
fail2ban  | 2023-07-07 15:49:11,016 fail2ban.configreader   [1]: WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'
fail2ban  | 2023-07-07 15:49:11,043 fail2ban.configreader   [1]: INFO    Loading configs for filter.d/npm-general-f-b under /etc/fail2ban
fail2ban  | 2023-07-07 15:49:11,045 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban  | 2023-07-07 15:49:11,047 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban  | 2023-07-07 15:49:11,051 fail2ban.configreader   [1]: INFO    Loading configs for action.d/action-ban-docker-f-b under /etc/fail2ban
fail2ban  | 2023-07-07 15:49:11,053 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban  | 2023-07-07 15:49:11,055 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban  | Server ready
fail2ban  | Setting timezone to America/Edmonton...
fail2ban  | Setting SSMTP configuration...
fail2ban  | WARNING: SSMTP_HOST must be defined if you want fail2ban to send emails
fail2ban  | Initializing files and folders...
fail2ban  | ln: /etc/localtime: File exists
fail2ban  | /entrypoint.sh: line 40: /etc/timezone: Read-only file system
fail2ban  | Setting Fail2ban configuration...
fail2ban  | Checking for custom actions in /data/action.d...
fail2ban  |   WARNING: action-ban-docker-f-b.conf already exists and will be overriden
fail2ban  |   Add custom action action-ban-docker-f-b.conf...
fail2ban  |   WARNING: telegram_notif.sh already exists and will be overriden
fail2ban  |   Add custom action telegram_notif.sh...
fail2ban  | Checking for custom filters in /data/filter.d...
fail2ban  |   WARNING: npm-general-f-b.conf already exists and will be overriden
fail2ban  |   Add custom filter npm-general-f-b.conf...
fail2ban  | 2023-07-08 23:00:12,148 fail2ban.configreader   [1]: INFO    Loading configs for fail2ban under /etc/fail2ban
fail2ban  | 2023-07-08 23:00:12,158 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban  | 2023-07-08 23:00:12,161 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban  | 2023-07-08 23:00:12,162 fail2ban                [1]: INFO    Using socket file /var/run/fail2ban/fail2ban.sock
fail2ban  | 2023-07-08 23:00:12,162 fail2ban                [1]: INFO    Using pid file /var/run/fail2ban/fail2ban.pid, [WARNING] logging to STDOUT
fail2ban  | 2023-07-08 23:00:12,197 fail2ban.configreader   [1]: INFO    Loading configs for jail under /etc/fail2ban
fail2ban  | 2023-07-08 23:00:12,201 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/jail.conf']
fail2ban  | 2023-07-08 23:00:12,254 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/paths-debian.conf']
fail2ban  | 2023-07-08 23:00:12,269 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/paths-common.conf']
fail2ban  | 2023-07-08 23:00:12,283 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/paths-overrides.local']
fail2ban  | 2023-07-08 23:00:12,285 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/jail.d/jail.local']
fail2ban  | 2023-07-08 23:00:12,289 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/paths-common.conf', '/etc/fail2ban/paths-debian.conf', '/etc/fail2ban/jail.conf', '/etc/fail2ban/jail.d/jail.local']
fail2ban  | 2023-07-08 23:00:12,295 fail2ban.configreader   [1]: WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'
fail2ban  | 2023-07-08 23:00:12,356 fail2ban.configreader   [1]: INFO    Loading configs for filter.d/npm-general-f-b under /etc/fail2ban
fail2ban  | 2023-07-08 23:00:12,358 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban  | 2023-07-08 23:00:12,365 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban  | 2023-07-08 23:00:12,379 fail2ban.configreader   [1]: INFO    Loading configs for action.d/action-ban-docker-f-b under /etc/fail2ban
fail2ban  | 2023-07-08 23:00:12,381 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban  | 2023-07-08 23:00:12,397 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban  | Server ready
fail2ban  | 2023-07-08 23:00:18,583 fail2ban.utils          [1]: ERROR   7f9a787270 -- exec: bash /data/action.d/telegram_notif.sh -a start
fail2ban  | 2023-07-08 23:00:18,583 fail2ban.utils          [1]: ERROR   7f9a787270 -- returned 6
fail2ban  | 2023-07-08 23:00:18,584 fail2ban.actions        [1]: ERROR   Failed to start jail 'npm-general-f-b' action 'action-ban-docker-f-b': Error starting action Jail('npm-general-f-b')/action-ban-docker-f-b: 'Script error'
fail2ban  | 2023-07-09 11:50:38,474 fail2ban.actions        [1]: WARNING [npm-general-f-b] 184.70.188.106 already banned

[repeated every 5 minutes until]

fail2ban  | 2023-07-09 12:58:15,383 fail2ban.actions        [1]: WARNING [npm-general-f-b] 184.70.188.106 already banned
fail2ban  | Setting timezone to America/Edmonton...
fail2ban  | Setting SSMTP configuration...
fail2ban  | WARNING: SSMTP_HOST must be defined if you want fail2ban to send emails
fail2ban  | Initializing files and folders...
fail2ban  | ln: /etc/localtime: File exists
fail2ban  | /entrypoint.sh: line 40: /etc/timezone: Read-only file system
fail2ban  | Setting Fail2ban configuration...
fail2ban  | Checking for custom actions in /data/action.d...
fail2ban  |   WARNING: action-ban-docker-f-b.conf already exists and will be overriden
fail2ban  |   Add custom action action-ban-docker-f-b.conf...
fail2ban  |   WARNING: telegram_notif.sh already exists and will be overriden
fail2ban  |   Add custom action telegram_notif.sh...
fail2ban  | Checking for custom filters in /data/filter.d...
fail2ban  |   WARNING: npm-general-f-b.conf already exists and will be overriden
fail2ban  |   Add custom filter npm-general-f-b.conf...
fail2ban  | 2023-07-10 08:24:32,151 fail2ban.configreader   [1]: INFO    Loading configs for fail2ban under /etc/fail2ban
fail2ban  | 2023-07-10 08:24:32,157 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban  | 2023-07-10 08:24:32,160 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban  | 2023-07-10 08:24:32,161 fail2ban                [1]: INFO    Using socket file /var/run/fail2ban/fail2ban.sock
fail2ban  | 2023-07-10 08:24:32,162 fail2ban                [1]: INFO    Using pid file /var/run/fail2ban/fail2ban.pid, [WARNING] logging to STDOUT
fail2ban  | 2023-07-10 08:24:32,176 fail2ban.configreader   [1]: INFO    Loading configs for jail under /etc/fail2ban
fail2ban  | 2023-07-10 08:24:32,178 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/jail.conf']
fail2ban  | 2023-07-10 08:24:32,223 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/paths-debian.conf']
fail2ban  | 2023-07-10 08:24:32,227 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/paths-common.conf']
fail2ban  | 2023-07-10 08:24:32,235 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/paths-overrides.local']
fail2ban  | 2023-07-10 08:24:32,237 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/jail.d/jail.local']
fail2ban  | 2023-07-10 08:24:32,243 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/paths-common.conf', '/etc/fail2ban/paths-debian.conf', '/etc/fail2ban/jail.conf', '/etc/fail2ban/jail.d/jail.local']
fail2ban  | 2023-07-10 08:24:32,245 fail2ban.configreader   [1]: WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'
fail2ban  | 2023-07-10 08:24:32,304 fail2ban.configreader   [1]: INFO    Loading configs for filter.d/npm-general-f-b under /etc/fail2ban
fail2ban  | 2023-07-10 08:24:32,306 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban  | 2023-07-10 08:24:32,309 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban  | 2023-07-10 08:24:32,312 fail2ban.configreader   [1]: INFO    Loading configs for action.d/action-ban-docker-f-b under /etc/fail2ban
fail2ban  | 2023-07-10 08:24:32,325 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban  | 2023-07-10 08:24:32,329 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban  | Server ready
fail2ban  | 2023-07-11 09:46:53,185 fail2ban.actions        [1]: WARNING [npm-general-f-b] 198.161.8.92 already banned
fail2ban  | 2023-07-11 17:05:32,996 fail2ban.actions        [1]: WARNING [npm-general-f-b] 198.161.51.65 already banned

[repeated every minute until]

fail2ban  | 2023-07-11 17:10:14,200 fail2ban.actions        [1]: WARNING [npm-general-f-b] 198.161.51.65 already banned
fail2ban  | 2023-07-12 13:24:47,594 fail2ban.observer       [1]: ERROR   can't start new thread
fail2ban  | Traceback (most recent call last):
fail2ban  |   File "/usr/lib/python3.10/site-packages/fail2ban/server/observer.py", line 221, in run
fail2ban  |     meth(*ev[1:])
fail2ban  |   File "/usr/lib/python3.10/site-packages/fail2ban/server/observer.py", line 361, in db_purge
fail2ban  |     self.add_named_timer('DB_PURGE', self.__db_purge_interval, 'db_purge')
fail2ban  |   File "/usr/lib/python3.10/site-packages/fail2ban/server/observer.py", line 119, in add_named_timer
fail2ban  |     t.start()
fail2ban  |   File "/usr/lib/python3.10/threading.py", line 935, in start
fail2ban  |     _start_new_thread(self._bootstrap, ())
fail2ban  | RuntimeError: can't start new thread
fail2ban  | Setting timezone to America/Edmonton...
fail2ban  | ln: /etc/localtime: File exists
fail2ban  | Setting SSMTP configuration...
fail2ban  | WARNING: SSMTP_HOST must be defined if you want fail2ban to send emails
fail2ban  | Initializing files and folders...
fail2ban  | /entrypoint.sh: line 40: /etc/timezone: Read-only file system
fail2ban  | Setting Fail2ban configuration...
fail2ban  | Checking for custom actions in /data/action.d...
fail2ban  |   WARNING: action-ban-docker-f-b.conf already exists and will be overriden
fail2ban  |   Add custom action action-ban-docker-f-b.conf...
fail2ban  |   WARNING: telegram_notif.sh already exists and will be overriden
fail2ban  |   Add custom action telegram_notif.sh...
fail2ban  | Checking for custom filters in /data/filter.d...
fail2ban  |   WARNING: npm-general-f-b.conf already exists and will be overriden
fail2ban  |   Add custom filter npm-general-f-b.conf...
fail2ban  | 2023-07-12 14:10:04,526 fail2ban.configreader   [1]: INFO    Loading configs for fail2ban under /etc/fail2ban
fail2ban  | 2023-07-12 14:10:04,531 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban  | 2023-07-12 14:10:04,535 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/fail2ban.conf']
fail2ban  | 2023-07-12 14:10:04,536 fail2ban                [1]: INFO    Using socket file /var/run/fail2ban/fail2ban.sock
fail2ban  | 2023-07-12 14:10:04,536 fail2ban                [1]: INFO    Using pid file /var/run/fail2ban/fail2ban.pid, [WARNING] logging to STDOUT
fail2ban  | 2023-07-12 14:10:04,558 fail2ban.configreader   [1]: INFO    Loading configs for jail under /etc/fail2ban
fail2ban  | 2023-07-12 14:10:04,560 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/jail.conf']
fail2ban  | 2023-07-12 14:10:04,593 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/paths-debian.conf']
fail2ban  | 2023-07-12 14:10:04,602 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/paths-common.conf']
fail2ban  | 2023-07-12 14:10:04,606 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/paths-overrides.local']
fail2ban  | 2023-07-12 14:10:04,608 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/jail.d/jail.local']
fail2ban  | 2023-07-12 14:10:04,615 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/paths-common.conf', '/etc/fail2ban/paths-debian.conf', '/etc/fail2ban/jail.conf', '/etc/fail2ban/jail.d/jail.local']
fail2ban  | 2023-07-12 14:10:04,617 fail2ban.configreader   [1]: WARNING 'allowipv6' not defined in 'Definition'. Using default one: 'auto'
fail2ban  | 2023-07-12 14:10:04,655 fail2ban.configreader   [1]: INFO    Loading configs for filter.d/npm-general-f-b under /etc/fail2ban
fail2ban  | 2023-07-12 14:10:04,657 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban  | 2023-07-12 14:10:04,664 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/filter.d/npm-general-f-b.conf']
fail2ban  | 2023-07-12 14:10:04,671 fail2ban.configreader   [1]: INFO    Loading configs for action.d/action-ban-docker-f-b under /etc/fail2ban
fail2ban  | 2023-07-12 14:10:04,673 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban  | 2023-07-12 14:10:04,677 fail2ban.configparserinc[1]: INFO      Loading files: ['/etc/fail2ban/action.d/action-ban-docker-f-b.conf']
fail2ban  | Server ready

Additional info

The latest issue occurred just before 2023-07-12 01:00 MDT. The only entry in the docker logs for this day was "ERROR can't start new thread" at 13:24 MDT, likely when I ssh'd into the server to check why it was so slow.

The fail2ban container can see the NPM logs:

$ docker exec fail2ban ls -lha /var/log/npm/error.log
-rwxrwxr-x    1 1000     1000       25.6K May 26 12:11 /var/log/npm/error.log

The database appears to be valid:

$ ls -lha data/db/
total 212K
drwxrwxr-x 2 idsvc docker 4.0K Jun 27 14:28 .
drwxrwxr-x 6 idsvc docker 4.0K Jul 11 14:48 ..
-rwxrwxr-x 1 idsvc docker 204K Jul 12 14:34 fail2ban.sqlite3

The issue appears to be iowait related:
image

Just before I rebooted the server, the number of fail2ban processes was at 1,655:
image

What am I missing here?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug status/triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@instantdreams