Stream copies of all logs to syslog server

[artntek]

Goal

Aggregate all k8s logs (application, system, kubernetes control plane), and stream a copy to our existing central syslog server for archival purposes (e.g. for forensic analysis on access logs, etc.)

Anything on our network can push logs to the syslog server at 128.111.85.44:514

Explanation

By default, logs from containers (including application pods and most system components running as containers) are written to files on the node's filesystem, typically under /var/log/pods/ (and in our case, also symlinked under /var/log/containers/); eg:

brooke@k8s-node-7:~$ la /var/log/pods/arctic_metacatarctic-0_c16216b2-f0f0-4469-b438-a4a0b536d388/metacat/
total 15M
drwxr-xr-x 2 root root  144 Jun 24 13:23 .
drwxr-xr-x 5 root root   50 Jun 18 19:48 ..
-rw-r----- 1 root root 1.8M Jun 24 15:18 0.log
-rw-r--r-- 1 root root 775K Jun 23 20:44 0.log.20250623-110013.gz
-rw-r--r-- 1 root root 1.3M Jun 24 06:42 0.log.20250623-204425.gz
-rw-r--r-- 1 root root 920K Jun 24 13:23 0.log.20250624-064226.gz
-rw-r----- 1 root root  11M Jun 24 13:23 0.log.20250624-132314

(The logs are rotated by containerd - there's nothing in logrotate or cron, on k8s-node-7)

Nick said:

those are on the / file system and get backed up nightly.

Backups go to Anacapa, currently we are keeping 90 daily and 36 monthly, however they have only been running for a few months, and I’m also battling running out of space
https://github.nceas.ucsb.edu/NCEAS/Computing/blob/master/server_backup_overview.md

They would need to end up in the server’s syslog if they were going to be forwarded to the remote syslog server, I’m keeping all logs collected by that server. Compression has kept the size pretty small so far

MatthewB said:

maybe my first step should be to send the logs there, and then look into the ELK stack thing after that

Nick said:

i’m good with that