Debugging

Debugging podman-based services which fail to start

The filebeat, opensearch and mysql services are containerised using podman. If the podman command fails, in some cases the reason for the failure is not surfaced in the syslog or journal for the service. Debugging this involves trying to run the equivalent command as the podman user but not under systemd. By default, all these services run on the control node.

Using the opensearch service as an example.

1. SSH into the control node.
2. Ensure the service is stopped: sudo systemctl stop opensearch.
3. Find the opensearch unit file, e.g as shown in systemctl status opensearch.
4. Copy the ExecStart command from this.
5. Change to the podman user and that user's home directory:

   sudo su podman
   cd
   

6. Run the ExecStart command but with the -d/--detach option removed.
7. Observe any error messages on stderr.