Logs in the merged log view are out of order

[jul1u5]

🔍 Check for existing issues

• Completed

How is Dozzle deployed?

Standalone Deployment

📦 Dozzle version

v8.11.9

✅ Command used to run Dozzle

Fragment of my docker-compose.yml

version: '3.8'

services:
  dozzle:
    image: amir20/dozzle
    container_name: dozzle
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./dozzle:/data
    environment:
      DOZZLE_NO_ANALYTICS: 1
      DOZZLE_AUTH_PROVIDER: simple
    ports:
      - 8088:8080
    restart: unless-stopped
    labels:
      - "com.centurylinklabs.watchtower.enable=true"

🐛 Describe the bug / provide steps to reproduce it

In the merged log view, the logs appear out of chronological order. I don't know yet how to reproduce this consistently.

💻 Environment

$ docker system info
Client:
 Version:    24.0.2
 Context:    default
 Debug Mode: false

Server:
 Containers: 12
  Running: 9
  Paused: 0
  Stopped: 3
 Images: 11
 Server Version: 24.0.2
 Storage Driver: btrfs
  Btrfs: 
 Logging Driver: db
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs db fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 067f5021280b8de2059026fb5c43c4adb0f3f244
 runc version: 0320c58
 init version: ed96d00
 Security Options:
  apparmor
 Kernel Version: 4.4.302+
 Operating System: DiskStation

 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 9.576GiB
 Name: SaltisNAS
 ID: 68bb2e6f-7a61-4fa9-b999-6272926c228c
 Docker Root Dir: /volume1/@docker
 Debug Mode: false
 Username: jul1u5
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No kernel memory TCP limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

📸 If applicable, add screenshots to help explain your bug

Merged view

Container view (home-assistant)

Notice how a single multilined log message below was interspersed with another container's logs (above).

📜 If applicable, attach your Dozzle logs. You many need to enable debug mode. See https://dozzle.dev/guide/debugging.

No response

[amir20]

Unfortunately, in merged view, that's the only way it can be. Dozzle streams each container in parallel, and it has no way to coordinate between the different streams. Which means that sometimes when two containers are emitting logs at the same time, it is possible for the logs to be merged.

I don't know of an easy solution for this. One idea I had was to send groups of logs together so that they could be grouped on the UI. But that's a lot of work.

[jul1u5]

I see, so Dozzle always reads logs for each container independently, regardless if the user is in the merge view?

Would it not be possible to change the listening logic (interaction between Dozzle and Docker) to read multiple container's logs if the merge view is open? What I mean is that with the docker-compose command, you can do the following:

1. read a single container's logs, e.g., docker-compose logs home-assistant
2. read multiple container's logs (where the order is consistent), e.g., docker-compose logs home-assistant esphome

I am suggesting that when merge view is opened, Dozzle would switch to using "the second option".

However, I don't know exactly how the Docker API works and if this is possible to do for all containers. I noticed that the non-compose docker logs command doesn't actually support streaming from multiple containers. If that's the case for the Docker API as well, then at least it would be nice to have a merge view for a single docker-compose project (merging logs from services defined in the docker-compose using the second command).

I believe this would make the merge view significantly more useful, as the current version shows logs where timestamps have large variations. From the screenshots above, you can see shifts of +1 hour, -1 hour, +1 hour, and so on.

I can give it a go at implementing this during the weekend if I find the time.

[amir20]

I see, so Dozzle always reads logs for each container independently, regardless if the user is in the merge view?

That's right.

Would it not be possible to change the listening logic (interaction between Dozzle and Docker) to read multiple container's logs if the merge view is open?

Docker API is very limited. It only allows for one container. This is why the more logs, the more CPU usage because each log stream opens up a thread. It's also similar to stats API. I have looked through the code for Docker CLI and they open up multiple go threads.

read multiple container's logs (where the order is consistent), e.g., docker-compose logs home-assistant esphome

I am not sure what the order being consistent has to do here. I am pretty sure docker-compose would behave similar to Dozzle because the logs are just shown as they come in. There is no sorting or consistency.

I believe this would make the merge view significantly more useful, as the current version shows logs where timestamps have large variations. From the screenshots above, you can see shifts of +1 hour, -1 hour, +1 hour, and so on.

Hmm I missed that. It might be something else happening here. The timestamp comes from Docker. So there might be something wrong with the application that is logging.

I can give it a go at implementing this during the weekend if I find the time.

My time is limited these days. Any help would be appreciated. I am not sure what the first would be though. If you just get the project up and running and test some idea might be a good start.

If you think there can be an improvement, then let me know here in this issue and we can figure out if its right.

[amir20]

I am closing this because there isn't anything that can be done. One idea would be to redesign how grouped logs are modeled and send them all in one atomic payload. But that's a pretty big change.