docker stats reported as null for ARM

[izestrea]

Got an error on my raspberry pi. There is not stats, and container log's contain

ERROR:    Exception in ASGI application". 
Error Logs:
RuntimeError: aclose(): asynchronous generator is already running
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/uvicorn/protocols/websockets/websockets_impl.py", line 154, in run_asgi
    result = await self.app(self.scope, self.asgi_receive, self.asgi_send)
  File "/usr/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/usr/lib/python3.8/site-packages/fastapi/applications.py", line 180, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/lib/python3.8/site-packages/starlette/middleware/errors.py", line 146, in __call__
    await self.app(scope, receive, send)
  File "/usr/lib/python3.8/site-packages/starlette/exceptions.py", line 58, in __call__
    await self.app(scope, receive, send)
  File "/usr/lib/python3.8/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "/usr/lib/python3.8/site-packages/starlette/routing.py", line 283, in handle
    await self.app(scope, receive, send)
  File "/usr/lib/python3.8/site-packages/starlette/routing.py", line 57, in app
    await func(session)
  File "/usr/lib/python3.8/site-packages/fastapi/routing.py", line 242, in app
    await dependant.call(**values)
  File "./api/routers/apps.py", line 145, in dashboard
    await asyncio.gather(*tasks)
  File "./api/routers/apps.py", line 154, in process_container
    mem_current = line["memory_stats"]["usage"]
KeyError: 'usage'

[SelfhostedPro]

Can you confirm you're on the latest version? I've see this happen if a container is stuck in the restarting state. Can you see if that is an issue?

[izestrea]

The container restarts without problems, and it was build on image selfhostedpro/yacht:latest

[SelfhostedPro]

Are there any other containers on your system that are stuck in a restarting state? I've made a few changes to devel that may resolve this. It's currently building in the actions tab of this repo and when it's done you should try it: selfhostedpro/yacht:devel

If that doesn't work for you, hop into our discord and I'll see if I can troubleshoot it a bit better: https://discord.gg/KpKutvC

[izestrea]

All containers are either running or stopped. I'll try current building - selfhostedpro/yacht:devel, and then I'll come back with details.

[SelfhostedPro]

If it continues happening can you also check if your browser console says anything? (f12)

[izestrea]

Unfortunately, trying current building, did not change anything.
Browser console log:
Home.vue:122 {ob: Ot}ob: Ot {value: {…}, dep: mt, vmCount: 0}dep: mt {id: 11405, subs: Array(0)}value: {ob: Ot}vmCount: 0__proto__: Object__proto__: Object
Home.vue:151 WebSocket is already in CLOSING or CLOSED state.
closeStats @ Home.vue:151
refresh @ Home.vue:123
click @ Home.vue?33e3:1
ne @ vue.runtime.esm.js:1854
n @ vue.runtime.esm.js:2179
Xr.a._wrapper @ vue.runtime.esm.js:6917
Home.vue:81 Starting connection to Websocket

https://imgur.com/K5Teq9e.png

[SelfhostedPro]

Hmmm. Interesting. Only seeing this issue on the rpi. I'll dig into it a bit more. The error on the backend wont stop the data from sending but it seems like it's closing the websocket for some reason. Refreshing the page doesn't help?

I'll try to setup a pi later today for testing.

[SelfhostedPro]

Is this just the main dashboard or is this in logs and stats too?

[nidua42]

Is this just the main dashboard or is this in logs and stats too?

I’m experiencing the same issue. Logs seem to work fine as far as I can tell. The problem is with the main dashboard and stats for containers.

[SelfhostedPro]

@nidua42 are you on arm as well? Can you tell me what the output of docker stats is? It may be an underlying issue with how docker produces stats on arm.

[nidua42]

@nidua42 are you on arm as well? Can you tell me what the output of docker stats is? It may be an underlying issue with how docker produces stats on arm.

Yes, I am ARM too. I would gladly provide the output but how do I get it? Not sure how to do it.

[SelfhostedPro]

Open up the command line and run docker stats <containerid> you can use docker ps to get the IDs for your containers.

[nidua42]

Hope I did this correctly

[SelfhostedPro]

Looks like docker stats is broken on ARM which is causing this issue.

[SelfhostedPro]

Once this action is finished the devel tag will be pushed to dockerhub for testing. I added a check to see if it's getting memory stats.
https://github.com/SelfhostedPro/Yacht/actions/runs/273216923

[SelfhostedPro]

@nidua42 Can you go to the issue where this was mentioned (docker/for-linux#1112) and provide the board you're using as well as the version of docker and your OS?

[nidua42]

The funny thing is that 2 days ago Portainer was reporting memory usage. Now when I checked it only shows CPU and network. Memory graph is empty. Anyway here is the info

Raspberry pi 4b 4GB version 1.2

Linux raspberrypi 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l GNU/Linux

Docker version
Client:
Version: 18.09.1
API version: 1.39
Go version: go1.11.6
Git commit: 4c52b90
Built: Fri, 13 Sep 2019 10:45:43 +0100
OS/Arch: linux/arm
Experimental: false

Server:
Engine:
Version: 18.09.1
API version: 1.39 (minimum version 1.12)
Go version: go1.11.6
Git commit: 4c52b90
Built: Fri Sep 13 09:45:43 2019
OS/Arch: linux/arm
Experimental: false

[Griefed]

Seems to be Docker related. Can confirm that neither of my Pis is reporting CPU & MEM usage when using docker stats
Raspberry Pi 3B+: https://pastebin.com/CnkzVtKx
Raspberry Pi 3B: https://pastebin.com/nm24pWkC
x86-64 Tablet running Linux: https://pastebin.com/bjFKNK4e

I know that docker stats used to report CPU & MEM usage before, so it must be due to an update in Docker itself.

Also mentioned them in docker/for-linux#1112 as per your comment up above.

[SelfhostedPro]

Documentation added to the bottom of the readme for this. Thanks for reporting it guys!