Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Invalid checksum for backup superblock #3661

Open
mikael-andersson opened this issue Nov 5, 2024 · 1 comment
Open

Invalid checksum for backup superblock #3661

mikael-andersson opened this issue Nov 5, 2024 · 1 comment
Labels
bug

Comments

@mikael-andersson
Copy link

Describe the issue you are experiencing

We have around 65 HA and have verified that at least 12 of those are or have been showing what we suspect is a problem. When executing journalctl -b 0 -k -g 'Invalid checksum' in the host OS, we get:

Feb 07 11:12:29 homeassistant kernel: EXT4-fs (mmcblk1p4): Invalid checksum for backup superblock 32768
Feb 07 11:12:29 homeassistant kernel: EXT4-fs (mmcblk1p4): Invalid checksum for backup superblock 98304
Feb 07 11:12:29 homeassistant kernel: EXT4-fs (mmcblk1p4): Invalid checksum for backup superblock 163840
Feb 07 11:12:29 homeassistant kernel: EXT4-fs (mmcblk1p4): Invalid checksum for backup superblock 229376
Feb 07 11:12:29 homeassistant kernel: EXT4-fs (mmcblk1p4): Invalid checksum for backup superblock 294912
Feb 07 11:12:29 homeassistant kernel: EXT4-fs (mmcblk1p4): Invalid checksum for backup superblock 819200
Feb 07 11:12:29 homeassistant kernel: EXT4-fs (mmcblk1p4): Invalid checksum for backup superblock 884736
Feb 07 11:12:29 homeassistant kernel: EXT4-fs (mmcblk1p4): Invalid checksum for backup superblock 1605632

I should say that it is only a small number, around 5, which may have had an impact on how HA works. I say "may", because I do of course not know if above is what causes the problems. For 2 or 3, the systems have not been able to boot. For others, integrations have not been able to load. But most of them are working fine, at least what it seems so far. Currently we have 7 systems which show "Invalid checksum..." in the host logs. Hence I can provide you with more information if you like.

What operating system image do you use?

odroid-n2 (Hardkernel ODROID-N2/N2+)

What version of Home Assistant Operating System is installed?

6.6.54-haos

Did the problem occur after upgrading the Operating System?

No

Hardware details

Odroid N2+

Some systems are equipped with this type of USB adapter: https://www.amazon.co.uk/dp/B07K3V381Z
I am not entirely sure if it is the same manufacturer though.

Steps to reproduce the issue

Unfortunately I do not know how the fault appeared and thus cannot tell how to reproduce it.

Anything in the Supervisor logs that might be useful for us?

Do not see anything strange here, but if you still would like to see them, I can provide them.

Anything in the Host logs that might be useful for us?

2024-11-05 05:00:00.314 homeassistant kernel: veth02d3951: entered allmulticast mode
2024-11-05 05:00:00.314 homeassistant kernel: veth02d3951: entered promiscuous mode
2024-11-05 05:00:00.314 homeassistant kernel: audit: type=1700 audit(1730782800.306:546): dev=veth02d3951 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
2024-11-05 05:00:00.314 homeassistant kernel: audit: type=1300 audit(1730782800.306:546): arch=c00000b7 syscall=206 success=yes exit=40 a0=d a1=40004ecdb0 a2=28 a3=0 items=0 ppid=1 pid=498 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="dockerd" exe="/usr/bin/dockerd" subj=unconfined key=(null)
2024-11-05 05:00:00.314 homeassistant kernel: audit: type=1327 audit(1730782800.306:546): proctitle=2F7573722F62696E2F646F636B657264002D480066643A2F2F002D2D636F6E7461696E6572643D2F72756E2F636F6E7461696E6572642F636F6E7461696E6572642E736F636B
2024-11-05 05:00:00.316 homeassistant NetworkManager[418]: <info>  [1730782800.3149] manager: (veth02d3951): new Veth device (/org/freedesktop/NetworkManager/Devices/47)
2024-11-05 05:00:00.390 homeassistant kernel: audit: type=1325 audit(1730782800.382:547): table=nat:144 family=2 entries=1 op=nft_register_rule pid=110546 subj=unconfined comm="iptables"
2024-11-05 05:00:00.390 homeassistant kernel: audit: type=1300 audit(1730782800.382:547): arch=c00000b7 syscall=211 success=yes exit=508 a0=3 a1=ffffeb975298 a2=0 a3=1 items=0 ppid=498 pid=110546 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-nft-multi" subj=unconfined key=(null)
2024-11-05 05:00:00.390 homeassistant kernel: audit: type=1327 audit(1730782800.382:547): proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D74006E6174002D4100444F434B4552002D7000746370002D6400302F30002D2D64706F7274003830002D6A00444E4154002D2D746F2D64657374696E6174696F6E003137322E33302E33332E333A38300000002D690068617373696F
2024-11-05 05:00:00.398 homeassistant kernel: audit: type=1325 audit(1730782800.390:548): table=nat:145 family=2 entries=1 op=nft_register_rule pid=110548 subj=unconfined comm="iptables"
2024-11-05 05:00:00.398 homeassistant kernel: audit: type=1300 audit(1730782800.390:548): arch=c00000b7 syscall=211 success=yes exit=600 a0=3 a1=ffffc0554988 a2=0 a3=1 items=0 ppid=498 pid=110548 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-nft-multi" subj=unconfined key=(null)
2024-11-05 05:00:00.398 homeassistant kernel: audit: type=1327 audit(1730782800.390:548): proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D74006E6174002D4100504F5354524F5554494E47002D7000746370002D73003137322E33302E33332E33002D64003137322E33302E33332E33002D2D64706F7274003830002D6A004D415351554552414445
2024-11-05 05:00:00.410 homeassistant kernel: audit: type=1325 audit(1730782800.402:549): table=filter:146 family=2 entries=1 op=nft_register_rule pid=110550 subj=unconfined comm="iptables"
2024-11-05 05:00:00.426 homeassistant dockerd[498]: time="2024-11-05T05:00:00.426259332Z" level=warning msg="Failed to delete conntrack state for 172.30.33.3: invalid argument"
2024-11-05 05:00:00.530 homeassistant systemd[1]: Started libcontainer container f40da728bb95825a0266c36ec2600e62b8a04a73224efd30b35ae464080c8ac2.
2024-11-05 05:00:00.694 homeassistant kernel: eth0: renamed from veth2bec7fe
2024-11-05 05:00:00.713 homeassistant NetworkManager[418]: <info>  [1730782800.7138] device (veth02d3951): carrier: link connected
2024-11-05 05:00:00.714 homeassistant kernel: hassio: port 9(veth02d3951) entered blocking state
2024-11-05 05:00:00.714 homeassistant kernel: hassio: port 9(veth02d3951) entered forwarding state
2024-11-05 05:00:13.072 homeassistant systemd[1]: docker-f40da728bb95825a0266c36ec2600e62b8a04a73224efd30b35ae464080c8ac2.scope: Deactivated successfully.
2024-11-05 05:00:13.074 homeassistant kernel: kauditd_printk_skb: 42 callbacks suppressed
2024-11-05 05:00:13.074 homeassistant kernel: audit: type=1334 audit(1730782813.067:564): prog-id=144 op=UNLOAD
2024-11-05 05:00:13.074 homeassistant systemd[1]: docker-f40da728bb95825a0266c36ec2600e62b8a04a73224efd30b35ae464080c8ac2.scope: Consumed 8.225s CPU time.
2024-11-05 05:00:13.099 homeassistant dockerd[498]: time="2024-11-05T05:00:13.098909708Z" level=info msg="ignoring event" container=f40da728bb95825a0266c36ec2600e62b8a04a73224efd30b35ae464080c8ac2 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
2024-11-05 05:00:13.142 homeassistant kernel: audit: type=1325 audit(1730782813.135:565): table=nat:147 family=2 entries=1 op=nft_unregister_rule pid=111205 subj=unconfined comm="iptables"
2024-11-05 05:00:13.142 homeassistant kernel: audit: type=1300 audit(1730782813.135:565): arch=c00000b7 syscall=211 success=yes exit=512 a0=3 a1=ffffdf7f9c68 a2=0 a3=1 items=0 ppid=498 pid=111205 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-nft-multi" subj=unconfined key=(null)
2024-11-05 05:00:13.142 homeassistant kernel: audit: type=1327 audit(1730782813.135:565): proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D74006E6174002D4400444F434B4552002D7000746370002D6400302F30002D2D64706F7274003830002D6A00444E4154002D2D746F2D64657374696E6174696F6E003137322E33302E33332E333A38300000002D690068617373696F
2024-11-05 05:00:13.166 homeassistant kernel: audit: type=1325 audit(1730782813.159:566): table=nat:148 family=2 entries=1 op=nft_unregister_rule pid=111208 subj=unconfined comm="iptables"
2024-11-05 05:00:13.166 homeassistant kernel: audit: type=1300 audit(1730782813.159:566): arch=c00000b7 syscall=211 success=yes exit=604 a0=3 a1=ffffc652ab58 a2=0 a3=1 items=0 ppid=498 pid=111208 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-nft-multi" subj=unconfined key=(null)
2024-11-05 05:00:13.166 homeassistant kernel: audit: type=1327 audit(1730782813.159:566): proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D74006E6174002D4400504F5354524F5554494E47002D7000746370002D73003137322E33302E33332E33002D64003137322E33302E33332E33002D2D64706F7274003830002D6A004D415351554552414445
2024-11-05 05:00:13.198 homeassistant kernel: audit: type=1325 audit(1730782813.191:567): table=filter:149 family=2 entries=1 op=nft_unregister_rule pid=111210 subj=unconfined comm="iptables"
2024-11-05 05:00:13.198 homeassistant kernel: audit: type=1300 audit(1730782813.191:567): arch=c00000b7 syscall=211 success=yes exit=652 a0=3 a1=ffffcc3755a8 a2=0 a3=1 items=0 ppid=498 pid=111210 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-nft-multi" subj=unconfined key=(null)
2024-11-05 05:00:13.198 homeassistant kernel: audit: type=1327 audit(1730782813.191:567): proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D740066696C746572002D4400444F434B45520000002D690068617373696F002D6F0068617373696F002D7000746370002D64003137322E33302E33332E33002D2D64706F7274003830002D6A00414343455054
2024-11-05 05:00:13.222 homeassistant dockerd[498]: time="2024-11-05T05:00:13.222439005Z" level=warning msg="Failed to delete conntrack state for 172.30.33.3: invalid argument"
2024-11-05 05:00:13.226 homeassistant kernel: hassio: port 9(veth02d3951) entered disabled state
2024-11-05 05:00:13.226 homeassistant kernel: veth2bec7fe: renamed from eth0
2024-11-05 05:00:13.296 homeassistant NetworkManager[418]: <info>  [1730782813.2960] manager: (veth2bec7fe): new Veth device (/org/freedesktop/NetworkManager/Devices/48)
2024-11-05 05:00:13.302 homeassistant kernel: hassio: port 9(veth02d3951) entered disabled state
2024-11-05 05:00:13.302 homeassistant kernel: veth02d3951 (unregistering): left allmulticast mode
2024-11-05 05:00:13.302 homeassistant kernel: veth02d3951 (unregistering): left promiscuous mode
2024-11-05 05:00:13.302 homeassistant kernel: hassio: port 9(veth02d3951) entered disabled state
2024-11-05 05:00:13.389 homeassistant systemd[1]: var-lib-docker-overlay2-3dddf3b87d82e444d7163a578949190b8bf2ba9e3ae21cd036312de734a7bf6e-merged.mount: Deactivated successfully.
2024-11-05 05:00:13.389 homeassistant systemd[1]: mnt-data-docker-overlay2-3dddf3b87d82e444d7163a578949190b8bf2ba9e3ae21cd036312de734a7bf6e-merged.mount: Deactivated successfully.
2024-11-05 05:48:50.874 homeassistant NetworkManager[418]: <info>  [1730785730.8744] dhcp4 (end0): state changed new lease, address=192.168.1.64
2024-11-05 05:48:50.923 homeassistant systemd[1]: Starting Network Manager Script Dispatcher Service...
2024-11-05 05:48:50.942 homeassistant systemd[1]: Started Network Manager Script Dispatcher Service.
2024-11-05 05:49:00.961 homeassistant systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
2024-11-05 06:05:51.642 homeassistant kernel: kauditd_printk_skb: 4 callbacks suppressed
2024-11-05 06:05:51.642 homeassistant kernel: audit: type=1334 audit(1730786751.638:570): prog-id=148 op=LOAD
2024-11-05 06:05:51.642 homeassistant kernel: audit: type=1334 audit(1730786751.638:571): prog-id=149 op=LOAD
2024-11-05 06:05:51.642 homeassistant kernel: audit: type=1334 audit(1730786751.638:572): prog-id=150 op=LOAD
2024-11-05 06:05:51.671 homeassistant systemd[1]: Starting Hostname Service...
2024-11-05 06:05:51.918 homeassistant systemd[1]: Started Hostname Service.
2024-11-05 06:05:51.930 homeassistant kernel: audit: type=1334 audit(1730786751.926:573): prog-id=151 op=LOAD
2024-11-05 06:05:51.930 homeassistant kernel: audit: type=1334 audit(1730786751.926:574): prog-id=152 op=LOAD
2024-11-05 06:05:51.930 homeassistant kernel: audit: type=1334 audit(1730786751.926:575): prog-id=153 op=LOAD
2024-11-05 06:05:51.967 homeassistant systemd[1]: Starting Time & Date Service...
2024-11-05 06:05:52.159 homeassistant systemd[1]: Started Time & Date Service.
2024-11-05 06:06:21.953 homeassistant systemd[1]: systemd-hostnamed.service: Deactivated successfully.
2024-11-05 06:06:22.054 homeassistant kernel: audit: type=1334 audit(1730786782.047:576): prog-id=150 op=UNLOAD
2024-11-05 06:06:22.054 homeassistant kernel: audit: type=1334 audit(1730786782.047:577): prog-id=149 op=UNLOAD
2024-11-05 06:06:22.054 homeassistant kernel: audit: type=1334 audit(1730786782.047:578): prog-id=148 op=UNLOAD
2024-11-05 06:06:22.182 homeassistant systemd[1]: systemd-timedated.service: Deactivated successfully.
2024-11-05 06:06:22.210 homeassistant kernel: audit: type=1334 audit(1730786782.203:579): prog-id=153 op=UNLOAD
2024-11-05 06:06:22.210 homeassistant kernel: audit: type=1334 audit(1730786782.203:580): prog-id=152 op=UNLOAD
2024-11-05 06:06:22.210 homeassistant kernel: audit: type=1334 audit(1730786782.203:581): prog-id=151 op=UNLOAD
2024-11-05 06:41:20.875 homeassistant NetworkManager[418]: <info>  [1730788880.8750] dhcp4 (end0): state changed new lease, address=192.168.1.64
2024-11-05 06:41:20.923 homeassistant systemd[1]: Starting Network Manager Script Dispatcher Service...
2024-11-05 06:41:20.941 homeassistant systemd[1]: Started Network Manager Script Dispatcher Service.
2024-11-05 06:41:30.960 homeassistant systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
2024-11-05 07:33:50.875 homeassistant NetworkManager[418]: <info>  [1730792030.8752] dhcp4 (end0): state changed new lease, address=192.168.1.64
2024-11-05 07:33:50.907 homeassistant systemd[1]: Starting Network Manager Script Dispatcher Service...
2024-11-05 07:33:50.926 homeassistant systemd[1]: Started Network Manager Script Dispatcher Service.
2024-11-05 07:34:00.945 homeassistant systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
2024-11-05 08:12:32.358 homeassistant kernel: audit: type=1334 audit(1730794352.353:582): prog-id=154 op=LOAD
2024-11-05 08:12:32.358 homeassistant kernel: audit: type=1334 audit(1730794352.353:583): prog-id=155 op=LOAD
2024-11-05 08:12:32.358 homeassistant kernel: audit: type=1334 audit(1730794352.353:584): prog-id=156 op=LOAD
2024-11-05 08:12:32.379 homeassistant systemd[1]: Starting Hostname Service...
2024-11-05 08:12:32.597 homeassistant systemd[1]: Started Hostname Service.
2024-11-05 08:12:32.618 homeassistant kernel: audit: type=1334 audit(1730794352.613:585): prog-id=157 op=LOAD
2024-11-05 08:12:32.618 homeassistant kernel: audit: type=1334 audit(1730794352.613:586): prog-id=158 op=LOAD
2024-11-05 08:12:32.618 homeassistant kernel: audit: type=1334 audit(1730794352.613:587): prog-id=159 op=LOAD
2024-11-05 08:12:32.639 homeassistant systemd[1]: Starting Time & Date Service...
2024-11-05 08:12:32.850 homeassistant systemd[1]: Started Time & Date Service.
2024-11-05 08:13:02.632 homeassistant systemd[1]: systemd-hostnamed.service: Deactivated successfully.
2024-11-05 08:13:02.770 homeassistant kernel: audit: type=1334 audit(1730794382.763:588): prog-id=156 op=UNLOAD
2024-11-05 08:13:02.770 homeassistant kernel: audit: type=1334 audit(1730794382.763:589): prog-id=155 op=UNLOAD
2024-11-05 08:13:02.770 homeassistant kernel: audit: type=1334 audit(1730794382.763:590): prog-id=154 op=UNLOAD
2024-11-05 08:13:02.891 homeassistant systemd[1]: systemd-timedated.service: Deactivated successfully.
2024-11-05 08:13:02.914 homeassistant kernel: audit: type=1334 audit(1730794382.907:591): prog-id=159 op=UNLOAD
2024-11-05 08:13:02.914 homeassistant kernel: audit: type=1334 audit(1730794382.907:592): prog-id=158 op=UNLOAD
2024-11-05 08:13:02.914 homeassistant kernel: audit: type=1334 audit(1730794382.907:593): prog-id=157 op=UNLOAD
2024-11-05 08:26:20.875 homeassistant NetworkManager[418]: <info>  [1730795180.8751] dhcp4 (end0): state changed new lease, address=192.168.1.64
2024-11-05 08:26:20.927 homeassistant systemd[1]: Starting Network Manager Script Dispatcher Service...
2024-11-05 08:26:20.937 homeassistant systemd[1]: Started Network Manager Script Dispatcher Service.
2024-11-05 08:26:30.955 homeassistant systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
2024-11-05 09:02:26.883 homeassistant kernel: audit: type=1334 audit(1730797346.874:594): prog-id=160 op=LOAD
2024-11-05 09:02:26.883 homeassistant systemd-timesyncd[487]: Network configuration changed, trying to establish connection.
2024-11-05 09:02:26.907 homeassistant systemd-timesyncd[487]: Contacted time server 162.159.200.1:123 (time.cloudflare.com).
2024-11-05 09:02:26.907 homeassistant systemd[1]: Started Journal Gateway Service.

System information

System Information

version core-2024.10.4
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.12.4
os_name Linux
os_version 6.6.54-haos
arch aarch64
timezone Europe/Stockholm
config_dir /config
Home Assistant Supervisor
host_os Home Assistant OS 13.2
update_channel stable
supervisor_version supervisor-2024.10.3
agent_version 1.6.0
docker_version 27.2.0
disk_total 113.9 GB
disk_used 35.5 GB
healthy true
supported true
host_connectivity true
supervisor_connectivity true
ntp_synchronized true
virtualization
board odroid-n2
supervisor_api ok
version_api ok
installed_addons Advanced SSH & Web Terminal (19.0.0), MariaDB (2.7.1), Mosquitto broker (6.4.1), Gandi DynDNS (0.1.2), Let's Encrypt (5.2.3), phpMyAdmin (0.10.0), Studio Code Server (5.17.2)
Dashboards
dashboards 2
resources 5
views 8
mode storage
Recorder
oldest_recorder_run October 15, 2024 at 02:25
current_recorder_run November 5, 2024 at 00:09
estimated_db_size 13574.33 MiB
database_engine mysql
database_version 10.11.6

Additional information

Here is a post by me from January this year:

https://community.home-assistant.io/t/invalid-checksum-for-backup-superblock/614463/4

The last time we had problems was a week ago. The system could not execute HA and showed this:

output

I had never run TestDisk before but I ran it on above system and it showed:

image

I suspect that this is related to the Odroid N2+ and/or the eMMC or the Linux version for Odroid N2, otherwise I would have expected many more bug reports from people. I have googled and not really found much about this error.
@FSund
Copy link

FSund commented Dec 19, 2024

I'm getting the same messages when rebooting HA. I'm running a very different setup though - HAOS runs in a VM on a Proxmox server.

I don't see any other issues, other than rebooting taking a bit long. I back up my config regularly (local + cloud copies), so I'm not that worried. But would like to know if this is something to worry about.

Dec 19 17:25:46 homeassistant kernel: EXT4-fs (sda8): Invalid checksum for backup superblock 32768
Dec 19 17:25:46 homeassistant kernel: EXT4-fs (sda8): Invalid checksum for backup superblock 98304
Dec 19 17:25:46 homeassistant kernel: EXT4-fs (sda8): Invalid checksum for backup superblock 32768
Dec 19 17:25:46 homeassistant kernel: EXT4-fs (sda8): Invalid checksum for backup superblock 32768
Dec 19 17:25:46 homeassistant kernel: EXT4-fs (sda8): Invalid checksum for backup superblock 32768
Dec 19 17:25:46 homeassistant kernel: EXT4-fs (sda8): Invalid checksum for backup superblock 98304
Dec 19 17:25:46 homeassistant kernel: EXT4-fs (sda8): Invalid checksum for backup superblock 163840
Dec 19 17:25:46 homeassistant kernel: EXT4-fs (sda8): Invalid checksum for backup superblock 229376
Dec 19 17:25:46 homeassistant kernel: EXT4-fs (sda8): Invalid checksum for backup superblock 294912
Dec 19 17:25:46 homeassistant kernel: EXT4-fs (sda8): Invalid checksum for backup superblock 819200
Dec 19 17:25:46 homeassistant kernel: EXT4-fs (sda8): Invalid checksum for backup superblock 884736
Dec 19 17:25:46 homeassistant kernel: EXT4-fs (sda8): Invalid checksum for backup superblock 1605632

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mikael-andersson @FSund