Replies: 1 comment
-
Resolved by:
|
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Resolved by:
|
Beta Was this translation helpful? Give feedback.
-
I have one control plane node and one worker node.
How am I supposed to safely reboot the worker node? Is it safe to run
kubectl drain <node-name> --ignore-daemonsets --delete-local-data
?I feared that the osd might not be restored if local data on the worker is erased.
Not knowing, I decided to just reboot the worker node - but it ended up being more of a force power off.
Currently, it looks like the osd is being detected by
rook-ceph-osd-prepare
:But
rook-ceph-osd
can't start. It looks likerook-ceph-osd
is trying to open the wrong device (sde
) (like https://github.com/rook/rook/pull/11567/files). The device on the host issda
. Is it possible to recover from this?Is
drain
harmless even if you have a single worker node? If yes, I could run it to see if it fixes the issue. Otherwise, I may have found a bug that happens after a "power failure" of a single node cluster.Here is the relevant output of the activate container:
kubectl -n rook-ceph logs rook-ceph-osd-0-<id> -c activate
Update
I decided to drain the node, reboot and uncordon. Seems like it didn't make things worse - but the outputs/errors remain the same.
I also tried
/var/lib/rook/rook-ceph/<id>/block
to/dev/sda
values.yaml
with a per-node filter to only usesda
- instead ofuseAllDevices
anduseAllNodes
.Issue may be related to #13564
Beta Was this translation helpful? Give feedback.
All reactions