Replies: 22 comments 22 replies
-
|
Followed the recommendation (in #206) and installed k3d beta, but facing more or the same issue on /var/run |
Beta Was this translation helpful? Give feedback.
-
|
I'm having the same error with different paths... Host: Ubuntu 20.04.2LTS (as a VM on Fusion) Describing the pods not starting, instead of /var/run I have the following paths It sounds like this is still outstanding. Thanks |
Beta Was this translation helpful? Give feedback.
-
|
@fabricev Your issue is the But interesting that your k3d cluster create demo --volume /private/tmp/longhorn:/var/lib/longhorn:sharedThat's the error message I'm getting: I'm using macOS Catalina and have no clue where the Edit:It seems to be related with that bug but I have the latest docker version where it should be fixed (according tho these release notes). |
Beta Was this translation helpful? Give feedback.
-
|
Same problem here on ubuntu 20.04 The following workarounds did not help:
|
Beta Was this translation helpful? Give feedback.
-
|
Hi @rdxmb , actually your workaround doesn't work because of a different issue. |
Beta Was this translation helpful? Give feedback.
-
|
@iwilltry42 Thanks. So it would be interesting, if mounting subpaths is enough for longhorn... |
Beta Was this translation helpful? Give feedback.
-
|
@mamiu I believe in your case the issue—which I'm also bumping up against right now and it's a real head-scratcher—is rooted in docker/for-mac#3431 (still open and not receiving enough attention IMO). A simple way to reproduce w/o docker run --rm -it -v /tmp:/mnt/tmp:shared alpine shYou should see output like: EDIT: Per Docker Docs "Bind propagation defaults to rprivate for both bind mounts and volumes. It is only configurable for bind mounts, and only on Linux host machines." The way I read that |
Beta Was this translation helpful? Give feedback.
-
|
@balibebas You're right, that seems to be the issue for me! Thanks for investigating it and letting me know. @fabricev Did my previous comment help you to solve your error? If so you can close this issue. |
Beta Was this translation helpful? Give feedback.
-
|
I'm having a similar problem on K3s trying to get Longhorn going |
Beta Was this translation helpful? Give feedback.
-
|
@doug62 if by K3s you mean the docker wrapper k3d let me save you some time and suggest to observe the iSCSI requirements of the device used and make sure you have dynamic provisioning enabled. My issue with Longhorn was because I was trying to run on macOS and didn't have iSCSI enabled (doable but a rather large hack). If you're doing stuff with K3s outside of docker your issue is not this issue. |
Beta Was this translation helpful? Give feedback.
-
|
Quick update: The /var/run sharedmounts seem to work with the following versions:
Using Longhorn 1.1.0 However the issue seems to be rather on the iscsi requirements on the hosts (ie: rancher/k3s:v1.20.0-k3s2 image) kubectl logs -n longhorn-system longhorn-manager-rd9xl
time="2021-01-28T12:29:38Z" level=error msg="Failed environment check, please make sure you have iscsiadm/open-iscsi installed on the host"
time="2021-01-28T12:29:38Z" level=fatal msg="Error starting manager: Environment check failed: Invalid mount namespace /host/proc/1/ns/mnt, error Failed to execute: nsenter [--mount=/host/proc/1/ns/mnt mount], output , stderr, nsenter: failed to execute mount: No such file or directory\n, error exit status 1" |
Beta Was this translation helpful? Give feedback.
-
|
I'm using RANCHER 2.5.5. gives me other problem.. the cluster do not work properly |
Beta Was this translation helpful? Give feedback.
-
This is not the same issue AFAICT as it's unrelated to anything K3D specific. I don't see any need open another issue either as it seems @picoroma you're just getting a bit ahead of the development roadmap for Rancher + Longhorn. And FYI. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, @iwilltry42 : any comments ? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @clescot , if it was due to a missing package, then the folks over at https://github.com/rancher/k3s would have to add it to the image... |
Beta Was this translation helpful? Give feedback.
-
|
Looks like the k3s Docker image is distroless (from scratch), so it won't be as simple as adding a missing package... |
Beta Was this translation helpful? Give feedback.
-
|
Yes, the packages do not come in at Dockerfile level but are pre-packaged and just deflated into the distroless filesystem. |
Beta Was this translation helpful? Give feedback.
-
|
As an aside, I wonder if it would be possible to emulate an iSCSI drive using MacFUSE and/or a case-sensitive APFS volume native to macOS. I've found case-sensitivity is important for parity with Linux file systems though that may not be a hard requirement. |
Beta Was this translation helpful? Give feedback.
-
|
@balibebas : i'm on Linux (Ubuntu). Do you think some open-scsi package install with some shared mount will solve the issue ? |
Beta Was this translation helpful? Give feedback.
-
|
There seems to be some dependency from Longhorn with the openscsi package (which is not a trivial thing https://packages.debian.org/unstable/open-iscsi).
|
Beta Was this translation helpful? Give feedback.
-
|
Hi, @fabricev , and maybe this docker image with its instructions : hope this helps, |
Beta Was this translation helpful? Give feedback.
-
|
@clescot : thx for the material, I had a quick look at first blog... Looks like what i had in mind was option 3, so this docker blog will really be helpful !!!! |
Beta Was this translation helpful? Give feedback.
-
|
Any luck with this ? k3d cluster create longhorn-test --servers 1 --agents 3 --volume $HOME/longhorn:/host/dev/ --volume $HOME/longhorn:/host/proc/ --volume $HOME/longhorn:/var/lib/longhorn-setting/ --volume $HOME/longhorn:/var/lib/longhorn:shared --verbosei tried it by adding missing mounts but getting : time="2021-03-18T04:32:16Z" level=error msg="Failed environment check, please make sure you have iscsiadm/open-iscsi installed on the host"
time="2021-03-18T04:32:16Z" level=fatal msg="Error starting manager: Environment check failed: Invalid mount namespace /host/proc/1/ns/mnt, error Failed to execute: nsenter [--mount=/host/proc/1/ns/mnt mount], output , stderr, nsenter: failed to execute mount: No such file or directory\n, error exit status 1"
|
Beta Was this translation helpful? Give feedback.
-
|
Just btw. instead of sharing volume while creating cluster this command on each server works for sharing the mount for me *no more error about / not shared docker ps -q --filter ancestor=rancher/k3s:v1.24.4-k3s1 | xargs -I {} docker exec {} sh -c 'mount --make-rshared /' |
Beta Was this translation helpful? Give feedback.
-
|
@SyedAsimAliSE - Noticed you are using the Docker K3s, I'm afraid you can't run longhorn in a container, longhorn requires 'apt-get open-iscsi' and the install features of the containers have been stripped out. I have since run K3d from virtual ubuntus and it is a much better expirence. |
Beta Was this translation helpful? Give feedback.
-
|
@doug62 yes i am using Docker K3s, but isn't it comes with K3d itself ? i mean K3d spins up containers in Docker, can you please elaborate how is your setup ? in vms ? without K3d ? |
Beta Was this translation helpful? Give feedback.
-
|
K3d is K3s built into an image that is then run inside docker, the K3d image is bare bones and you can't install 'open-iscsi' which is a requirement for Longhorn so the only way to enjoy Longhorn is to install it on Linux. No pod that requires Longhorn as a storage class can be scheduled on a node that can't run longhorn. Although you can join a K3d agent/node to a K3s master that node/agent wont work with Longhorn - the best way to go is pure Linux. I use Ubuntu 20, I run some inside of hyper v on my various multi-purpose windows PC's in the house while I installed Ubuntu on retired PCs and dedicate them. I force Longhorn to keep at least one replica on a stable machine - it is super cool. |
Beta Was this translation helpful? Give feedback.
-
|
https://rancher.com/docs/k3s/latest/en/installation/install-options/ And then: Server: Agents: I use Ubuntu 20 on vms and hardware |
Beta Was this translation helpful? Give feedback.
-
|
will try to do this in Vagrant vms |
Beta Was this translation helpful? Give feedback.
-
|
@doug62 have you seen my post above? |
Beta Was this translation helpful? Give feedback.
-
|
I've compiled a static iscsiadm and placed into k3d using mounts: and the executable works fine inside the docker instance of k3s, the problem looks different, longhorn-manager wants to access to host namespace, but it fails, there a way to make it works? called by |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
We started to use k3d / k3s for acceptance tests and were happy with the ultra fast deployment on bare-metal, the rancher integration and the chart deployments. |
Beta Was this translation helpful? Give feedback.
-
|
@sherpya , maybe you can make this an issue or discussion in the Longhorn repo then if you think that's suitable? |
Beta Was this translation helpful? Give feedback.
-
@sherpya thanks for the feedback, could you please provide the steps you've used to compile the correct static binaries? |
Beta Was this translation helpful? Give feedback.
-
|
sorry for the delay, I've used an Alpine docker image with some trick, but as I see you already built it |
Beta Was this translation helpful? Give feedback.
-
|
For those interested, I've built an image which contains the iscsiadm binary (here: https://github.com/fastlorenzo/k3s/pkgs/container/k3s) The Dockerfile used to build it is pretty straightforward: https://github.com/fastlorenzo/k3s/blob/iscsi/iscsi_utils/Dockerfile.template For now I've built and published the following versions:
If you need another version, just ping me and I'll publish it. |
Beta Was this translation helpful? Give feedback.
-
|
@fastlorenzo thanks for the building the image, but I still have issues trying to deploy longhorn with your image: $ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2ecf429b57c6 rancher/k3d-proxy:5.2.2 "/bin/sh -c nginx-pr…" 2 minutes ago Up 2 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp, 0.0.0.0:45173->6443/tcp k3d-k3s-default-serverlb
3ff3ff7d2614 ghcr.io/fastlorenzo/k3s:v1.21.7-k3s1 "/bin/k3s server --t…" 2 minutes ago Up 2 minutes k3d-k3s-default-server-0
$ kubectl get pods -n longhorn-system
NAME READY STATUS RESTARTS AGE
longhorn-ui-9fdb94f9-nlmfl 1/1 Running 0 2m23s
longhorn-driver-deployer-784546d78d-7wn8s 0/1 Init:0/1 0 2m23s
longhorn-manager-flpn9 0/1 CrashLoopBackOff 4 2m23s
$ kubectl logs -n longhorn-system longhorn-manager-flpn9
2022/01/13 14:11:35 proto: duplicate proto type registered: VersionResponse
time="2022-01-13T14:11:35Z" level=error msg="Failed environment check, please make sure you have iscsiadm/open-iscsi installed on the host"
time="2022-01-13T14:11:35Z" level=fatal msg="Error starting manager: Environment check failed: Invalid mount namespace /host/proc/1/ns/mnt, error Failed to execute: nsenter [--mount=/host/proc/1/ns/mnt mount], output , stderr, nsenter: failed to execute mount: No such file or directory\n, error exit status 127" |
Beta Was this translation helpful? Give feedback.
-
|
@ceccone which mounts did you pass when creating the k3d cluster? I've also noticed that some mounts on the helm chart for longhorn are hardcoded to If I recall, the issue was the
|
Beta Was this translation helpful? Give feedback.
-
|
any ideas on how to fix this ? |
Beta Was this translation helpful? Give feedback.
-
|
so longhorn currently is not an option for k3d RWX storage, correct? |
Beta Was this translation helpful? Give feedback.
-
|
Could you put the limitation for k3d elsewhere to the documentation, that Longhorn is currently just unsupported? |
Beta Was this translation helpful? Give feedback.
-
|
It's not like k3d could do anything about it, as it's basically about the container image, libraries, etc. which are not handled in any way by k3d. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, is there any suggested way to build a k3d image with open-iscsi daemon running behind the scene? I have tried:
|
Beta Was this translation helpful? Give feedback.
-
What did you do
N/A
What did you expect to happen
Longhorn managers pods from the daemonset created and running
Screenshots or terminal output
Which OS & Architecture
Ubuntu 20.04.1 on VMWare Workstation Player 15 / Windows 10
Which version of
k3dk3d version v3.0.2-5-g60069f6
k3s version v1.18.9-k3s1 (default)
Which version of docker
docker version
docker info
Beta Was this translation helpful? Give feedback.
All reactions