Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Correct way to mount NFS share #8623

Open
BurningTheSky opened this issue Feb 23, 2020 · 23 comments
Open

Correct way to mount NFS share #8623

BurningTheSky opened this issue Feb 23, 2020 · 23 comments
Assignees
@YanzhaoLi

Comments

@BurningTheSky
Copy link

BurningTheSky commented Feb 23, 2020 •
edited
Loading

Question

I want to be able to mount a rw share into a container, I have tried some Docker specific methods but they don't appear to work. Is there a way to do it with VIC?

See also

[burningsky@engal ~]$ export NFS_VOL_NAME=nfs NFS_LOCAL_MNT=/mnt/nfs NFS_SERVER=192.168.0.14 NFS_SHARE=/data/nfs NFS_OPTS=vers=4,soft
docker --tls run --mount "src=$NFS_VOL_NAME,dst=$NFS_LOCAL_MNT,volume-opt=device=:$NFS_SHARE,"volume-opt=o=addr=$NFS_SERVER,$NFS_OPTS",type=volume,volume-driver=local,volume-opt=type=nfs" busybox ls $NFS_LOCAL_MNT
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
bdbbaa22dec6: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6
Status: Downloaded newer image for library/busybox:latest
ERRO[0193] error waiting for container: Error response from daemon: Server error from portlayer: ContainerWaitHandler(3c6d0cdf98e44764bc982ec25133c16aaf4ff30ea151165a4d430eb6796fd769) Error: context deadline exceeded
docker: Error response from daemon: Server error from portlayer: unable to wait for process launch status: Post https://vcenter.domain.local/sdk: context deadline exceeded.
[burningsky@engal ~]$ docker --tls volume create --driver local --opt type=nfs --opt o=addr=$NFS_SERVER,$NFS_OPTS --opt device=:$NFS_SHARE $NFS_VOL_NAME
Error response from daemon: bad driver value - device is not a supported option
@YanzhaoLi
Copy link
Member

Just as this doc(https://vmware.github.io/vic-product/assets/files/html/1.5/vic_vsphere_admin/backup_volumes.html) shows, you should first create a NFS-type volumeStore, and then use this volumeStore to create docker volume. And this is the only way to create NFS Shared Volumes in VIC

# create or configure vch with the volumestore parameter
--volume-store nfs://10.118.68.164/mnt/nfs-vol?uid=0&gid=0&proto=tcp:shared
# create nfs volume
docker volume create --opt VolumeStore=shared myshared
# use nfs volume
docker run -v myshared:/share xx

@BurningTheSky
Copy link
Author

Thanks, so I would need to redeploy the vch, there's no way to reconfigure a running one?

@YanzhaoLi
Copy link
Member

It's ok to reconfigure the VCH. See the "Add Volume Stores" section in this doc: https://vmware.github.io/vic-product/assets/files/html/1.5/vic_vsphere_admin/configure_vch.html

@BurningTheSky
Copy link
Author

I'm not sure what I've done wrong but the mount options appear to be wrong?

C:\Users\burningsky>vic-machine-windows configure --target vcenter.domain.local --user [email protected] --password password --thumbprint xxxxxxxxxxxxxx --id virtual-container-host --volume-store VM SAS 15K/DockerFolder:default --volume-store VM SAS 15K/DockerFolder:DockerVolume --volume-store nfs://192.168.0.14/Multimedia?uid=0&gid=0&proto=tcp:NFSMultimedia
time="2020-02-24T15:07:32Z" level=error msg=--------------------
time="2020-02-24T15:07:32Z" level=error msg="vic-machine-windows configure failed: volume store input must be in format <datastore url w/ path>:label or nfs:///?:\n"
'gid' is not recognized as an internal or external command,
operable program or batch file.
'proto' is not recognized as an internal or external command,
operable program or batch file.

@YanzhaoLi
Copy link
Member

It seems the windows shell could not parse special characters. Please try to use quote like --volume-store "nfs://192.168.0.14/Multimedia?uid=0&gid=0&proto=tcp:NFSMultimedia"

@BurningTheSky
Copy link
Author

I did give that a try before but no luck:

C:\Users\burningsky>vic-machine-windows configure --target vcenter.domain.local --user [email protected] --password password --thumbprint xxxxxxxxxxxxxx --id virtual-container-host --volume-store VM SAS 15K/DockerFolder:default --volume-store VM SAS 15K/DockerFolder:DockerVolume --volume-store "nfs://192.168.0.14/Multimedia?uid=0&gid=0&proto=tcp:NFSMultimedia"
time="2020-02-24T18:18:31Z" level=error msg=--------------------
time="2020-02-24T18:18:31Z" level=error msg="vic-machine-windows configure failed: volume store input must be in format <datastore url w/ path>:label or nfs:///?:\n"

@BurningTheSky
Copy link
Author

So I tried from linux but got a similar response:
[burningsky@engal ~]$ vic-machine-linux configure --target vcenter.domain.local --user [email protected] --password password --thumbprint xxxxxx --id virtual-container-host --volume-store VM SAS 15K/DockerFolder:default --volume-store VM SAS 15K/DockerFolder:DockerVolume --volume-store nfs://192.168.0.14/Multimedia?uid=0&gid=0&proto=tcp:NFSMultimedia
[1] 19971
[2] 19972
[burningsky@engal ~]$ ERRO[0000] --------------------
ERRO[0000] vic-machine-linux configure failed: volume store input must be in format <datastore url w/ path>:label or nfs:///?:

Then I tried to get the current config but it failed on the ID, but that is the name of the VCH, is there another ID other than its name and if so how do I find it?
[burningsky@engal ~]$ vic-machine-linux inspect config --target vcenter.domain.local --user [email protected] --password password --thumbprint xxxxxx --id virtual-container-host
INFO[0000] ### Inspecting VCH ####
INFO[0000] Validating target
ERRO[0000] Failed to get Virtual Container Host virtual-container-host
ERRO[0000] id "virtual-container-host" could not be found
ERRO[0000] --------------------
ERRO[0000] vic-machine-linux inspect config failed: inspect failed

@YanzhaoLi
Copy link
Member

  1. you can use --name VCH_NAME. And also to find the id, you can use vic-machine-linux ls
  2. In linux, please use single quote --volume-store 'nfs://192.168.0.14/Multimedia?uid=0&gid=0&proto=tcp:NFSMultimedia'

@BurningTheSky
Copy link
Author

Thanks, the single quotes worked but now I'm running into a timeout error, any idea what's causing that?

[burningsky@engal ~]$ vic-machine-linux configure --target vcenter.domain.local --user [email protected] --password password --thumbprint xxxxx --id vm-298 --volume-store 'VM SAS 15K/DockerFolder:default' --volume-store 'VM SAS 15K/DockerFolder:DockerVolume' --volume-store 'nfs://192.168.0.14/Multimedia?uid=0&gid=0&proto=tcp:NFSMultimedia'
INFO[0000] ### Configuring VCH ####
INFO[0000] Validating target
INFO[0000]
INFO[0000] VCH ID: VirtualMachine:vm-298
INFO[0000] Creating directory [VM SAS 15K] virtual-container-host
INFO[0000] Datastore path is [VM SAS 15K] virtual-container-host
INFO[0001] No certificate regeneration requested. No new certificates provided. Certificates left unchanged.
INFO[0001] Validating supplied configuration
INFO[0001] Configuring static IP for additional networks using port group "Management"
INFO[0001] Network configuration OK on "Containers"
INFO[0001] Network configuration OK on "Management"
INFO[0001] vCenter settings check OK
INFO[0001] Firewall status: ENABLED on "/domain/host/esxi1.domain.local/esxi1.domain.local"
INFO[0001] Firewall configuration OK on hosts:
INFO[0001] "/domain/host/esxi1.domain.local/esxi1.domain.local"
INFO[0001] vCenter settings check OK
INFO[0002] License check OK on hosts:
INFO[0002] "/domain/host/esxi1.domain.local/esxi1.domain.local"
INFO[0002] DRS check SKIPPED - target is standalone host
WARN[0003] Guest tools unavailable, resorting to power off - sessions will be left open
INFO[0005] Creating directory [VM SAS 15K] DockerFolder
INFO[0005] Datastore path is [VM SAS 15K] DockerFolder
INFO[0005] Creating directory [VM SAS 15K] DockerFolder
INFO[0005] Datastore path is [VM SAS 15K] DockerFolder
INFO[0005] Setting VM configuration
INFO[0007] Waiting for IP information
INFO[0007] Waiting for major appliance components to launch
INFO[0014] Obtained IP address for client interface: "192.168.0.202"
INFO[0014] Checking VCH connectivity with vSphere target
INFO[0015] vSphere API Test: https://vcenter.domain.local vSphere API target responds as expected
ERRO[0180] vic/lib/install/management.(*Dispatcher).CheckDockerAPI: CheckServiceReady during update error: context deadline exceeded
vic/lib/install/management.(*Dispatcher).update:348 CheckServiceReady during update
vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux
vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux
INFO[0180] Docker API endpoint check failed: context deadline exceeded
ERRO[0180] vic/lib/install/management.(*Dispatcher).update: CheckServiceReady during update error: context deadline exceeded
vic/lib/install/management.(*Dispatcher).update:348 CheckServiceReady during update
vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux
vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux
INFO[0180] API may be slow to start - please retry with increased timeout using --timeout
ERRO[0180] Failed to configure: Upgrading VCH exceeded time limit of 3m0s. Please increase the timeout using --timeout to accommodate for a busy vSphere target
INFO[0180] Rolling back configure
INFO[0180] Reverting to snapshot reconfigure for 21221
ERRO[0180] vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded
vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux
vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux
ERRO[0180] vic/pkg/vsphere/tasks.WaitForResultAndRetryIf: vic-machine-linux error: context deadline exceeded
vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux
vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux
ERRO[0180] vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded
vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux
vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux
ERRO[0180] Failed to revert appliance to snapshot: Failed to roll back upgrade: context deadline exceeded.
INFO[0180] Collecting ed2845f0-de50-46d3-b1fa-91f005e5b487 vpxd.log
ERRO[0180] vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded
vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux
vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux
ERRO[0180] Failed to collect ed2845f0-de50-46d3-b1fa-91f005e5b487 vpxd.log: Post https://vcenter.domain.local/sdk: context deadline exceeded
WARN[0180] No log data for ed2845f0-de50-46d3-b1fa-91f005e5b487 vpxd.log
ERRO[0180] vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded
vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux
vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux
ERRO[0180] vic/pkg/vsphere/tasks.WaitForResultAndRetryIf: vic-machine-linux error: context deadline exceeded
vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux
vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux
ERRO[0180] vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded
vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux
vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux
ERRO[0180] Failed to reset UpdateInProgress
ERRO[0180] context deadline exceeded
ERRO[0180] vic/pkg/trace.(*Operation).Err: vic-machine-linux error: context deadline exceeded
vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux
vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux
ERRO[0180] vic/cmd/vic-machine/configure.(*Configure).Run.func2: vic-machine-linux error: context deadline exceeded
vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux
vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux
ERRO[0180] vic/cmd/vic-machine/configure.(*Configure).Run.func2: vic-machine-linux error: context deadline exceeded
vic/cmd/vic-machine/configure.(*Configure).Run:368 vic-machine-linux
vic/cmd/vic-machine/common.NewOperation:27 vic-machine-linux
ERRO[0180] --------------------
ERRO[0180] vic-machine-linux configure failed: Configure timed out: use --timeout to add more time

@BurningTheSky
Copy link
Author

Shall I just add more timeout and see if it completes? How much should I add?

@BurningTheSky
Copy link
Author

If I check the config of the vch is is showing the new datastore in there, and the vcenter is reporting constant reconfigurations on the container VMs now... Something strange is going on

@YanzhaoLi
Copy link
Member

It is probably due to failing to connect the nfsserver.

  • Please make sure the nfs-server work well, which means you can mount it.
  • Check the vch's tether.debug. Download it from VC: datastore->vch_name->tether.debug. There might be errors like error occurred while attempting to mount volumestore (NFSMultimedia). If not, we should do further debugging.

@BurningTheSky
Copy link
Author

I can't see anything NFS related but the error loop in there has been going constantly from then until now:
2020/02/26 08:25:59 unknown command: "Time_Synchronize"
2020/02/26 08:25:59 Message: Unable to send a message over the communication channel 0
all stderr redirected to debug logall stdout redirected to debug logFeb 26 2020 08:26:23.172Z INFO Registering tether extension Toolbox
Feb 26 2020 08:26:23.173Z INFO Registering tether extension logrotate
Feb 26 2020 08:26:23.174Z INFO opening var/log/vic/init.log for debug log
Feb 26 2020 08:26:23.175Z INFO Started reaping child processes
Feb 26 2020 08:26:23.176Z INFO unmounting /etc/hosts
Feb 26 2020 08:26:23.177Z INFO bind-mounting /.tether/etc/hosts on /etc/hosts
Feb 26 2020 08:26:23.179Z INFO Starting extension Toolbox
Feb 26 2020 08:26:23.180Z INFO toolbox: waiting for initialization
Feb 26 2020 08:26:23.181Z INFO Launching vch-init pprof server on 127.0.0.1:6060
2020/02/26 08:26:23 dispatching power op "OS_PowerOn"
Feb 26 2020 08:26:23.187Z INFO toolbox: service is ready (power on event received)
Feb 26 2020 08:26:23.188Z INFO Starting extension logrotate
Feb 26 2020 08:26:23.189Z INFO Loading main configuration
Feb 26 2020 08:26:23.219Z INFO unmounting /etc/hostname
Feb 26 2020 08:26:23.220Z INFO bind-mounting /.tether/etc/hostname on /etc/hostname
Feb 26 2020 08:26:23.223Z INFO unmounting /etc/hosts
Feb 26 2020 08:26:23.225Z INFO bind-mounting /.tether/etc/hosts on /etc/hosts
Feb 26 2020 08:26:23.227Z INFO Renaming link eth1 to bridge
Feb 26 2020 08:26:23.231Z INFO successfully applied config for network bridge
Feb 26 2020 08:26:23.232Z INFO Renaming link eth0 to client
Feb 26 2020 08:26:23.242Z INFO setting ip address 192.168.0.202/24 for link client
Feb 26 2020 08:26:23.245Z WARN ignoring incomplete line "127.0.1.1"
Feb 26 2020 08:26:23.246Z INFO unmounting /etc/hosts
Feb 26 2020 08:26:23.248Z INFO bind-mounting /.tether/etc/hosts on /etc/hosts
Feb 26 2020 08:26:23.250Z INFO Added nameservers: [192.168.0.87]
Feb 26 2020 08:26:23.251Z INFO unmounting /etc/resolv.conf
Feb 26 2020 08:26:23.252Z INFO bind-mounting /.tether/etc/resolv.conf on /etc/resolv.conf
Feb 26 2020 08:26:23.254Z INFO successfully applied config for network client
Feb 26 2020 08:26:23.256Z INFO setting ip address 192.168.0.202/24 for link client
Feb 26 2020 08:26:23.257Z WARN address 192.168.0.202/24 already set on interface client
Feb 26 2020 08:26:23.258Z INFO unmounting /etc/hosts
Feb 26 2020 08:26:23.260Z INFO bind-mounting /.tether/etc/hosts on /etc/hosts
Feb 26 2020 08:26:23.262Z INFO Added nameservers: [192.168.0.87]
Feb 26 2020 08:26:23.264Z INFO unmounting /etc/resolv.conf
Feb 26 2020 08:26:23.266Z INFO bind-mounting /.tether/etc/resolv.conf on /etc/resolv.conf
Feb 26 2020 08:26:23.268Z INFO successfully applied config for network management
Feb 26 2020 08:26:23.269Z INFO Aliasing link client to public
Feb 26 2020 08:26:23.271Z INFO setting ip address 192.168.0.202/24 for link client
Feb 26 2020 08:26:23.272Z WARN address 192.168.0.202/24 already set on interface client
Feb 26 2020 08:26:23.274Z INFO Setting default gateway to 192.168.0.1
Feb 26 2020 08:26:23.275Z INFO updated default route to public interface, gateway: 192.168.0.1
Feb 26 2020 08:26:23.276Z INFO unmounting /etc/hosts
Feb 26 2020 08:26:23.278Z INFO bind-mounting /.tether/etc/hosts on /etc/hosts
Feb 26 2020 08:26:23.280Z INFO Added nameservers: [192.168.0.87]
Feb 26 2020 08:26:23.281Z INFO unmounting /etc/resolv.conf
Feb 26 2020 08:26:23.282Z INFO bind-mounting /.tether/etc/resolv.conf on /etc/resolv.conf
Feb 26 2020 08:26:23.284Z INFO successfully applied config for network public
Feb 26 2020 08:26:23.304Z INFO Firewalld running: false
Feb 26 2020 08:26:23.309Z INFO opening var/log/vic/vicadmin.log for session logging
Feb 26 2020 08:26:23.310Z INFO opening var/log/vic/docker-personality.log for session logging
Feb 26 2020 08:26:23.312Z INFO opening var/log/vic/port-layer.log for session logging
Feb 26 2020 08:26:23.313Z INFO Launching process for session port-layer
Feb 26 2020 08:26:23.314Z INFO Launching process for session vicadmin
Feb 26 2020 08:26:23.316Z INFO Launching process for session docker-personality
Feb 26 2020 08:26:35.390Z INFO Triggered reload
Feb 26 2020 08:26:35.390Z INFO Loading main configuration
Feb 26 2020 08:26:35.419Z WARN Session docker-personality already initialized
Feb 26 2020 08:26:35.420Z WARN Session port-layer already initialized
Feb 26 2020 08:26:35.421Z WARN Session vicadmin already initialized
Feb 26 2020 08:26:35.425Z WARN Re-launching process for session port-layer (count: 1)
Feb 26 2020 08:26:35.427Z INFO opening var/log/vic/port-layer.log for session logging
Feb 26 2020 08:26:45.855Z INFO Triggered reload
Feb 26 2020 08:26:45.856Z INFO Loading main configuration
Feb 26 2020 08:26:45.905Z WARN Session port-layer already initialized
Feb 26 2020 08:26:45.907Z WARN Session vicadmin already initialized
Feb 26 2020 08:26:45.908Z WARN Session docker-personality already initialized
Feb 26 2020 08:26:45.916Z WARN Re-launching process for session port-layer (count: 2)
Feb 26 2020 08:26:45.919Z INFO opening var/log/vic/port-layer.log for session logging
.
.
.
.
.
.
.
Feb 26 2020 16:45:05.698Z INFO Triggered reload
Feb 26 2020 16:45:05.699Z INFO Loading main configuration
Feb 26 2020 16:45:05.733Z WARN Session port-layer already initialized
Feb 26 2020 16:45:05.734Z WARN Session vicadmin already initialized
Feb 26 2020 16:45:05.735Z WARN Session docker-personality already initialized
Feb 26 2020 16:45:05.743Z WARN Re-launching process for session port-layer (count: 2822)
Feb 26 2020 16:45:05.744Z INFO opening var/log/vic/port-layer.log for session logging

@BurningTheSky
Copy link
Author

I've restared the vic and vch containers but the containers are still constantly reconfiguring

@BurningTheSky
Copy link
Author

I also can't connect to the docker now:
[burningsky@engal ~]$ docker --tls ps -a
Cannot connect to the Docker daemon at tcp://192.168.0.202:2376. Is the docker daemon running?

@YanzhaoLi
Copy link
Member

That'r really weird.

  1. Have you ever tested mounting your nfs-server manually in your local host? Like mount -t nfs 192.168.0.14:/Multimedia LOCALDIR
  2. If OK, please collect vch bundle logs for further debug: https://vmware.github.io/vic-product/assets/files/html/1.5/vic_vsphere_admin/log_bundles.html

Thanks.

@BurningTheSky
Copy link
Author

I mistyped the path to the NFS, woops. Can I remove it from the config and add the correct path or can you not remove volumes even if they aren't working?

@YanzhaoLi
Copy link
Member

em... We didn't support change volume-stores.
Thanks again for your use-case. It seems we should not store the configuration info if we failed to mount the nfs-store.

@BurningTheSky
Copy link
Author

Glad to help! End users are good for testing haha. I will redeloy the VCH and see if it goes smoother.

@BurningTheSky
Copy link
Author

Making good progress, is there any way to mount an NFS share into the container rather than having to create a shared volume? It would be a pain to have to move all of my data

@YanzhaoLi
Copy link
Member

We don't support other ways.
But why do you need move all of the data?

@BurningTheSky
Copy link
Author

So I want to be able to access data shared with servers as well as the containers. The file structure is /mnt/user/data, but if I put that at as the volume location then it creates the ../volumes/data so I end up with /mnt/user/data/volumes/data. I want the container to have access directly into /mnt/user/data

@m4rkireland
Copy link

Any movement on this one?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@YanzhaoLi YanzhaoLi

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@YanzhaoLi @BurningTheSky @m4rkireland