tests fail with build of ntirpc 3.3 on Debian unstable

[christmart]

While building ntirpc 3.3 on Debian unstable I tried to run the test suit with

for i in raw rdma tcp udp; do \
	./tests/rpcping $$i localhost --count=1000 --threads=4 --workers=2; \
done

Every call version above results in a segmentation fault. strace shows:

[pid 23495] sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\200\0\0(", iov_len=4}, {iov_base="\0\0\0\2\0\0\0\0\0\0\0\2\0\1\206\243\0\0\0\3\0\0\0\0\0\0\0\0\0\0\0\0"..., iov_len=40}], msg_iovlen=2, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT <unfinished ...>
[pid 23494] <... write resumed>) = 4
[pid 23492] rt_sigprocmask(SIG_SETMASK, [], <unfinished ...>
[pid 23495] <... sendmsg resumed>) = -1 ECONNREFUSED (Connection refused)
[pid 23493] <... read resumed>"\0\0\0\0", 4) = 4
[pid 23492] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid 23495] epoll_ctl(6, EPOLL_CTL_DEL, 3, 0x55c6c6a98158 <unfinished ...>
[pid 23494] write(4, "\0\0\0\0", 4 <unfinished ...>
[pid 23492] mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0 <unfinished ...>
[pid 23493] epoll_wait(6, <unfinished ...>
[pid 23495] <... epoll_ctl resumed>) = 0
[pid 23492] <... mmap resumed>) = 0x7fa6a00e6000
[pid 23494] <... write resumed>) = 4
[pid 23492] mprotect(0x7fa6a00e7000, 8388608, PROT_READ|PROT_WRITE <unfinished ...>
[pid 23495] futex(0x7fa69c000bb4, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1600699135, tv_nsec=456362868}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 23493] <... epoll_wait resumed>[{EPOLLIN, {u32=5, u64=5}}], 512, 29996) = 1
[pid 23492] <... mprotect resumed>) = 0
[pid 23494] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x60} ---

I can provide a coredump ..

[dang]

RDMA certainly won't work, as it was never properly finished. UDP and TCP worked last I'm I tried rpcping, but that was, admittedly, a long time ago. Go ahead and post the backtrace.

[christmart]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./tests/rpcping tcp localhost --count=1000 --threads=4 --workers=2'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fe048ee0760 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
[Current thread is 1 (Thread 0x7fe0488aa700 (LWP 128873))]

(gdb) back
#0 0x00007fe048ee0760 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fe048f19ca6 in svc_rqst_expire_insert (cc=cc@entry=0x7fe03c001770) at ./src/svc_rqst.c:309
#2 0x00007fe048f039d9 in clnt_req_callback (cc=0x7fe03c001770) at ./src/clnt_generic.c:450
#3 0x000055c640493c29 in worker (arg=0x55c640f263a0) at ./tests/rpcping.c:175
#4 0x00007fe048eddea7 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fe048e0deaf in clone () from /lib/x86_64-linux-gnu/libc.so.6

[simon28li]

@dang I encountered the same problem in version 4.0. I observed that this test file has not been updated for a long time. Is it out of date and not applicable? How do I test myself?

[dang]

It is certainly out-of-date. It was written to help develop RDMA, and hasn't been used since the developer working on RDMA left the project. I'm open to having it working again, but I don't think I'll have time to work on it myself.

[Gaurav-Gangalwar]

server side rdma support is enabled now in tirpc, but we need to fix rdma clnt also to make rpcping work.

[ffilz]

server side rdma support is enabled now in tirpc, but we need to fix rdma clnt also to make rpcping work.

Are you considering making this work eventually? If not, should we remove rpcping?

[dang]

Don't we need clnt anyway for the backchannel?

[Gaurav-Gangalwar]

yeah, eventually we want to fix rdma_clnt to enable callback channels, that should fix rpcping also.
But for now we should disable rdma from rpcping till we fix it.

[Rajpr2024]

@christmart Today, I executed the same script on my setup, and it appears to be functioning correctly (except RDMA) for me.

siwserver2:/home/rp/sb/build_dir/libntirpc# for i in raw rdma tcp udp; do ./tests/rpcping $i localhost --count=1000 --threads=4 --workers=2; done
rpcping raw localhost count=1000 threads=4 workers=2 (port=2049 program=100003 version=3 procedure=0): failures 0 timeouts 0 mean 65.4310, total 261.7239

rpcping rdma localhost count=1000 threads=4 workers=2 (port=2049 program=100003 version=3 procedure=0): failures 0 timeouts 0 mean 99.9256, total 399.7023

rpcping tcp localhost count=1000 threads=4 workers=2 (port=2049 program=100003 version=3 procedure=0): failures 0 timeouts 0 mean 112.4624, total 449.8494

rpcping udp localhost count=1000 threads=4 workers=2 (port=2049 program=100003 version=3 procedure=0): failures 0 timeouts 0 mean 89.8524, total 359.4096

I ran the mentioned commands on x86_64 (CentOS Stream release 9) with ntirpc version 6.3. Currently, the "rpcping" utility does not support RDMA. Could you please check with the latest version of Ganesha.

[christmart]

In my environment libntirpc 6.3 still segfaults:

• ./tests/rpcping raw localhost --count=1000 --threads=4 --workers=2
  Segmentation fault

[Rajpr2024]

Even I can see the segfault on my setup but this is happening only in case the server is not running.

siwserver2:/home/rp/sb/build_dir/libntirpc/tests# ./rpcping raw localhost --count=1000 --threads=4 --workers=2
Segmentation fault (core dumped)

[ffilz]

Hmm, we know rpcping rdma needs fixing.

Not sure we support raw... That would be a raw "Unix" socket I believe.

[Rajpr2024]

@ffilz Even if we don't support RAW transport, rpcping shouldn't cause a segmentation fault. Upon investigation, I found that the utility will segfault for any transport (supported or not) if Ganesha is not running.

[ffilz]

Can we fix that?