How to use shared memory to send and receive messages within same host in UCX ?

[PrateekJoshi]

According to UCX documentation, it appears that UCX provides intra-node communication using the shared memory of the host machine. I want to implement client-server communication utilizing UCX shared memory, where messages are stored and exchanged using host memory.

There is no clear example in the openucx repo on how to achieve this. If the UCX team can add this example or provide any reference, it will be helpful for people who are looking to use shm transport. Also which log trace to check if messages are using shm transport from UCX DEBUG logs ?

[PrateekJoshi]

Can anyone please help regarding this query ? @bosilca @yosefe @dmitrygx

[yosefe]

There is nothing special needed in the code to use shared memory. It should be used automatically when communicating between processes on the same node.

[PrateekJoshi]

From the docs my assumption was also same but I suspect it is NOT using SHM when I run client and server in same host. Note that I am using AM protocol and UCX_RNDV_THRESH=8192. My payload size is 10 MB for BW measurements.

With UCX_TLS=all :

1. My client and server app is working fine but send BW at client side is very low ~300 MB/sec.
2. I am suspecting SHM is not getting used as I see very low bandwidth. Is there any log to confirm if SHM getting used for message transfer ?

With UCX_TLD=shm :

1. My client application is throwing below error:

1750326976.173910] [prateek:16802:0]          uct_cm.c:98   UCX  DIAG  resolve callback failed with error: Destination is unreachable

How can I force SHM transport and run my client-server app without this error ?

Full server logs :

prateek@prateek:~/Workspace/UCX/build$ ./examples/AM/server.bin 
[1750327126.759550] [prateek:18577:0]           debug.c:1157 UCX  DEBUG using signal stack 0x70c9cd8cb000 size 141824
[1750327126.759784] [prateek:18577:0]            init.c:120  UCX  DEBUG /lib/x86_64-linux-gnu/libucs.so.0 loaded at 0x70c9cda6f000
[1750327126.759819] [prateek:18577:0]            init.c:122  UCX  DEBUG cmd line: ./examples/AM/server.bin 
[1750327126.759833] [prateek:18577:0]          module.c:72   UCX  DEBUG ucs library path: /lib/x86_64-linux-gnu/libucs.so.0
[1750327126.759842] [prateek:18577:0]          module.c:280  UCX  DEBUG loading modules for ucs
[1750327126.760412] [prateek:18577:0]            time.c:22   UCX  DEBUG arch clock frequency: 2600000000.00 Hz
UCX_NET_DEVICES=all
UCX_SHM_DEVICES=all
UCX_ACC_DEVICES=all
UCX_SELF_DEVICES=all
UCX_TLS=shm
UCX_PROTOS=all
UCX_ALLOC_PRIO=md:sysv,md:posix,thp,md:*,mmap,heap
UCX_RNDV_FRAG_SIZE=host:512K,cuda:4M
UCX_RNDV_FRAG_ALLOC_COUNT=host:128,cuda:128
UCX_SOCKADDR_TLS_PRIORITY=rdmacm,tcp,sockcm
UCX_WARN_INVALID_CONFIG=y
UCX_RX_MPOOL_SIZES=64,1K
UCX_RCACHE_ENABLE=try
UCX_RCACHE_MEM_PRIO=500
UCX_RCACHE_OVERHEAD=auto
UCX_RCACHE_ADDR_ALIGN=64
UCX_RCACHE_MAX_REGIONS=inf
UCX_RCACHE_MAX_SIZE=inf
UCX_RCACHE_MAX_UNRELEASED=inf
UCX_RCACHE_PURGE_ON_FORK=y
UCX_SELECT_DISTANCE_MD=cuda_cpy
UCX_MEMTYPE_REG_WHOLE_ALLOC_TYPES=cuda
UCX_RNDV_MEMTYPE_DIRECT_SIZE=inf
UCX_BCOPY_THRESH=auto
UCX_RNDV_THRESH=8K
UCX_RNDV_SEND_NBR_THRESH=256K
UCX_RNDV_THRESH_FALLBACK=inf
UCX_RNDV_PERF_DIFF=1.000
UCX_MULTI_LANE_MAX_RATIO=4.000
UCX_MULTI_PATH_RATIO=auto
UCX_MAX_EAGER_RAILS=1
UCX_MAX_RNDV_RAILS=2
UCX_MAX_RMA_RAILS=1
UCX_MIN_RNDV_CHUNK_SIZE=16K
UCX_RNDV_SCHEME=auto
UCX_RKEY_PTR_SEG_SIZE=512K
UCX_ZCOPY_THRESH=auto
UCX_BCOPY_BW=auto
UCX_ATOMIC_MODE=guess
UCX_ADDRESS_DEBUG_INFO=n
UCX_MAX_WORKER_ADDRESS_NAME=32
UCX_USE_MT_MUTEX=n
UCX_ADAPTIVE_PROGRESS=y
UCX_SEG_SIZE=8K
UCX_TM_THRESH=1K
UCX_TM_MAX_BB_SIZE=1K
UCX_TM_FORCE_THRESH=8K
UCX_TM_SW_RNDV=n
UCX_NUM_EPS=auto
UCX_NUM_PPN=auto
UCX_RNDV_FRAG_MEM_TYPE=host
UCX_RNDV_PIPELINE_SEND_THRESH=inf
UCX_RNDV_PIPELINE_SHM_ENABLE=y
UCX_FLUSH_WORKER_EPS=y
UCX_FENCE_MODE=auto
UCX_UNIFIED_MODE=n
UCX_CM_USE_ALL_DEVICES=y
UCX_LISTENER_BACKLOG=auto
UCX_PROTO_ENABLE=y
UCX_PROTO_REQUEST_RESET=n
UCX_KEEPALIVE_INTERVAL=20000000.00us
UCX_KEEPALIVE_NUM_EPS=128
UCX_RESOLVE_REMOTE_EP_ID=off
UCX_PROTO_INDIRECT_ID=auto
UCX_RNDV_PUT_FORCE_FLUSH=n
UCX_SA_DATA_VERSION=v2
UCX_RKEY_MPOOL_MAX_MD=2
UCX_ADDRESS_VERSION=v1
UCX_PROTO_INFO=n
UCX_RNDV_ALIGN_THRESH=64K
UCX_PROTO_INFO_DIR=
UCX_REG_NONBLOCK_MEM_TYPES=
UCX_PREFER_OFFLOAD=y
[1750327126.760636] [prateek:18577:0]     ucp_context.c:2137 UCX  WARN  UCP API version is incompatible: required >= 1.18, actual 1.16.0 (loaded from /lib/x86_64-linux-gnu/libucp.so.0)
[1750327126.760679] [prateek:18577:0]     ucp_context.c:1903 UCX  DEBUG estimated number of endpoints is 1
[1750327126.760683] [prateek:18577:0]     ucp_context.c:1910 UCX  DEBUG estimated number of endpoints per node is 1
[1750327126.760689] [prateek:18577:0]     ucp_context.c:1921 UCX  DEBUG estimated bcopy bandwidth is 7340032000.000000
[1750327126.760703] [prateek:18577:0]     ucp_context.c:1980 UCX  DEBUG allocation method[0] is md 'sysv'
[1750327126.760706] [prateek:18577:0]     ucp_context.c:1980 UCX  DEBUG allocation method[1] is md 'posix'
[1750327126.760717] [prateek:18577:0]     ucp_context.c:1992 UCX  DEBUG allocation method[2] is 'thp'
[1750327126.760721] [prateek:18577:0]     ucp_context.c:1980 UCX  DEBUG allocation method[3] is md '*'
[1750327126.760725] [prateek:18577:0]     ucp_context.c:1992 UCX  DEBUG allocation method[4] is 'mmap'
[1750327126.760728] [prateek:18577:0]     ucp_context.c:1992 UCX  DEBUG allocation method[5] is 'heap'
[1750327126.760746] [prateek:18577:0]          module.c:280  UCX  DEBUG loading modules for uct
[1750327126.762043] [prateek:18577:0]          module.c:280  UCX  DEBUG loading modules for uct_ib
[1750327126.763983] [prateek:18577:0]     ucp_context.c:1561 UCX  DEBUG closing md self because it has no selected transport resources
[1750327126.764358] [prateek:18577:0]            topo.c:787  UCX  DEBUG /sys/class/net/lo: sysfs path undetected
[1750327126.764368] [prateek:18577:0]            topo.c:479  UCX  DEBUG lo: system device unknown
[1750327126.764489] [prateek:18577:0]            topo.c:791  UCX  DEBUG /sys/class/net/wlp2s0: PF sysfs path is '/sys/devices/pci0000:00/0000:00:1c.3/0000:02:00.0'
[1750327126.764500] [prateek:18577:0]            topo.c:240  UCX  DEBUG added sys_dev 0 for bus id 02:00.0
[1750327126.764504] [prateek:18577:0]            topo.c:475  UCX  DEBUG wlp2s0: bdf_name 0000:02:00.0 sys_dev 0
[1750327126.764520] [prateek:18577:0]     ucp_context.c:1561 UCX  DEBUG closing md tcp because it has no selected transport resources
[1750327126.764805] [prateek:18577:0]    ib_mlx5dv_md.c:1743 UCX  DEBUG rxe_wlp2s0: DEVX is not supported
[1750327126.764812] [prateek:18577:0]           ib_md.c:1072 UCX  DEBUG rxe_wlp2s0: md open by 'uct_ib_mlx5_devx_md_ops' failed, trying next
[1750327126.764817] [prateek:18577:0]           ib_md.c:1072 UCX  DEBUG rxe_wlp2s0: md open by 'uct_ib_mlx5_md_ops' failed, trying next
[1750327126.764908] [prateek:18577:0]            topo.c:787  UCX  DEBUG /sys/class/infiniband/rxe_wlp2s0: sysfs path undetected
[1750327126.764914] [prateek:18577:0]            topo.c:479  UCX  DEBUG rxe_wlp2s0: system device unknown
[1750327126.764919] [prateek:18577:0]       ib_device.c:486  UCX  DEBUG rxe_wlp2s0: vendor_id 0x0 device_id 0
[1750327126.764924] [prateek:18577:0]            topo.c:744  UCX  DEBUG rxe_wlp2s0: pci bandwidth undetected, using maximal value
[1750327126.764942] [prateek:18577:0]             sys.c:431  UCX  DEBUG failed to open /sys/class/infiniband/rxe_wlp2s0/device/local_cpus: No such file or directory
[1750327126.764964] [prateek:18577:0]           async.c:231  UCX  DEBUG added async handler 0x6462df910310 [id=4 ref 1] ???() to hash
[1750327126.765129] [prateek:18577:0]           async.c:493  UCX  DEBUG listening to async event fd 4 events 0x1 mode thread_spinlock
[1750327126.765139] [prateek:18577:0]       ib_device.c:584  UCX  DEBUG initialized device 'rxe_wlp2s0' (InfiniBand channel adapter) with 1 ports
[1750327126.765149] [prateek:18577:0]           ib_md.c:1126 UCX  DEBUG rxe_wlp2s0: cuda GPUDirect RDMA is disabled
[1750327126.765157] [prateek:18577:0]           ib_md.c:1126 UCX  DEBUG rxe_wlp2s0: rocm GPUDirect RDMA is disabled
[1750327126.765165] [prateek:18577:0]           ib_md.c:1147 UCX  DEBUG rxe_wlp2s0: ibv_reg_dmabuf_mr(fd=-1) returned Operation not supported, dmabuf is not supported
[1750327126.765298] [prateek:18577:0]           ib_md.c:1115 UCX  DEBUG rxe_wlp2s0: relaxed order memory access is disabled
[1750327126.765304] [prateek:18577:0]           ib_md.c:1066 UCX  DEBUG rxe_wlp2s0: md open by 'uct_ib_verbs_md_ops' is successful
[1750327126.765346] [prateek:18577:0]          uct_md.c:96   UCX  DEBUG failed to query dc_mlx5 resources: No such device
[1750327126.765461] [prateek:18577:0]          uct_md.c:96   UCX  DEBUG failed to query rc_mlx5 resources: No such device
[1750327126.765489] [prateek:18577:0]          uct_md.c:96   UCX  DEBUG failed to query ud_mlx5 resources: No such device
[1750327126.765506] [prateek:18577:0]     ucp_context.c:1561 UCX  DEBUG closing md rxe_wlp2s0 because it has no selected transport resources
[1750327126.765516] [prateek:18577:0]       ib_device.c:605  UCX  DEBUG destroying ib device rxe_wlp2s0
[1750327126.765527] [prateek:18577:0]           async.c:156  UCX  DEBUG removed async handler 0x6462df910310 [id=4 ref 1] ???() from hash
[1750327126.765532] [prateek:18577:0]           async.c:546  UCX  DEBUG removing async handler 0x6462df910310 [id=4 ref 1] ???()
[1750327126.768115] [prateek:18577:0]           async.c:171  UCX  DEBUG release async handler 0x6462df910310 [id=4 ref 0] ???()
[1750327126.768251] [prateek:18577:0]     ucp_context.c:1626 UCX  DEBUG register host memory on: cma
[1750327126.768270] [prateek:18577:0]     ucp_context.c:1614 UCX  DEBUG no memory domain supports registering cuda memory
[1750327126.768283] [prateek:18577:0]     ucp_context.c:1614 UCX  DEBUG no memory domain supports registering cuda-managed memory
[1750327126.768297] [prateek:18577:0]     ucp_context.c:1614 UCX  DEBUG no memory domain supports registering rocm memory
[1750327126.768310] [prateek:18577:0]     ucp_context.c:1614 UCX  DEBUG no memory domain supports registering rocm-managed memory
[1750327126.768322] [prateek:18577:0]     ucp_context.c:1614 UCX  DEBUG no memory domain supports registering rdma memory
[1750327126.768366] [prateek:18577:0]           mpool.c:136  UCX  DEBUG mpool rcache_mp: align 8, maxelems 4294967295, elemsize 144
[1750327126.768407] [prateek:18577:0]           async.c:231  UCX  DEBUG added async handler 0x6462df9064f0 [id=3 ref 1] ???() to hash
[1750327126.768495] [prateek:18577:0]           async.c:493  UCX  DEBUG listening to async event fd 3 events 0x1 mode thread_spinlock
[1750327126.768737] [prateek:18577:0]          module.c:280  UCX  DEBUG loading modules for ucm
[1750327126.769137] [prateek:18577:0]     ucp_context.c:2209 UCX  DEBUG created ucp context ucp_server 0x6462df901550 [3 mds 3 tls] features 0xe3 tl bitmap 0x7 0x0
[1750327126.769321] [prateek:18577:0]         mm_sysv.c:97   UCX  DEBUG   mm failed to allocate 33023 bytes with hugetlb
[1750327126.772134] [prateek:18577:0]           mpool.c:136  UCX  DEBUG mpool mm_recv_desc: align 64, maxelems 4294967295, elemsize 8368
[1750327126.772177] [prateek:18577:0]         mm_sysv.c:97   UCX  DEBUG   mm failed to allocate 4292720 bytes with hugetlb
[1750327126.772216] [prateek:18577:0]           mpool.c:281  UCX  DEBUG mpool mm_recv_desc: allocated chunk 0x70c9cc3e7018 of 4296680 bytes with 512 elements
[1750327126.773970] [prateek:18577:0]        mm_iface.c:711  UCX  DEBUG created mm iface 0x6462df900730 FIFO id 0x18008 va 0x70c9cd144000 size 36864 (128 x 256 elems)
[1750327126.774066] [prateek:18577:0]      ucp_worker.c:1424 UCX  DEBUG created interface[0]=0x6462df900730 using sysv/memory on worker 0x6462df9188b0
[1750327126.774272] [prateek:18577:0]        mm_posix.c:569  UCX  DEBUG   allocated posix shared memory at 0x70c9cd13b000 length 36864
[1750327126.774316] [prateek:18577:0]           mpool.c:136  UCX  DEBUG mpool mm_recv_desc: align 64, maxelems 4294967295, elemsize 8368
[1750327126.776133] [prateek:18577:0]        mm_posix.c:332  UCX  DEBUG   shared memory mmap(addr=(nil), length=6291456, flags= HUGETLB, fd=11) failed: Invalid argument
[1750327126.777476] [prateek:18577:0]        mm_posix.c:569  UCX  DEBUG   allocated posix shared memory at 0x70c9cbfce000 length 4296704
[1750327126.777556] [prateek:18577:0]           mpool.c:281  UCX  DEBUG mpool mm_recv_desc: allocated chunk 0x70c9cbfce018 of 4296680 bytes with 512 elements
[1750327126.778626] [prateek:18577:0]        mm_iface.c:711  UCX  DEBUG created mm iface 0x6462df8fedc0 FIFO id 0xc000000240004891 va 0x70c9cd13b000 size 36864 (128 x 256 elems)
[1750327126.778765] [prateek:18577:0]      ucp_worker.c:1424 UCX  DEBUG created interface[1]=0x6462df8fedc0 using posix/memory on worker 0x6462df9188b0
[1750327126.778876] [prateek:18577:0]           mpool.c:136  UCX  DEBUG mpool uct_scopy_iface_tx_mp: align 64, maxelems 4294967295, elemsize 736
[1750327126.778952] [prateek:18577:0]      ucp_worker.c:1424 UCX  DEBUG created interface[2]=0x6462df8ff8e0 using cma/memory on worker 0x6462df9188b0
[1750327126.779035] [prateek:18577:0]      ucp_worker.c:1144 UCX  DEBUG selected scalable tl bitmap: 0x7 0x0 (3 tls)
[1750327126.779092] [prateek:18577:0]           async.c:231  UCX  DEBUG added async handler 0x6462df910310 [id=8 ref 1] ???() to hash
[1750327126.779122] [prateek:18577:0]           async.c:493  UCX  DEBUG listening to async event fd 8 events 0x0 mode thread_spinlock
[1750327126.779202] [prateek:18577:0]           async.c:231  UCX  DEBUG added async handler 0x6462df9134f0 [id=10 ref 1] ???() to hash
[1750327126.779250] [prateek:18577:0]           async.c:493  UCX  DEBUG listening to async event fd 10 events 0x0 mode thread_spinlock
[1750327126.779518] [prateek:18577:0]           async.c:231  UCX  DEBUG added async handler 0x6462df914290 [id=12 ref 1] ???() to hash
[1750327126.779611] [prateek:18577:0]           async.c:493  UCX  DEBUG listening to async event fd 12 events 0x1 mode thread_spinlock
[1750327126.779687] [prateek:18577:0]       rdmacm_cm.c:981  UCX  DEBUG created rdmacm_cm 0x6462df938810 with event_channel 0x6462df8fc5d0 (fd=12)
[1750327126.779763] [prateek:18577:0]      tcp_sockcm.c:225  UCX  DEBUG created tcp_sockcm 0x6462df93b180
[1750327126.779842] [prateek:18577:0]           mpool.c:136  UCX  DEBUG mpool ucp_requests: align 64, maxelems 4294967295, elemsize 280
[1750327126.779905] [prateek:18577:0]           mpool.c:136  UCX  DEBUG mpool ucp_rkeys: align 64, maxelems 4294967295, elemsize 112
[1750327126.779921] [prateek:18577:0]           mpool.c:136  UCX  DEBUG mpool ucp_reg_bufs: align 64, maxelems 4294967295, elemsize 8216
[1750327126.779936] [prateek:18577:0]           mpool.c:136  UCX  DEBUG mpool ucp_am_bufs: align 64, maxelems 4294967295, elemsize 153
[1750327126.779974] [prateek:18577:0]           mpool.c:136  UCX  DEBUG mpool ucp_am_bufs: align 64, maxelems 4294967295, elemsize 1113
[1750327126.779996] [prateek:18577:0]           mpool.c:136  UCX  DEBUG mpool ucp_am_bufs: align 64, maxelems 4294967295, elemsize 8345
[1750327126.780036] [prateek:18577:0]       mpool_set.c:129  UCX  DEBUG mpool_set:ucp_am_bufs, sizes map 0x80000440, largest size 8256, mpools num 3
[1750327126.780075] [prateek:18577:0]          parser.c:2042 UCX  INFO  UCX_* env variables: UCX_RNDV_THRESH=8192 UCX_TLS=shm UCX_LOG_LEVEL=DEBUG
FILE: /home/prateek/Workspace/UCX/src/common/common.cc FUNCTION : SetSockAddr LINE : 37 message: Server will listen on 192.168.29.202:9090
[1750327126.780311] [prateek:18577:0] rdmacm_listener.c:104  UCX  DEBUG listener 0x6462df90fde0: created on cm 0x6462df938810 192.168.29.202:9090 rdma_cm_id 0x6462df93c660
[1750327126.780431] [prateek:18577:0]           async.c:231  UCX  DEBUG added async handler 0x6462df907160 [id=15 ref 1] ???() to hash
[1750327126.780505] [prateek:18577:0]           async.c:493  UCX  DEBUG listening to async event fd 15 events 0x5 mode thread_spinlock
[1750327126.780557] [prateek:18577:0]    tcp_listener.c:134  UCX  DEBUG created a TCP listener 0x6462df910ff0 on cm 0x6462df93b180 with fd: 15 listening on 192.168.29.202:9090
FILE: /home/prateek/Workspace/UCX/src/server/server.cc FUNCTION : StartListener LINE : 57 message: Server is listening on IP 192.168.29.202 port 9090
FILE: /home/prateek/Workspace/UCX/src/server/server.cc FUNCTION : StartListener LINE : 61 message: Waiting for connection...
FILE: /home/prateek/Workspace/UCX/src/server/server.cc FUNCTION : RegisterUserCb LINE : 556 message: User message callback registered successfully
FILE: /home/prateek/Workspace/UCX/src/server/server.cc FUNCTION : StartServer LINE : 358 message: Starting Server 
FILE: /home/prateek/Workspace/UCX/src/server/server.cc FUNCTION : ReceiveActiveMsg LINE : 382 message: Receive messages using active message protocol
FILE: /home/prateek/Workspace/UCX/src/server/server.cc FUNCTION : ProgressWorker LINE : 507 message: Worker progress started

Full client logs :

prateek@prateek:~/Workspace/UCX/build$ ./examples/AM/client.bin -s 100000000 -n 100
[1750326976.104754] [prateek:16802:0]           debug.c:1158 UCX  DEBUG using signal stack 0x71072bcc3000 size 141824
[1750326976.104984] [prateek:16802:0]            init.c:121  UCX  DEBUG /lib/libucs.so.0 loaded at 0x71072b200000
[1750326976.105024] [prateek:16802:0]            init.c:122  UCX  DEBUG cmd line: ./examples/AM/client.bin -s 100000000 -n 100 
[1750326976.105039] [prateek:16802:0]          module.c:72   UCX  DEBUG ucs library path: /lib/libucs.so.0
[1750326976.105046] [prateek:16802:0]          module.c:280  UCX  DEBUG loading modules for ucs
FILE: /home/prateek/Workspace/UCX/examples/AM/client.cc FUNCTION : main LINE : 24 message: Server Address: 192.168.29.202, Port: 9090, Payload Size: 100000000 bytes, Iteration Count: 100
[1750326976.161076] [prateek:16802:0]            time.c:22   UCX  DEBUG arch clock frequency: 2600000000.00 Hz
UCX_NET_DEVICES=all
UCX_SHM_DEVICES=all
UCX_ACC_DEVICES=all
UCX_SELF_DEVICES=all
UCX_TLS=shm
UCX_PROTOS=all
UCX_ALLOC_PRIO=md:sysv,md:posix,thp,md:*,mmap,heap
UCX_RNDV_FRAG_SIZE=host:512K,cuda:4M
UCX_RNDV_FRAG_ALLOC_COUNT=host:128,cuda:128
UCX_SOCKADDR_TLS_PRIORITY=rdmacm,tcp,sockcm
UCX_WARN_INVALID_CONFIG=y
UCX_RX_MPOOL_SIZES=64,1K
UCX_RCACHE_ENABLE=try
UCX_RCACHE_MEM_PRIO=500
UCX_RCACHE_OVERHEAD=auto
UCX_RCACHE_MAX_REGIONS=inf
UCX_RCACHE_MAX_SIZE=inf
UCX_RCACHE_MAX_UNRELEASED=inf
UCX_RCACHE_PURGE_ON_FORK=y
UCX_SELECT_DISTANCE_MD=cuda_cpy
UCX_MEMTYPE_REG_WHOLE_ALLOC_TYPES=cuda
UCX_RNDV_MEMTYPE_DIRECT_SIZE=inf
UCX_BCOPY_THRESH=auto
UCX_RNDV_THRESH=intra:8K,inter:8K
UCX_RNDV_SEND_NBR_THRESH=256K
UCX_RNDV_THRESH_FALLBACK=inf
UCX_RNDV_PERF_DIFF=1.000
UCX_MULTI_LANE_MAX_RATIO=4.000
UCX_MULTI_PATH_RATIO=auto
UCX_MAX_EAGER_RAILS=1
UCX_MAX_RNDV_RAILS=2
UCX_MAX_RMA_RAILS=1
UCX_MIN_RNDV_CHUNK_SIZE=16K
UCX_RMA_ZCOPY_MAX_SEG_SIZE=auto
UCX_RNDV_SCHEME=auto
UCX_RKEY_PTR_SEG_SIZE=512K
UCX_ZCOPY_THRESH=auto
UCX_BCOPY_BW=auto
UCX_ATOMIC_MODE=guess
UCX_ADDRESS_DEBUG_INFO=n
UCX_MAX_WORKER_ADDRESS_NAME=32
UCX_USE_MT_MUTEX=n
UCX_ADAPTIVE_PROGRESS=y
UCX_SEG_SIZE=8K
UCX_TM_THRESH=1K
UCX_TM_MAX_BB_SIZE=1K
UCX_TM_FORCE_THRESH=8K
UCX_TM_SW_RNDV=n
UCX_NUM_EPS=auto
UCX_NUM_PPN=auto
UCX_RNDV_FRAG_MEM_TYPES=host,cuda
UCX_RNDV_PIPELINE_SEND_THRESH=inf
UCX_RNDV_PIPELINE_SHM_ENABLE=y
UCX_RNDV_PIPELINE_ERROR_HANDLING=n
UCX_FLUSH_WORKER_EPS=y
UCX_FENCE_MODE=auto
UCX_UNIFIED_MODE=n
UCX_CM_USE_ALL_DEVICES=y
UCX_LISTENER_BACKLOG=auto
UCX_PROTO_ENABLE=y
UCX_PROTO_REQUEST_RESET=n
UCX_KEEPALIVE_INTERVAL=20000000.00us
UCX_KEEPALIVE_NUM_EPS=128
UCX_DYNAMIC_TL_SWITCH_INTERVAL=inf
UCX_DYNAMIC_TL_PROGRESS_FACTOR=88537.01us
UCX_RESOLVE_REMOTE_EP_ID=off
UCX_PROTO_INDIRECT_ID=auto
UCX_RNDV_PUT_FORCE_FLUSH=n
UCX_SA_DATA_VERSION=v2
UCX_RKEY_MPOOL_MAX_MD=2
UCX_ADDRESS_VERSION=v1
UCX_PROTO_INFO=n
UCX_RNDV_ALIGN_THRESH=64K
UCX_PROTO_INFO_DIR=
UCX_REG_NONBLOCK_MEM_TYPES=
UCX_PREFER_OFFLOAD=y
UCX_PROTO_OVERHEAD=single:0.01us,multi:0.01us,rndv_offload:0.04us,rndv_rtr:0.04us,rndv_rts:0.28us,sw:0.04us,rkey_ptr:0.00us
UCX_GVA_ENABLE=off
UCX_GVA_MLOCK=y
UCX_GVA_PREFETCH=y
[1750326976.161268] [prateek:16802:0]     ucp_context.c:2257 UCX  INFO  Version 1.18.1 (loaded from /lib/libucp.so.0)
[1750326976.161301] [prateek:16802:0]     ucp_context.c:2016 UCX  DEBUG estimated number of endpoints is 1
[1750326976.161304] [prateek:16802:0]     ucp_context.c:2023 UCX  DEBUG estimated number of endpoints per node is 1
[1750326976.161307] [prateek:16802:0]     ucp_context.c:2033 UCX  DEBUG estimated bcopy bandwidth is 7340032000.000000
[1750326976.161316] [prateek:16802:0]     ucp_context.c:2092 UCX  DEBUG allocation method[0] is md 'sysv'
[1750326976.161318] [prateek:16802:0]     ucp_context.c:2092 UCX  DEBUG allocation method[1] is md 'posix'
[1750326976.161321] [prateek:16802:0]     ucp_context.c:2104 UCX  DEBUG allocation method[2] is 'thp'
[1750326976.161324] [prateek:16802:0]     ucp_context.c:2092 UCX  DEBUG allocation method[3] is md '*'
[1750326976.161326] [prateek:16802:0]     ucp_context.c:2104 UCX  DEBUG allocation method[4] is 'mmap'
[1750326976.161328] [prateek:16802:0]     ucp_context.c:2104 UCX  DEBUG allocation method[5] is 'heap'
[1750326976.161340] [prateek:16802:0]          module.c:280  UCX  DEBUG loading modules for uct
[1750326976.161522] [prateek:16802:0]          module.c:259  UCX  DEBUG dlopen('/lib/ucx/libuct_cuda.so.0', mode=0x1) failed: libcuda.so.1: cannot open shared object file: No such file or directory
[1750326976.161616] [prateek:16802:0]          module.c:259  UCX  DEBUG dlopen('/usr/lib/ucx/libuct_cuda.so.0', mode=0x1) failed: libcuda.so.1: cannot open shared object file: No such file or directory
[1750326976.162045] [prateek:16802:0]          module.c:280  UCX  DEBUG loading modules for uct_ib
[1750326976.164094] [prateek:16802:0]     ucp_context.c:1648 UCX  DEBUG closing md self because it has no selected transport resources
[1750326976.164453] [prateek:16802:0]            topo.c:895  UCX  DEBUG /sys/class/net/lo: sysfs path undetected
[1750326976.164465] [prateek:16802:0]            topo.c:556  UCX  DEBUG lo: system device unknown
[1750326976.164613] [prateek:16802:0]            topo.c:899  UCX  DEBUG /sys/class/net/wlp2s0: PF sysfs path is '/sys/devices/pci0000:00/0000:00:1c.3/0000:02:00.0'
[1750326976.164666] [prateek:16802:0]            numa.c:155  UCX  DEBUG failed to discover numa node for device: /sys/devices/pci0000:00/0000:00:1c.3/0000:02:00.0, status Success,                   parsed_node -1
[1750326976.164672] [prateek:16802:0]            topo.c:298  UCX  DEBUG added sys_dev 0 for bus id 02:00.0
[1750326976.164677] [prateek:16802:0]            topo.c:552  UCX  DEBUG wlp2s0: bdf_name 0000:02:00.0 sys_dev 0
[1750326976.164703] [prateek:16802:0]     ucp_context.c:1648 UCX  DEBUG closing md tcp because it has no selected transport resources
[1750326976.165007] [prateek:16802:0]    ib_mlx5dv_md.c:2502 UCX  DEBUG rxe_wlp2s0: DEVX is not supported
[1750326976.165016] [prateek:16802:0]           ib_md.c:1034 UCX  DEBUG rxe_wlp2s0: md open by 'uct_ib_mlx5_devx_md_ops' failed, trying next
[1750326976.165021] [prateek:16802:0]           ib_md.c:1034 UCX  DEBUG rxe_wlp2s0: md open by 'uct_ib_mlx5_md_ops' failed, trying next
[1750326976.165096] [prateek:16802:0]            topo.c:895  UCX  DEBUG /sys/class/infiniband/rxe_wlp2s0: sysfs path undetected
[1750326976.165102] [prateek:16802:0]            topo.c:556  UCX  DEBUG rxe_wlp2s0: system device unknown
[1750326976.165108] [prateek:16802:0]       ib_device.c:503  UCX  DEBUG rxe_wlp2s0: vendor_id 0x0 device_id 0
[1750326976.165114] [prateek:16802:0]            topo.c:848  UCX  DEBUG rxe_wlp2s0: pci bandwidth undetected, using maximal value
[1750326976.165129] [prateek:16802:0]             sys.c:433  UCX  DEBUG failed to open /sys/class/infiniband/rxe_wlp2s0/device/local_cpus: No such file or directory
[1750326976.165148] [prateek:16802:0]           async.c:248  UCX  DEBUG added async handler 0x58e7546c6220 [id=4 ref 1] ???() to hash
[1750326976.165249] [prateek:16802:0]           async.c:522  UCX  DEBUG listening to async event fd 4 events 0x1 mode thread_spinlock
[1750326976.165262] [prateek:16802:0]       ib_device.c:602  UCX  DEBUG initialized device 'rxe_wlp2s0' (InfiniBand channel adapter) with 1 ports
[1750326976.165274] [prateek:16802:0]           ib_md.c:1184 UCX  DEBUG rxe_wlp2s0: cuda GPUDirect RDMA is not detected by checking /sys/kernel/mm/memory_peers/nv_mem/version
[1750326976.165283] [prateek:16802:0]           ib_md.c:1184 UCX  DEBUG rxe_wlp2s0: cuda GPUDirect RDMA is not detected by checking /sys/module/nvidia_peermem/version
[1750326976.165290] [prateek:16802:0]           ib_md.c:1184 UCX  DEBUG rxe_wlp2s0: cuda GPUDirect RDMA is not detected by checking /sys/module/nv_peer_mem/version
[1750326976.165299] [prateek:16802:0]           ib_md.c:1184 UCX  DEBUG rxe_wlp2s0: rocm GPUDirect RDMA is not detected by checking /dev/kfd
[1750326976.165304] [prateek:16802:0]           ib_md.c:1168 UCX  DEBUG rxe_wlp2s0: relaxed order memory access is disabled
[1750326976.165310] [prateek:16802:0]           ib_md.c:1236 UCX  DEBUG rxe_wlp2s0: ODP is disabled because device does not support IBV_ACCESS_ON_DEMAND
[1750326976.165315] [prateek:16802:0]           ib_md.c:1038 UCX  DEBUG rxe_wlp2s0: md open by 'uct_ib_verbs_md_ops' is successful
[1750326976.165475] [prateek:16802:0]          uct_md.c:97   UCX  DEBUG failed to query dc_mlx5 resources: No such device
[1750326976.165486] [prateek:16802:0]          uct_md.c:97   UCX  DEBUG failed to query rc_mlx5 resources: No such device
[1750326976.165491] [prateek:16802:0]          uct_md.c:97   UCX  DEBUG failed to query ud_mlx5 resources: No such device
[1750326976.165513] [prateek:16802:0]     ucp_context.c:1648 UCX  DEBUG closing md rxe_wlp2s0 because it has no selected transport resources
[1750326976.165523] [prateek:16802:0]       ib_device.c:621  UCX  DEBUG destroying ib device rxe_wlp2s0
[1750326976.165532] [prateek:16802:0]           async.c:173  UCX  DEBUG removed async handler 0x58e7546c6220 [id=4 ref 1] ???() from hash
[1750326976.165538] [prateek:16802:0]           async.c:576  UCX  DEBUG removing async handler 0x58e7546c6220 [id=4 ref 1] ???()
[1750326976.165596] [prateek:16802:0]           async.c:188  UCX  DEBUG release async handler 0x58e7546c6220 [id=4 ref 0] ???()
[1750326976.165667] [prateek:16802:0]     ucp_context.c:1713 UCX  DEBUG register host memory on: cma
[1750326976.165672] [prateek:16802:0]     ucp_context.c:1701 UCX  DEBUG no memory domain supports registering cuda memory
[1750326976.165678] [prateek:16802:0]     ucp_context.c:1701 UCX  DEBUG no memory domain supports registering cuda-managed memory
[1750326976.165683] [prateek:16802:0]     ucp_context.c:1701 UCX  DEBUG no memory domain supports registering rocm memory
[1750326976.165688] [prateek:16802:0]     ucp_context.c:1701 UCX  DEBUG no memory domain supports registering rocm-managed memory
[1750326976.165693] [prateek:16802:0]     ucp_context.c:1701 UCX  DEBUG no memory domain supports registering rdma memory
[1750326976.165698] [prateek:16802:0]     ucp_context.c:1701 UCX  DEBUG no memory domain supports registering ze-host memory
[1750326976.165702] [prateek:16802:0]     ucp_context.c:1701 UCX  DEBUG no memory domain supports registering ze-device memory
[1750326976.165707] [prateek:16802:0]     ucp_context.c:1701 UCX  DEBUG no memory domain supports registering ze-managed memory
[1750326976.165735] [prateek:16802:0]           mpool.c:138  UCX  DEBUG mpool rcache_mp: align 8, maxelems 4294967295, elemsize 144
[1750326976.165760] [prateek:16802:0]           async.c:248  UCX  DEBUG added async handler 0x58e7546d04f0 [id=3 ref 1] ???() to hash
[1750326976.165817] [prateek:16802:0]           async.c:522  UCX  DEBUG listening to async event fd 3 events 0x1 mode thread_spinlock
[1750326976.166018] [prateek:16802:0]          module.c:280  UCX  DEBUG loading modules for ucm
[1750326976.166156] [prateek:16802:0]          module.c:259  UCX  DEBUG dlopen('/lib/ucx/libucm_cuda.so.0', mode=0x1001) failed: libcuda.so.1: cannot open shared object file: No such file or directory
[1750326976.166245] [prateek:16802:0]          module.c:259  UCX  DEBUG dlopen('/usr/lib/ucx/libucm_cuda.so.0', mode=0x1001) failed: libcuda.so.1: cannot open shared object file: No such file or directory
[1750326976.166265] [prateek:16802:0]     ucp_context.c:2332 UCX  DEBUG created ucp context ucp_server 0x58e7546caae0 [3 mds 3 tls] features 0xe3 tl bitmap 0x7 0x0
[1750326976.166376] [prateek:16802:0]         mm_sysv.c:97   UCX  DEBUG   mm failed to allocate 33023 bytes with hugetlb
[1750326976.166423] [prateek:16802:0]           mpool.c:138  UCX  DEBUG mpool mm_recv_desc: align 64, maxelems 4294967295, elemsize 8368
[1750326976.166436] [prateek:16802:0]         mm_sysv.c:97   UCX  DEBUG   mm failed to allocate 4292720 bytes with hugetlb
[1750326976.166454] [prateek:16802:0]           mpool.c:282  UCX  DEBUG mpool mm_recv_desc: allocated chunk 0x7107217e7018 of 4296680 bytes with 512 elements
[1750326976.168632] [prateek:16802:0]        mm_iface.c:733  UCX  DEBUG created mm iface 0x58e7546c9e80 FIFO id 0x18003 va 0x71072b75a000 size 36864 (128 x 256 elems)
[1750326976.168710] [prateek:16802:0]      ucp_worker.c:1436 UCX  DEBUG created interface[0]=0x58e7546c9e80 using sysv/memory on worker 0x71072b7a0010
[1750326976.169041] [prateek:16802:0]        mm_posix.c:573  UCX  DEBUG   allocated posix shared memory at 0x71072b72c000 length 36864
[1750326976.169077] [prateek:16802:0]           mpool.c:138  UCX  DEBUG mpool mm_recv_desc: align 64, maxelems 4294967295, elemsize 8368
[1750326976.170853] [prateek:16802:0]        mm_posix.c:347  UCX  DEBUG   shared memory mmap(addr=(nil), length=6291456, flags= HUGETLB, fd=11) failed: Invalid argument
[1750326976.170881] [prateek:16802:0]        mm_posix.c:573  UCX  DEBUG   allocated posix shared memory at 0x7107213ce000 length 4296704
[1750326976.170900] [prateek:16802:0]           mpool.c:282  UCX  DEBUG mpool mm_recv_desc: allocated chunk 0x7107213ce018 of 4296680 bytes with 512 elements
[1750326976.172161] [prateek:16802:0]        mm_iface.c:733  UCX  DEBUG created mm iface 0x58e7546c8690 FIFO id 0xc0000002400041a2 va 0x71072b72c000 size 36864 (128 x 256 elems)
[1750326976.172205] [prateek:16802:0]      ucp_worker.c:1436 UCX  DEBUG created interface[1]=0x58e7546c8690 using posix/memory on worker 0x71072b7a0010
[1750326976.172262] [prateek:16802:0]           mpool.c:138  UCX  DEBUG mpool uct_scopy_iface_tx_mp: align 64, maxelems 4294967295, elemsize 736
[1750326976.172279] [prateek:16802:0]      ucp_worker.c:1436 UCX  DEBUG created interface[2]=0x58e7546e22b0 using cma/memory on worker 0x71072b7a0010
[1750326976.172285] [prateek:16802:0]      ucp_worker.c:1153 UCX  DEBUG selected scalable tl bitmap: 0x7 0x0 (3 tls)
[1750326976.172307] [prateek:16802:0]           async.c:248  UCX  DEBUG added async handler 0x58e7546c6220 [id=8 ref 1] ???() to hash
[1750326976.172329] [prateek:16802:0]           async.c:522  UCX  DEBUG listening to async event fd 8 events 0x0 mode thread_spinlock
[1750326976.172374] [prateek:16802:0]           async.c:248  UCX  DEBUG added async handler 0x58e7546dd3f0 [id=10 ref 1] ???() to hash
[1750326976.172387] [prateek:16802:0]           async.c:522  UCX  DEBUG listening to async event fd 10 events 0x0 mode thread_spinlock
[1750326976.172613] [prateek:16802:0]           async.c:248  UCX  DEBUG added async handler 0x58e7546e0c50 [id=12 ref 1] ???() to hash
[1750326976.172635] [prateek:16802:0]           async.c:522  UCX  DEBUG listening to async event fd 12 events 0x1 mode thread_spinlock
[1750326976.172644] [prateek:16802:0]       rdmacm_cm.c:982  UCX  DEBUG created rdmacm_cm 0x58e7546e2800 with event_channel 0x58e7546d1230 (fd=12)
[1750326976.172663] [prateek:16802:0]      tcp_sockcm.c:225  UCX  DEBUG created tcp_sockcm 0x58e7546e50d0
[1750326976.172675] [prateek:16802:0]           mpool.c:138  UCX  DEBUG mpool ucp_requests: align 64, maxelems 4294967295, elemsize 272
[1750326976.172682] [prateek:16802:0]           mpool.c:138  UCX  DEBUG mpool ucp_rkeys: align 64, maxelems 4294967295, elemsize 104
[1750326976.172687] [prateek:16802:0]           mpool.c:138  UCX  DEBUG mpool ucp_reg_bufs: align 64, maxelems 4294967295, elemsize 8216
[1750326976.172695] [prateek:16802:0]           mpool.c:138  UCX  DEBUG mpool ucp_am_bufs: align 64, maxelems 4294967295, elemsize 153
[1750326976.172700] [prateek:16802:0]           mpool.c:138  UCX  DEBUG mpool ucp_am_bufs: align 64, maxelems 4294967295, elemsize 1113
[1750326976.172706] [prateek:16802:0]           mpool.c:138  UCX  DEBUG mpool ucp_am_bufs: align 64, maxelems 4294967295, elemsize 8345
[1750326976.172712] [prateek:16802:0]       mpool_set.c:130  UCX  DEBUG mpool_set:ucp_am_bufs, sizes map 0x80000440, largest size 8256, mpools num 3
[1750326976.172747] [prateek:16802:0]          parser.c:2337 UCX  INFO  UCX_* env variables: UCX_RNDV_THRESH=8192 UCX_TLS=shm UCX_LOG_LEVEL=DEBUG
FILE: /home/prateek/Workspace/UCX/src/common/common.cc FUNCTION : SetSockAddr LINE : 37 message: Server will listen on 192.168.29.202:9090
[1750326976.172940] [prateek:16802:0]          ucp_ep.c:405  UCX  DEBUG created ep 0x71072b70c000 to <no debug data> from api call
[1750326976.172979] [prateek:16802:0]      proto_rndv.c:515  UCX  DEBUG       no active message lane for tag/rndv
[1750326976.173017] [prateek:16802:0]      proto_rndv.c:515  UCX  DEBUG             no active message lane for rndv/rtr
[1750326976.173024] [prateek:16802:0]      proto_rndv.c:515  UCX  DEBUG             no active message lane for rndv/rtr/mtype
[1750326976.173061] [prateek:16802:0]      proto_rndv.c:515  UCX  DEBUG           no active message lane for rndv/ats
[1750326976.173067] [prateek:16802:0]      proto_rndv.c:515  UCX  DEBUG           no active message lane for rndv/rtr
[1750326976.173082] [prateek:16802:0]      proto_rndv.c:515  UCX  DEBUG               no active message lane for rndv/rtr
[1750326976.173087] [prateek:16802:0]      proto_rndv.c:515  UCX  DEBUG               no active message lane for rndv/rtr/mtype
[1750326976.173102] [prateek:16802:0]      proto_rndv.c:515  UCX  DEBUG           no active message lane for rndv/rtr/mtype
[1750326976.173114] [prateek:16802:0]      proto_rndv.c:515  UCX  DEBUG       no active message lane for rndv/rtr
[1750326976.173120] [prateek:16802:0]      proto_rndv.c:515  UCX  DEBUG       no active message lane for rndv/rtr/mtype
[1750326976.173138] [prateek:16802:0]      proto_rndv.c:515  UCX  DEBUG       no active message lane for rndv/rtr
[1750326976.173143] [prateek:16802:0]      proto_rndv.c:515  UCX  DEBUG       no active message lane for rndv/rtr/mtype
[1750326976.173154] [prateek:16802:0]      proto_rndv.c:515  UCX  DEBUG       no active message lane for am/rndv
[1750326976.173169] [prateek:16802:0]      proto_rndv.c:515  UCX  DEBUG       no active message lane for rndv/rtr
[1750326976.173175] [prateek:16802:0]      proto_rndv.c:515  UCX  DEBUG       no active message lane for rndv/rtr/mtype
[1750326976.173186] [prateek:16802:0]      proto_rndv.c:515  UCX  DEBUG       no active message lane for am/rndv
[1750326976.173243] [prateek:16802:0]    rdmacm_cm_ep.c:808  UCX  DEBUG [cep 0x58e7546f25c0 <invalid>->192.168.29.202:9090 client Success] created an endpoint on rdmacm 0x58e7546e2800 id: 0x58e7546f2620
[1750326976.173257] [prateek:16802:0]       wireup_ep.c:551  UCX  DEBUG ep 0x71072b70c000: wireup_ep 0x58e7546f22d0 set next_ep 0x58e7546f25c0
FILE: /home/prateek/Workspace/UCX/src/client/client.cc FUNCTION : Client LINE : 13 message: Client created with server address: 192.168.29.202 and port: 9090
FILE: /home/prateek/Workspace/UCX/examples/AM/client.cc FUNCTION : main LINE : 41 message: Iteration : 1 of 100
FILE: /home/prateek/Workspace/UCX/src/client/client.cc FUNCTION : SendAcvtiveMsg LINE : 125 message: ENTRY
[1750326976.173493] [prateek:16802:0]           mpool.c:282  UCX  DEBUG mpool ucp_requests: allocated chunk 0x58e7546f2b94 of 41044 bytes with 128 elements
[1750326976.173521] [prateek:16802:a]       wireup_cm.c:602  UCX  DIAG  client ep 0x71072b70c000 connect to 192.168.29.202:9090 failed: device rxe_wlp2s0:1 is not enabled, enable it in UCX_NET_DEVICES or use corresponding ip address
[1750326976.173542] [prateek:16802:a]          uct_cm.c:98   UCX  DIAG  resolve callback failed with error: Destination is unreachable
[1750326976.173583] [prateek:16802:0]       wireup_cm.c:78   UCX  DIAG  ep 0x71072b70c000: client switching from rdmacm to tcp in attempt to connect to the server
[1750326976.173692] [prateek:16802:0]            sock.c:347  UCX  DEBUG connect(fd=15, src_addr=192.168.29.202:46998 dest_addr=192.168.29.202:9090): Operation now in progress
[1750326976.173711] [prateek:16802:0]           async.c:248  UCX  DEBUG added async handler 0x58e7546fd490 [id=15 ref 1] uct_tcp_sa_data_handler() to hash
[1750326976.173749] [prateek:16802:0]           async.c:522  UCX  DEBUG listening to async event fd 15 events 0x2 mode thread_spinlock
[1750326976.173758] [prateek:16802:0]   tcp_sockcm_ep.c:925  UCX  DEBUG created a TCP SOCKCM endpoint (fd=15) on tcp cm 0x58e7546e50d0, remote addr: 192.168.29.202:9090
[1750326976.173764] [prateek:16802:0]   tcp_sockcm_ep.c:1128 UCX  DEBUG client created an endpoint on tcp_sockcm 0x58e7546e50d0 id: 15 state: 2
[1750326976.173769] [prateek:16802:0]       wireup_ep.c:551  UCX  DEBUG ep 0x71072b70c000: wireup_ep 0x58e7546f22d0 set next_ep 0x58e7546fcbf0
[1750326976.173781] [prateek:16802:0]            sock.c:1077 UCX  DEBUG check ifname for socket on 192.168.29.202:0
[1750326976.173886] [prateek:16802:0]            sock.c:1094 UCX  DEBUG matching ip found iface on wlp2s0
[1750326976.173896] [prateek:16802:0]       wireup_cm.c:602  UCX  DIAG  client ep 0x71072b70c000 connect to 192.168.29.202:9090 failed: device wlp2s0 is not enabled, enable it in UCX_NET_DEVICES or use corresponding ip address
[1750326976.173904] [prateek:16802:0]       wireup_cm.c:133  UCX  DIAG  client ep 0x71072b70c000 failed to connect to 192.168.29.202:9090 using rdmacm,tcp cms
[1750326976.173910] [prateek:16802:0]          uct_cm.c:98   UCX  DIAG  resolve callback failed with error: Destination is unreachable
[1750326976.173918] [prateek:16802:0]           async.c:173  UCX  DEBUG removed async handler 0x58e7546fd490 [id=15 ref 2] uct_tcp_sa_data_handler() from hash
[1750326976.173923] [prateek:16802:0]           async.c:576  UCX  DEBUG removing async handler 0x58e7546fd490 [id=15 ref 2] uct_tcp_sa_data_handler()
[1750326976.173931] [prateek:16802:0]           async.c:188  UCX  DEBUG release async handler 0x58e7546fd490 [id=15 ref 0] uct_tcp_sa_data_handler()
[1750326976.173939] [prateek:16802:0]          ucp_ep.c:1476 UCX  DEBUG ep 0x71072b70c000: set_ep_failed status Destination is unreachable on lane[0]=0x58e7546f22d0
[1750326976.173944] [prateek:16802:0]          ucp_ep.c:1437 UCX  DEBUG ep 0x71072b70c000: discarding lanes
[1750326976.173951] [prateek:16802:0]          ucp_ep.c:1445 UCX  DEBUG ep 0x71072b70c000: discard uct_ep[0]=0x58e7546f22d0
[1750326976.173957] [prateek:16802:0]    proto_common.c:825  UCX  DEBUG abort request 0x58e7546fca80 proto reconfig status Destination is unreachable
[1750326976.173965] [prateek:16802:0]       wireup_ep.c:440  UCX  DEBUG ep 0x71072b70c000: destroy wireup ep 0x58e7546f22d0
[1750326976.173973] [prateek:16802:0]          ucp_ep.c:3507 UCX  DEBUG ep 0x71072b70c000: calling user error callback 0x58e75086e98c with arg 0x7ffd749d62a0 and status Destination is unreachable
FILE: /home/prateek/Workspace/UCX/src/client/client.cc FUNCTION : ErrorHandlingCb LINE : 100 message: Error handling callback invoked with status: Destination is unreachable
terminate called after throwing an instance of 'std::runtime_error'
  what():  Error handling callback invoked with status: Destination is unreachable
Aborted (core dumped)