Question about memory domain rcache lru spin lock

[huzhijiang]

Some of our use cases(tag zcopy) always hit the following spin lock code which seems to be a performance bottleneck:

ucx/src/ucs/memory/rcache.c

Line 377 in 42b9f55

ucs_spin_lock(&rcache->lru.lock);

It seems since the lock is used for portecting per-context memory domain rcache lru, all workers belong to one ucx context will compete with each other on it. So we tried to split ucx context to let each worker has its own one, and the lock can be omit. But we are not sure if it is the right way to do (ucx context and worker will be as 1:1 mapping). Any ideas?

Thanks!

[yosefe]

Some of our use cases(tag zcopy) always hit the following spin lock code which seems to be a performance bottleneck:

ucx/src/ucs/memory/rcache.c

Line 377 in 42b9f55

ucs_spin_lock(&rcache->lru.lock);

It seems since the lock is used for portecting per-context memory domain rcache lru, all workers belong to one ucx context will compete with each other on it. So we tried to split ucx context to let each worker has its own one, and the lock can be omit. But we are not sure if it is the right way to do (ucx context and worker will be as 1:1 mapping). Any ideas?

Thanks!

Can you pls post the backtrace of this lock where you see contention?
Anyway, it should be ok to create multiple contexts, however it may consume more memory since each context will have its own memory regions handles.

[huzhijiang]

Can you pls post the backtrace of this lock where you see contention? Anyway, it should be ok to create multiple contexts, however it may consume more memory since each context will have its own memory regions handles.

Thanks for the info! Below is the backtrace captured by perf. The enviroment is an aarch64 machine, I didn't see this bottleneck on x64 platform.

- 33.83% ucp_tag_send_nbx
   - 33.81% ucp_tag_send_req (inlined)
      - 33.21% ucp_request_send_start
         - ucp_request_send_buffer_reg_lane (inlined)
         - ucp_request_send_buffer_reg_lane_check (inlined) 
            - 33.19% ucp_request_send_buffer_reg (inlined)
               - ucp_request_memory_reg
                  - 33.12% ucp_mem_rereg_mds
                     - 33.08% uct_ib_mem_rcache_reg
                        - ucs_rcache_get
                           - 27.64% ucs_rcache_region_lru_get (inlined)
                              - 27.07% ucs_spin_lock (inlined)
                                 - pthread_spin_lock 
                                      3.54% __aarch64_swp4_acq 
                                      1.94% __aarch64_cas4_acq
                                0.57% ucs_rcache_region_lru_remove (inlined)
                           - 2.62% __pthread_rwlock_unlock
                                __aarch64_cas4_rel
                             0.75% __pthread_rwlock_rdlock
                             0.73% ucs_rcache_region_hold
      - 0.56% ucp_request_send (inlined)
           ucp_request_try_send (inlined)
           ucp_tag_eager_zcopy_multi
           ucp_do_am_zcopy_multi (inlined)

[yosefe]

The backtrace suggest you are using older UCX version which uses protov1 and does not use lock-less rcache APIs. Can you pls try with latest master branch, or v1.16.0 release?

[huzhijiang]

Hi @yosefe .
Can I backport the change to order branch, or is there any config option in the older branch can switch protov1 to lock-less APIs? I can not upgrade whole UCX currently.

[yosefe]

Unfortunately, the change is too big to backport. However if the application is single threaded it should be safe to remove this spin lock as a local patch.

[huzhijiang]

Thanks for the info. @yosefe