how to test UCM allocator?

[Honggang-LI]

hi,

In header ucx/src/ucm/api/ucm.h

  typedef enum ucm_event_type {
................
      /* Non-accessible memory alloc/free events */
      UCM_EVENT_MEM_TYPE_ALLOC  = UCS_BIT(20),
      UCM_EVENT_MEM_TYPE_FREE   = UCS_BIT(21),

The comment says "it is non-accessible". I have troubles to understand it. But it seems function ucm_event_install never install handler for those two events, as line 525-526 clean up those events. So, my questions are:

1. what does "non-accessible" mean?
2. why ucm_envent_install does not install handler for those events.

And in function

  507 static ucs_status_t ucm_event_install(int events)
  508 {
  509     UCS_MODULE_FRAMEWORK_DECLARE(ucm);
  510     ucm_event_installer_t *event_installer;
  511     int malloc_events;
  512     ucs_status_t status;
  513 
  514     ucm_prevent_dl_unload();
  515 
  516     /* TODO lock */
  517     status = ucm_mmap_install(events, 0);
  518     if (status != UCS_OK) {
  519         ucm_diag("failed to install mmap events");
  520         goto out_unlock;
  521     }
  522 
  523     ucm_debug("mmap hooks are ready");
  524 
  525     malloc_events = events & ~(UCM_EVENT_MEM_TYPE_ALLOC |
  526                                UCM_EVENT_MEM_TYPE_FREE);
  527     status = ucm_malloc_install(malloc_events);

I want to run code coverage (gcc --coverage) for UCM module. Which tool (ucx_perftest?) can be used to trigger code in directory src/ucm/ptmalloc286/?

Thanks

[yosefe]

1. Non-accessible means the memory being allocated/released is not directly accessible (for example it's GPU memory)
2. It happens here:

   ucx/src/ucm/event/event.c

   Line 538 in 7db0646

   status = event_installer->install(events);

3. src/ucm/ptmalloc286/ is not being used by default. Can force using it by setting "UCX_MEM_MMAP_HOOK_MODE=reloc" and run ucx_perftest of ucx_info. For example:
   UCX_MEM_MMAP_HOOK_MODE=reloc ucx_info -u t -p

[Honggang-LI]

1. Non-accessible means the memory being allocated/released is not directly accessible (for example it's GPU memory)

I don't have GPU device to test this. How can I test it with IB device?

2. It happens here: https://github.com/openucx/ucx/blob/7db064659958f678199d2c04952ead746180cc8a/src/ucm/event/event.c#L538

I tried to search keyword ucm_event_installer_list in ucx repo. But I did not find any line insert something in this list. That means line 538 never be executed. I confirmed it with code coverage.

     533           1 :     ucm_debug("malloc hooks are ready");
     534             : 
     535             :     /* Call extra event installers */
     536           1 :     UCS_MODULE_FRAMEWORK_LOAD(ucm, UCS_MODULE_LOAD_FLAG_NODELETE);
     537           1 :     ucs_list_for_each(event_installer, &ucm_event_installer_list, list) {
     538           0 :         status = event_installer->install(events);
     539           0 :         if (status != UCS_OK) {
     540           0 :             goto out_unlock;
     541             :         }
     542             :     }
     543             : 
     544           1 :     status = UCS_OK;

3. `src/ucm/ptmalloc286/` is not being used by default. Can force using it by setting "UCX_MEM_MMAP_HOOK_MODE=reloc" and run ucx_perftest of ucx_info. For example:
   `UCX_MEM_MMAP_HOOK_MODE=reloc ucx_info -u t -p`

Confirmed some code of src/ucm/ptmalloc286/ had been executed.

[yosefe]

1+2. it's not possible to test this without GPU. The GPU specific modules are the ones adding entries to ucm_event_installer_list

[Honggang-LI]

I use Sourcetrail to browser the ucx code. Sourcetrail built the database with the compilation database which was generated in compiling process. That is why I could not found the GPU modules add entries to ucm_event_installer_list. As no GPU device on my machine, so the GPU modules were not enabled/compiled when built ucx.

[yosefe]

You should still be able to compile Cuda modules by installing Cuda development libraries on the machine. It will not be used in runtime, though.