How to bind created threads to specific CPU cores? just like pthread_setaffinity_np

[phil-skillwon]

Our application is migrating from pthreads to HPX, but some thread functions need to be manually bound to specific CPU cores, i.e., setting the thread's CPU affinity. However, I haven't found similar usage examples in HPX documentation. How can I bind the created threads to specific CPU cores in HPX? just like pthread_setaffinity_np

[Pansysk75]

@phil-skillwon There might be better ways to achieve what you want, depending on why you need CPU affinity in the first place, but I recently had to do something similar so here's one mechanism related to what you need:

The thread_schedule_hint can be used to schedule an hpx thread on a specific queue in the hpx scheduler (each queue is associated with a single physical core).

If you are using hpx::async to launch tasks, you can pass an executor with an appropriate thread_schedule_hint:

hpx/libs/core/resource_partitioner/tests/unit/scheduler_binding_check.cpp

Lines 64 to 75 in d08354d

	// launch tasks on threads using numbering 0,1,2,3...0,1,2,3
	std::size_t threads = hpx::get_num_worker_threads();
	for (std::size_t i = 0; i < iterations; ++i)
	{
	auto exec = hpx::execution::parallel_executor(
	hpx::threads::thread_priority::bound,
	hpx::threads::thread_stacksize::default_,
	hpx::threads::thread_schedule_hint(
	hpx::threads::thread_schedule_hint_mode::thread,
	std::int16_t(i % threads)));
	hpx::async(exec, f, i, (i % threads)).get();
	}

(or decorate an existing executor using hpx::execution::experimental::with_hint)

If not using hpx::async, the thread_schedule_hint can also be used to initialize a hpx::threads::thread_init_data object and then use hpx::threads::register_thread to schedule the hpx thread:

hpx/examples/throttle/throttle/server/throttle.cpp

Lines 121 to 127 in d08354d

	hpx::threads::thread_init_data data(
	hpx::threads::make_thread_function_nullary(
	hpx::bind(&throttle::throttle_controller, this, shepherd)),
	description.c_str(), hpx::threads::thread_priority::high,
	hpx::threads::thread_schedule_hint(
	static_cast<std::uint16_t>(shepherd)));
	hpx::threads::register_thread(data);

Using hpx::threads::thread_priority::bound will also place the hpx thread in special queues, ensuring it cannot be stolen by other worker-threads.

Also, in contrast to pthread_setaffinity_np, the schedule hint method must be used at the time of creating/scheduling, so it cannot help with an hpx thread that has already been scheduled.

Finally, there are also thread pools and the resource partitioner, which can be used to control which hardware resources you are using, but I'm not too familiar with those.

[phil-skillwon]

I am currently doing it like this:

int hpx_main(int argc, char** argv)
{
    TRACE("hpx_main function");

    std::size_t numCores = hpx::threads::hardware_concurrency();
    DEBUG("number of cores: %d", numCores);
    std::vector<hpx::future<void>> futures;

    hpx::threads::topology& topo = hpx::threads::create_topology();
    for (int i = 0; i < 5; ++i) {
        futures.push_back(hpx::async([i, numCores, &topo]() {
            hpx::threads::mask_type mask = hpx::threads::mask_type();
            hpx::threads::resize(mask, numCores);
            size_t coreIndex = i % numCores;
            hpx::threads::set(mask, coreIndex);
            topo.set_thread_affinity_mask(mask);

            computeTask(i);
        }));
    }

    hpx::wait_all(futures);

    return hpx::finalize();
}

These codes run fine, but I am worried there might be other issues.

[hkaiser]

This will set the affinity for the underlying kernel thread, I don't think messing with this would be a good idea as HPX maintains that by itself. You want to change the affinity of the tasks that you schedule on top of HPX, do you?

[phil-skillwon]

This will set the affinity for the underlying kernel thread, I don't think messing with this would be a good idea as HPX maintains that by itself. You want to change the affinity of the tasks that you schedule on top of HPX, do you?

Yes, my requirement is to allocate 10 compute-intensive tasks (which are created by calling HPX's API) to run on specified CPU cores. @Pansysk75 As Mr. Pansysk75 mentioned earlier, I tested using hpx::execution::parallel_executor, and it fully met my needs. However, this also made me a bit confused. I want to learn HPX, but after reading the example codes in HPX examples, I am even more confused. At least for me, it is not a smooth learning path. I then read the HPX API documentation, and although the APIs are detailed, I still don't have a systematic understanding. Do you have any better learning suggestions for this?

[hkaiser]

Thanks for your feedback, we're always in search for a better way to document HPX. Would you be able to ask concrete questions regarding the unclear aspects? We will try to answer those.

[hkaiser]

As a side note, @Pansysk75 and I discussed yesterday how we could implement an API similar to pthread_setaffinity_np, just for HPX tasks; i.e. enabling changing the affinity after an HPX thread has been created. I have created a feature request to track the progress on this here: #6506.

[phil-skillwon]

Thank you very much! If HPX can achieve something similar to pthread_setaffinity_np, that would be really cool.