Exit worker threads early when there are no tasks after a while

[hai-rise]

No description provided.

[hai-rise]

@byhashdong Here it is, sir!

[hai-rise]

after trying rayon, get same result, the performance is worse than original thread pool implementation

Hopefully one day we understand why!

having a little test, found time gap between all threads completed and scoped completed normally less than 100us.

Ah yes, the scope should end immediately when all threads join. The original idea is more about starting to post-process when Scheduler announces that execution has been done, so even before all threads have completed joining!

confused, while task.is_some() { as the code, thread should exit if there is no new work to do.

While looking for a Scheduler::next_task, we currently loop until execution is done or aborted:

pevm/src/scheduler.rs

Lines 118 to 119 in bed6f22

	thread::yield_now();
	continue;

We can add a counter here to return a None to join a thread early when it fails to find a job after X iterations.

[hai-rise]

@byhashdong Yes, something like that.

FYI, yield_now was better than spin_loop and much better than not doing either (aggressive loop that may cause congestion at the atomics within) on my x86, but it didn't yield much difference on ARM. Hopefully, we will know the optimal strategy for each input block, and how to play nice with the OS scheduler!

[hai-rise]

@byhashdong Is s sequential? If so it should be unaffected by the counter, right?

I've also played with a few numbers and cannot find a sweet spot. There are always strong wins (no late dependencies so exiting workers early removes tail latency) and losses (many dependencies so when we exit threads early we lack workers when we reset the execution index way back early in the block).

I think we should bring more variables to the equation. For instance, to only count empty tasks and exit threads when num_validated + concurrency_level > block_size - min_validation_idx 🤔.

[hai-rise]

@byhashdong For micro-benchmarks like this it's best to close most processes (especially large ones like browsers and code editors with rust-analyzer) to minimize noise. And good point on per-task counter vs. all-task counter. I'll try different formulas next week 🙏.