Using fork() in a multi-threaded environment can cause deadlocks

[shachafl]

Description

Python 3.12, which has stricter warnings for potentially unsafe operations raises thousands of warnings, when running testings, due to the fact that use of fork() in a multi-threaded environment can cause deadlocks.

This warning is only relevant for Linux as from Python 3.8 the default multiprocessing start method for MacOs and Windows is spawn.
https://docs.python.org/3.12/library/multiprocessing.html
The recommended solution is to change the multiprocessing start method to spawn or forkserver, especially if you are working on macOS or Linux.

• spawn: Starts a fresh interpreter, safer but slightly slower. Best for cross-platform compatibility.
• forkserver: Faster and avoids issues with fork in multithreaded applications.

From initial testing the difference is testing time when running make fast-test is huge: fork takes 48 [sec], spawn takes 3089 [sec], forkserver takes 1747 [sec].

I have also tried to replace multiprocessing with Joblib:
output_imagestacks = Parallel(n_jobs=2)(delayed(filter_fn)(imagestack) for imagestack in all_imagestacks)
But got PicklingError and put it on hold for now.

Steps/Code to Reproduce

python -m venv .venv3.12
source .venv3.12/bin/activate
make install-dev
make fast-test

Expected Results

No warnings.

Actual Results

starfish/core/spots/DecodeSpots/test/test_check_all.py: 2516 warnings
./.pyenv/versions/3.12.7/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=1511822) is multi-threaded, use of fork() may lead to deadlocks in the child.
self.pid = os.fork()

starfish/core/test/test_multiprocessing_workflow.py::test_multiprocessing_workflow
./.pyenv/versions/3.12.7/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=1511819) is multi-threaded, use of fork() may lead to deadlocks in the child.
self.pid = os.fork()

Versions

Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.39
Python 3.12.7 (main, Oct 11 2024, 10:07:41) [GCC 13.2.0]
NumPy 1.26.4
SciPy 1.13.1
scikit-image 0.24.0
pandas 2.2.3
sklearn 1.6.1
xarray 2023.8.0
sympy 1.13.3
starfish 0.3.2+2.gf311a13d