Fix tests on Linux when psutil is installed

[lesteve]

This fixes the error at the bottom of this comment when psutil is installed on a Linux machine.

delattr seems the right approach to make sure that psutil.Process.cpu_affinity triggers AttributeError and we end up in the branch where psutil.cpu_count is used in the following code:

pytest-run-parallel/src/pytest_run_parallel/utils.py

Lines 196 to 232 in 8699aaa

	def get_logical_cpus():
	try:
	import psutil
	except ImportError:
	pass
	else:
	process = psutil.Process()
	try:
	cpu_cores = process.cpu_affinity()
	if cpu_cores is not None:
	return len(cpu_cores)
	except AttributeError:
	cpu_cores = psutil.cpu_count()
	if cpu_cores is not None:
	return cpu_cores
	try:
	from os import process_cpu_count
	except ImportError:
	pass
	else:
	cpu_cores = process_cpu_count()
	if cpu_cores is not None:
	return cpu_cores
	try:
	from os import sched_getaffinity
	except ImportError:
	pass
	else:
	cpu_cores = sched_getaffinity(0)
	if cpu_cores is not None:
	return len(cpu_cores)
	from os import cpu_count
	return cpu_count()

Note that on macOS psutil.Process has no attribute cpu_affinity so maybe someone with a macOS machine wants to double-check and run the tests.

By the way, I don't think any of the CI build has psutil installed, I had a look at the logs and the psutil-related tests are skipped.

____________________________________________________________________________________________________ test_auto_detect_cpus_psutil_cpu_count ____________________________________________________________________________________________________

pytester = <Pytester PosixPath('/tmp/pytest-of-lesteve/pytest-45/test_auto_detect_cpus_psutil_cpu_count0')>, monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x2162c9123f0>

    @pytest.mark.skipif(psutil is None, reason="psutil needs to be installed")
    def test_auto_detect_cpus_psutil_cpu_count(
        pytester: pytest.Pytester, monkeypatch: pytest.MonkeyPatch
    ) -> None:
        import psutil
    
        monkeypatch.setattr(
            psutil.Process, "cpu_affinity", lambda self: None, raising=False
        )
        monkeypatch.setattr(psutil, "cpu_count", lambda: 10)
    
        pytester.makepyfile("""
            def test_auto_detect_cpus(num_parallel_threads):
                assert num_parallel_threads == 10
        """)
    
        # run pytest with the following cmd args
        result = pytester.runpytest("--parallel-threads=auto", "-v")
    
        # fnmatch_lines does an assertion internally
>       result.stdout.fnmatch_lines(
            [
                "*::test_auto_detect_cpus PARALLEL PASSED*",
            ]
        )
E       Failed: nomatch: '*::test_auto_detect_cpus PARALLEL PASSED*'
E           and: '============================= test session starts =============================='
E           and: 'platform linux -- Python 3.13.0, pytest-8.3.3, pluggy-1.5.0 -- /home/lesteve/micromamba/envs/py313t/bin/python'
E           and: 'cachedir: .pytest_cache'
E           and: "hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/home/lesteve/dev/pytest-run-parallel/.hypothesis/examples'))"
E           and: 'rootdir: /tmp/pytest-of-lesteve/pytest-45/test_auto_detect_cpus_psutil_cpu_count0'
E           and: 'plugins: repeat-0.9.3, run-parallel-0.4.3.dev0, hypothesis-6.131.16, order-1.3.0'
E           and: 'collecting ... collected 1 item'
E           and: 'Collected 1 items to run in parallel'
E           and: ''
E           and: 'test_auto_detect_cpus_psutil_cpu_count.py::test_auto_detect_cpus PARALLEL FAILED [100%]'
E           and: ''
E           and: '==================================== ERRORS ===================================='
E           and: '____________________ ERROR at call of test_auto_detect_cpus ____________________'
E           and: ''
E           and: 'num_parallel_threads = 12'
E           and: ''
E           and: '    def test_auto_detect_cpus(num_parallel_threads):'
E           and: '>       assert num_parallel_threads == 10'
E           and: 'E       assert 12 == 10'
E           and: ''
E           and: 'test_auto_detect_cpus_psutil_cpu_count.py:2: AssertionError'
E           and: '************************** pytest-run-parallel report **************************'
E           and: 'All tests were run in parallel! 🎉'
E           and: '=========================== short test summary info ============================'
E           and: 'PARALLEL FAILED test_auto_detect_cpus_psutil_cpu_count.py::test_auto_detect_cpus'
E           and: '=============================== 1 error in 0.02s ==============================='
E       remains unmatched: '*::test_auto_detect_cpus PARALLEL PASSED*'

/home/lesteve/dev/pytest-run-parallel/tests/test_run_parallel.py:686: Failed
------------------------------------------------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.13.0, pytest-8.3.3, pluggy-1.5.0 -- /home/lesteve/micromamba/envs/py313t/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/home/lesteve/dev/pytest-run-parallel/.hypothesis/examples'))
rootdir: /tmp/pytest-of-lesteve/pytest-45/test_auto_detect_cpus_psutil_cpu_count0
plugins: repeat-0.9.3, run-parallel-0.4.3.dev0, hypothesis-6.131.16, order-1.3.0
collecting ... collected 1 item
Collected 1 items to run in parallel

test_auto_detect_cpus_psutil_cpu_count.py::test_auto_detect_cpus PARALLEL FAILED [100%]

==================================== ERRORS ====================================
____________________ ERROR at call of test_auto_detect_cpus ____________________

num_parallel_threads = 12

    def test_auto_detect_cpus(num_parallel_threads):
>       assert num_parallel_threads == 10
E       assert 12 == 10

test_auto_detect_cpus_psutil_cpu_count.py:2: AssertionError
************************** pytest-run-parallel report **************************
All tests were run in parallel! 🎉
=========================== short test summary info ============================
PARALLEL FAILED test_auto_detect_cpus_psutil_cpu_count.py::test_auto_detect_cpus
=============================== 1 error in 0.02s ===============================
=========================================================================================================== short test summary info ============================================================================================================
FAILED tests/test_run_parallel.py::test_auto_detect_cpus_psutil_cpu_count - Failed: nomatch: '*::test_auto_detect_cpus PARALLEL PASSED*'
========================================================================================================= 1 failed, 28 passed in 2.70s =========================================================================================================

[ngoldbaum]

Thanks for pointing this out. I'll try to take a look to make sure there's a test run that exercises those tests.

[lysnikolaou]

I cannot reproduce this in a linux container on my macos. I'll have a closer look on my Linux machine once I get home and can run the tests from there.

[lysnikolaou]

I've reproduced this on my Linux machine and the fix looks right! Thanks @lesteve! 🎉

Could you maybe additionally change the rest of the monkeypatch.setattr calls for cpu_affinity to monkeypatch.delattr? Also, it look like cpu_affinity cannot return None, so removing that branch might be a good idea as well.

[lysnikolaou]

LGTM! Thanks @lesteve!

[lesteve]

Thanks!