race condition in output dir creation

[lgarrison]

Description

While invoking cibuildwheel in parallel (multiple processes, each building one wheel), I encountered an error like:

Traceback (most recent call last):
  File "cibuildwheel/__main__.py", line 403, in build_in_directory
    output_dir.mkdir(parents=True)
  File "/mnt/sw/nix/store/71ksmx7k6xy3v9ksfkv5mp5kxxp64pd6-python-3.10.13-view/lib/python3.10/pathlib.py", line 1175, in mkdir
    self._accessor.mkdir(self, mode)
FileExistsError: [Errno 17] File exists: 'wheelhouse'

I'm embarrassed to say that the logs on the CI server with the exact error message have been purged, so the above is my recreation from memory. Take it with a grain of salt! However, I think the TOCTOU race is clear just by reading the code:

cibuildwheel/cibuildwheel/__main__.py

Lines 402 to 403 in c93d51e

	if not output_dir.exists():
	output_dir.mkdir(parents=True)

These two lines should probably be this one line: output_dir.mkdir(parents=True, exist_ok=True).

I would submit a PR, except a quick search of exists suggests there may be similar issues throughout the code. Or perhaps not, but I wanted to ask the experts. First I should ask, though: is it safe/supported to run cibuildwheel in parallel this way (shared filesystem, shared output dir)?

Build log

No response

CI config

No response

[henryiii]

I expect this is technical debt from supporting older versions of Python as runners. Unless we were workaround some issue.

[joerick]

I agree with Henry - probably this is from the days before exist_ok. Please send a PR for the change! :)