missing npys during unpack in parallel training runs

[IngmarVoigt2]

This does not happen when just launching a single fold first (and then even relaunching, all parallel folds).
Is there any workaround or fix for this? I am using v2.5.1

2024-11-29 09:52:42.786059: unpacking dataset...
Error when checking nnUNet_preprocessed/Dataset001/nnUNetPlans_3d_fullres/a3cc26b2-bc9b-4a7f-9aff-e49746a77b44.npy and nnUNet_preprocessed/Dataset001/nnUNetPlans_3d_fullres/a3cc26b2-bc9b-4a7f-9aff-e49746a77b44_seg.npy, fixing...
Error when checking nnUNet_preprocessed/Dataset001/nnUNetPlans_3d_fullres/aec35402-d792-454e-a88e-e486d64f7b21.npy and nnUNet_preprocessed/Dataset001/nnUNetPlans_3d_fullres/aec35402-d792-454e-a88e-e486d64f7b21_seg.npy, fixing...
Error when checking nnUNet_preprocessed/Dataset001/nnUNetPlans_3d_fullres/e9713daf-9b18-4bb1-ace2-abf14062e73e.npy and nnUNet_preprocessed/Dataset001/nnUNetPlans_3d_fullres/e9713daf-9b18-4bb1-ace2-abf14062e73e_seg.npy, fixing...
Error when checking nnUNet_preprocessed/Dataset001/nnUNetPlans_3d_fullres/61969266-0e87-4813-bb26-7946ec4287d0.npy and nnUNet_preprocessed/Dataset001/nnUNetPlans_3d_fullres/61969266-0e87-4813-bb26-7946ec4287d0_seg.npy, fixing...
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "env/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
 File "env/lib/python3.10/multiprocessing/pool.py", line 125, in worker                               [0/1852]
    result = (True, func(*args, **kwds))
  File "env/lib/python3.10/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "env/lib/python3.10/site-packages/nnunetv2/training/dataloading/utils.py", line 38, in _convert_to_npy
    np.load(data_npy, mmap_mode='r')
  File "env/lib/python3.10/site-packages/numpy/lib/npyio.py", line 427, in load
    fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: 'nnUNet_preprocessed/Dataset001/nnUNetPlans_3d_fullres/3dda360f-949d-402b-a324-ec34b195f3ba.npy'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "env/bin/nnUNetv2_train", line 8, in <module>
    sys.exit(run_training_entry())
  File "env/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 275, in run_training_entry
    run_training(args.dataset_name_or_id, args.configuration, args.fold, args.tr, args.p, args.pretrained_weights,
  File "env/lib/python3.10/site-packages/nnunetv2/run/run_training.py", line 211, in run_training
    nnunet_trainer.run_training()
  File "env/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 1362, in run_training
    self.on_train_start()
  File "env/lib/python3.10/site-packages/nnunetv2/training/nnUNetTrainer/nnUNetTrainer.py", line 919, in on_train_start
    unpack_dataset(self.preprocessed_dataset_folder, unpack_segmentation=True, overwrite_existing=False,
  File "env/lib/python3.10/site-packages/nnunetv2/training/dataloading/utils.py", line 66, in unpack_dataset
    p.starmap(_convert_to_npy, zip(npz_files,
  File "env/lib/python3.10/multiprocessing/pool.py", line 375, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "env/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
FileNotFoundError: [Errno 2] No such file or directory: 'nnUNet_preprocessed/Dataset001/nnUNetPlans_3d_fullres/3dda360f-949d-402b-a324-ec34b195f3ba.npy'