MEG - Add support for slurm job scheduler

[allermat]

Suggested by @RikHenson

• parallel execution
• output logging and storage

[allermat]

Hi @RikHenson, @olafhauk, @dcdace

I thought about this, specifically the parallel execution. Below is an example data structure from this mne-bids tutorial:

|MNE-sample-data-bids\
|--- README
|--- dataset_description.json
|--- participants.json
|--- participants.tsv
|--- sub-01\
|------ ses-01\
|--------- sub-01_ses-01_scans.tsv
|--------- meg\
|------------ sub-01_ses-01_acq-calibration_meg.dat
|------------ sub-01_ses-01_acq-crosstalk_meg.fif
|------------ sub-01_ses-01_coordsystem.json
|------------ sub-01_ses-01_task-audiovisual_run-1_channels.tsv
|------------ sub-01_ses-01_task-audiovisual_run-1_events.json
|------------ sub-01_ses-01_task-audiovisual_run-1_events.tsv
|------------ sub-01_ses-01_task-audiovisual_run-1_meg.fif
|------------ sub-01_ses-01_task-audiovisual_run-1_meg.json
|--- sub-emptyroom\
|------ ses-20021206\
|--------- sub-emptyroom_ses-20021206_scans.tsv
|--------- meg\
|------------ sub-emptyroom_ses-20021206_task-noise_channels.tsv
|------------ sub-emptyroom_ses-20021206_task-noise_meg.fif
|------------ sub-emptyroom_ses-20021206_task-noise_meg.json

When we call mne_bids.write_raw_bids it writes the current fif file into the BIDS folder, but also updates several other files, (e.g. the participants.json, participants.tsv, and the *_scans.tsv file). If we execute the code in parallel, then there is a chance that two processes attempt to write the same file at the same time, which would lead to an error. If we parallelize across subjects then these would be the participants.json and .tsv files.

I'm sure there's a way of implementing this so that it works in parallel, the mne-bids-pipeline has even more complicated dependencies between files and it works fine in parallel. But looking at their implementation of parallelization I feel I'm out of my depht there.

Do you have any thoughts on this?

Thanks,
Máté

[dcdace]

I run MRI BIDS parallel across subjects and never have any problems. MRI BIDS also writes in participants.json/.tsv files.

[allermat]

I added a simple slurm job script which runs the conversion in serial mode. Parallel execution is to be implemented, taking in mind the above complications. I created issue #9 for this and I'm closing this issue.