Multi-processing w/ s3 logs, unless you use spawn to launch your subprocesses

[max-kaufmann]

We use s3fs to access s3, which is apparently not safe for multi-processing. i.e. I have data loading code which calls read_eval_log. Using a ProcessPoolExecutor (but not a ThreadPoolExecutor!) hangs when using read_eval_log on an s3 bucket:

with ProcessPoolExecutor(max_workers=num_workers) as executor:
    futures = {executor.submit(read_eval_log, eval_log): eval_log for eval_log in eval_logs}

I opened an issue in their repo, and they replied saying this was a known limitation, and suggested to use spawn (new python interpret state) when doing multi-processing, i.e.:

with ProcessPoolExecutor(max_workers=num_workers,mp_context=get_context("spawn")) as executor:
    futures = {executor.submit(process_eval_log_single, eval_log): eval_log for eval_log in eval_logs}

This works, but we should have a more robust solution to multi-processing