Multiprocess Fails Accessing Sharrow Cache

[dhensle]

Describe the bug
Model run is crashing with a permissions error trying to access the sharrow flow cache in the vehicle allocation model.

I was successfully able to complete the run with 4, 8, 12, and 20 cores, but 16 and 24 cores failed. (Machine has 24 cores total.) I suspect the two runs that failed just got unlucky with the different processes trying to access the same file at the same time. I do not know why this has happened on the vehicle allocation model both times though.

Potential fix: Put in a wait statement if access to the sharrow cache is denied?

To Reproduce
Steps to reproduce the behavior:
Run MTC example model with sharrow on and multiprocess with many cores.

Screenshots

crash_logs.zip

Additional context
I will try to re-run 16 and 24 and see if I hit the same error again.

[i-am-sijia]

Is it somehow recompiling? The trace shows it's calling sharrow flow rewrite(). It would make sense if multiple processors are rewriting the same file and they clash.

[dhensle]

Tried mp with sharrow on for the SANDAG ABM3 model as well using 4 cores. It also crashed with the same error in vehicle allocation. I agree that the problem seems to be sharrow recompiling after missing one of the vehicle type alternatives.

Interestingly, I do not see sharrow recompiling the the vehicle_allocation model when running in single process...

Log files are attached below:
logs.zip

[jpn--]

It looks like you are maybe setting the state.filesystem.sharrow_cache_dir attribute to be "cache", so that get_sharrow_cache_dir() is coming up outputs\output_sh_mp_full_4cores\cache on your multiprocess run. When you run single process it comes up as outputs\output_sh_full_1thread\cache. So, when you're running multiprocess you're not picking up the pre-compiled sharrow code, but attempting to recompile it. I think you then crash on vehicle_allocation.simple_simulate.eval_mnl probably because it is the first thing that take takes enough time to compile such that a processes starts the compile step while another processes is still doing the compiling.

We can tell it's not a data-type problem because two processes are competing to compile the same "flow", which has a unique hash that ensures the data types are actually the same (if they were different it wouldn't crash as there would be different files with different hashes, it would just be slow).

You can fix this by using the same explicit absolute-path cache location for both model runs, or just by activating the persistent sharrow cache, as I did here in the example exercise script. This moves the sharrow cache out of a run-specific directory and into a common run-agnostic directory, so it gets re-used in cases like this.

[dhensle]

We all discussed offline and came to a conclusion. TL;DR We need to convert the vehicle allocation body type and fuel types into integers so as to not trigger sharrow recompilation when running a full sample size.

First, the above comment doesn't really apply in this case as I copy the cache from the compile folder to the output folder I am running in.

Tracing through the cache of the compiled run with 100 households and the multiprocessing run that failed shows that not all alternatives in the vehicle allocation model are captured by the small sample size. The below screenshot shows how the alternatives for body type and fuel type in the vehicle_allocation model are not the same. (The left file is the compile run and is missing alternatives that are present in the full model run on the right.)

This issue existed in single processing but was not caught since the single process just recompiled the vehicle allocation model and proceeded on its way. Multi-processing crashed when re-compiling due to the multiple cores trying to write to the same cache file.

The fix is to integer encode the body type and fuel type in the vehicle allocation model so that sharrow does not need to know all of the possible categorical options when compiling.

[JoeJimFlood]

I got the same thing for nonmandatory tour scheduling

[JoeJimFlood]

I got it in trip destination as well

[jpn--]

@JoeJimFlood are you running multiprocessing with a small(ish) household sample size?

For the fuel and body type variables discussed higher up this thread, they were getting created as categorical within a preprocessor, and not having stable dtype on that account. Your issue with tour_type is different because that column is getting created by core Python code. I don't at first glance see how the categories could get out of order unless possibly some subprocess is handling a sample part so small that some tour types are just missing.

[jpn--]

I am closing this, as the original bug in this issue has been addressed by changing the data type of variables in the vehicle allocation pre-processor.

@JoeJimFlood, if you are still encountering problems you can reopen or create a new issue with additional details.