Project Meeting 2018.10.05

Multiprocessing

• Prototype configuration example on the Multi-Processing design page
• Rough prototype ActivitySim multiprocessing (asim mp) is running
  • Running on Windows as long as skims don’t exceed 2GB due to limitation of RawArray size in Python 2.7 on Windows due to sys.maxint
  • The runtime gains from single to mp are dependent on the machine RAM, processors available, number of HHs, chunk size, and amount of machine RAM to leave for other processes. We will test and document a number of configuration scenarios as we work on this.
  • Running the entire example dataset (797,674 persons with 190 zones) on the SD modeling server took:

windows single process:
INFO - activitysim.core.tracing - Time to execute mp_simulation : 4383.497 sec (73.1 min)

windows with 3 processes:
INFO - activitysim.core.tracing - Time to execute mp_simulation : 2693.601 sec (44.9 min)

• Supporting both Python 2.7 and Python 3?
  • Tests show no limitation to size of RawArray in Python 3
  • Tests show that variable number of shared RawArraw objects can be passed to subprocess for implementation in 2.7.
  • Python 2 will not be maintained past 2020 - https://pythonclock.org
  • Started looking into porting to Python 3. It looks like we could have a code base that runs on both Python 2 and 3.
  • There are 3 modules that don’t have python3 distributions
    • We should probably move orca into activitysim and strip out everything we don’t need
    • Someone needs to port omx
    • zbox already works with python3 so it probably just needs a setup tweak

Outreach

• Florida Model Task Force invite