Project Meeting 2024.05.16

Agenda

• Update on Phase 9a activities

Action Items

Consultant team to run the following tests:

1. WSP: Running with the blosc skims (instead of zlive)

2. WSP: Setting all the threads set to 1

    import os
    os.environ["MKL_NUM_THREADS"] = "1"
    os.environ["OMP_NUM_THREADS"] = "1"
    os.environ["OPENBLAS_NUM_THREADS"] = "1"
    os.environ["NUMBA_NUM_THREADS"] = "1"
    os.environ["VECLIB_MAXIMUM_THREADS"] = "1"
    os.environ["NUMEXPR_NUM_THREADS"] = "1"
   

3. WSP: Turn off default memory reporting in ActivitySim.

4. RSG: Run the loading step, with sharrow on, using SEMCOG’s skims (about 65GBs)

5. RSG: Run SANDAG ABM3 with sharrow on (to replicate Sijia's same results)

6. WSP: Running the notebook with threads set to 1

Meeting Notes

• Running SANDAG ABM3 with sharrow on (WSP)
  • Skims are still taking a very long time to load.
    • Before it was taking 18 hours, now it's taking 12 hours - less time, but still very long. This result has been replicated, so it is not an anomaly.
    • Loading skims independently in the notebook took a short time, but those times have not been replicated when running in the ActivitySim environment.
  • There was a hypothesis that the long run times are due to the memory profiling process. That was turned off but did not help the problem.
  • A number of tests were suggested. WSP and RSG will run individual tests and see if anything changes. CS will also run this same test on their machines.
• Running SANDAG ABM3 with sharrow off (RSG)
  • David was able to run the model with sharrow off and results are similar as before (24 hours to run, RAM peaked at 536GBs), so nothing was broken with recent changes. Details can be found in the issue.