Skip to content

Multiple Zone Systems Tests

Jump to bottom
Wu Sun edited this page Mar 20, 2017 · 12 revisions

ActivitySim Multiple Zone Branch Performance Tests

  1. Objectives:

SANDAG staff (@wusun2) tested the performance of ActivitySim Multiple Zone branch. The objectives of this effort are:

  • Evaluate the performance of implemented methods in ActivitySim Multiple Zone Systems by mimicking SANDAG model settings;
  • Provide a peek at the overall ActivitySim performance, especially the performance of methods shared by both the Multiple Zone and master ActivitySim branches;
  • Identify performance bottlenecks; and
  • Summarize findings.

The focus of this effort is on testing performance of critical 'building blocks' that are called numerous times in utility computations, including retrieving zonal TAZ, TAP, and MAZ values, TAZ, TAP, and MAZ skim values, and the transit virtual path building (TVPB) procedure used to find best transit path (BTP).

  1. Test Environment:

Intel® Xeon® Processor E5-2650 v3 @ 2.30GHz; 256 RAM; 20 cores; Windows Server 2012 R2 Std, 64-bit

  1. Limitations:

Although the aim of this effort is to test scenarios designed to mimic SANDAG 2012 model settings, the test scenarios don't fully reflect model complexity for the following reasons:

  • Many UEC components such as logsums and zonal, household, and personal attributes are not included in the tests.
  • Utility expression evaluation tests are not included, expect for TVPB test in finding BTP.
  • Although the TVPB test is a comprehensive case somewhat like the implementation in CT-RAMP, the tested TVPB expression is simplified with no probability computations and no Monte Carlo choices.

The performance of a real model scenario could be significantly different from those of the tested scenarios due to the above limitations. Regardless, these tests server as an initial peek at the performance of ActivitySim.

  1. Approaches:

This section describes the process of creating test scenarios, starting with a breakdown analysis of SANDAG CT-RAMP model components (Table 1). Only San Diego resident models, the most time-consuming part, are included in this analysis. Unlike special market models, the resident models are more generic with similar structures shared by models of most other partner agencies.

For each model component in Table 1, we estimated the computational burden of retrieving skim data by counting the frequency of calling network LOS methods implemented in ActivitySim Multiple Zone Systems branch. The utility expression calculators (UECs) in CT-RAMP provide a vehicle for estimating computational burden, which is affected by the numbers of decision making units (DMU), skims used in each UEC, and alternatives. Each component is then assigned with a low, medium, or high burden levels based on the counts of LOS network method calls. At the end, mandatory tour mode, non-mandatory tour mode, trip mode, and stop location choices are assigned with 'high' burden status; work, school, and non-mandatory tour location choices are assigned with 'medium' burden status; all other components are assigned with 'low' burden status. Based on assigned burden status of each component, we constructed test scenarios following these rules:

  • Test work/school location choices; skip non-mandatory tour location choice because its performance is similar to that of work/school location choices.
  • Test trip mode choice; skip tour mode choice as it has significantly smaller burden compared with trip mode choice.
  • Test stop location choice
  • Skip all 'low' burden components

Table 1: SANDAG CT-RAMP Resident Models

Model Components DMU #Units Skims Alts Method Calls/Alt Times called Burden Test Note
Work location #workers 1,313,256 OP_GP_DIST 30 taz_skims() 1 39,397,680 Med. Yes
School location #students 965,880 OP_GP_DIST 30 taz_skims() 1 28,976,400 Med. Yes
Auto ownership #households Low No Fast
Transponder #households Low No Fast
Free parking #workers in CBD Low No Fast
CDAP #households Low No No Skims
Mandatory tour frequency #persons with M tours 2,279,136 Distances to work/school * taz_skims() 1 2,279,136 Low No
Mandatory tour mode #Mandatory tours 1,913,084 100 auto skims 26 taz_skims(); TVPB 4 High No
Mandatory TOD #Mandatory tours 1,913,084 OP_GP_DIST * taz_skims() 1 1,913,084 Low No
Join Tour frequency #persons with J tours Low No small size
Join Tour location #joint tours Low No small size
Join Tour Mode #joint tours Low No small size
Join Tour TOD #joint tours Low No small size
Non-Mandatory tour frequency #persons with N tours Low No No Skims
Non-Mandatory tour location #Non-Mandatory tours 1,932,217 OP_GP_DIST 30 taz_skims() 1 57,966,510 Medium No
Non-Mandatory tour mode #Non-Mandatory tours 1,932,217 100 auto skims 26 taz_skims(); TVPB 4 High No
Non-Mandatory TOD #Non-Mandatory tours 1,932,217 OP_GP_DIST * taz_skims() 1 1,932,217 Low No
AtWork tour frequency #persons with AtWork tours Low No small size
AtWork tour location #At-Work tours Low No small size
AtWork tour mode #At-Work tours Low No small size
AtWork TOD #At-Work tours Low No small size
Stop frequency #tours 3,966,590 OP_GP_DIST * taz_skims() 1 3,966,590 Low No
Stop location #trips 10,146,986 distances: os;sd; od; tos; sto 30 get_maz_pairs() 5 1,522,047,900 High Yes
trip mode choice #trips 10,146,986 100 auto skims 26 taz_skims(); TVPB 4 See Table 5 High Yes
  1. Test Scenarios & Test Sizes:

The relevant method in mandatory tour mode choice is taz_skims(). The test size is set to 68,374,080 (Table 2), estimated from the numbers of workers and students, skims used in UEC, and choice alternatives (30 location alternative samples).

Table 2: Mandatory Tour Choice Test

Variables affect test size
# workers 1,313,256
#students 965,880
# skim calls in UEC per unit per alternative 1
skim off peak distance
method taz_skims()
#alternatives 30
Test Size 68,374,080

The relevant method in stop location choice is get_maz_pairs(). The test size is set to 1,522,047,900 (Table 3), estimated from the numbers of trips, skims used in UEC, and choice alternatives (30 location alternative samples).

Table 3: Stop Location Choice Test

Variables affect test size
# trips 10,146,986
# skim calls in UEC per trip per alternative 5
skim Distances: O to S, S to D, O to D, tour O to S, and S to tour D
method get_maz_pairs()
#alternatives 30
Test Size 1,522,047,900

The test size estimation for trip mode choice model is more nuanced. Trip mode choice UEC utilizes auto, walk, bike, and transit skims (Table 4). There are 100 auto skims alone used in SANDAG CT-RAMP. However, the utility computation of each mode utilizes only a few relevant skims specific to that mode. For example, the only skim needed for computing walk mode utility is walk time. For transit modes, instead of transit skims, the utility relies on the BTP found via the TVPB procedure.

Table 4: Trip Mode Choice Skims

Method Skims
#trips 10,146,986
# auto skim calls in UEC per trip per alternative 4 taz_skims() time and distance by inbound/outbound time period
# walk skim calls 1 get_maz_pairs() walk time
# bike skim calls 1 get_maz_pairs() bike time
# TVPB Calls 1 TVPB best transit path
# auto alternatives 14

The burden of computing mode choice utility is mitigated because many trips only have access to a subset of the 26 mode alternatives due to the maximum allowed walk, bike, walk to transit, and drive to transit time thresholds. However, this also makes it difficult to estimate how many modes are available to a trip because mode availability depends on trip origin and destination. In this effort, we constructed 4 hypothetical trip mode choice scenarios (Table 5) that represent various levels of mode availability to trips:

  • 10% trips have access to transit; 25% trips have access to walk and bike; 100% trips have access to drive
  • 25% trips have access to transit; 50% trips have access to walk and bike; 100% trips have access to drive
  • 33% trips have access to transit; 60% trips have access to walk and bike; 100% trips have access to drive
  • 50% trips have access to transit; 75% trips have access to walk and bike; 100% trips have access to drive

The frequency of method calls (Table 5) are derived from the total number of trips and mode availability assumptions above. For example, taz_skims(), get_maz_pairs() and TVPB are called 568,231,216 (, 5,073,493 (100% of all 10,146,986 trips have access to drive mode, each trip has 14 auto mode alternatives, and each alternative uses 4 skims), 2,536,747 bike and walk distance calls each), and 1,014,699 times respectively in scenario 1.

Table 5: Trip Mode Choice Test Scenarios

Method Trip Mode 1 Trip Mode 2 Trip Mode 3 Trip Mode 4
transit available trips 10% 25% 33% 50%
walk available trips 25% 50% 60% 75%
bike available trips 25% 50% 60% 75%
drivable trips 100% 100% 100% 100%
# auto skims calls taz_skims() 568,231,216 568,231,216 568,231,216 568,231,216
# walk skim calls get_maz_pairs() 2,536,747 5,073,493 6,088,192 7,610,240
# bike skim calls get_maz_pairs() 2,536,747 5,073,493 6,088,192 7,610,240
# TVPB Calls TVPB 1,014,699 2,536,747 3,348,505 5,073,493

Based on the analysis in tables 2, 3 and 5, the test sizes of relevant methods in mandatory location, stop location, and trip mode choice models are summarized in Table 6.

Table 6: Test Scenarios & Test Sizes

Model Component Relevant Method Test Size
Mandatory tour mode choice taz_skims() 68,374,080
Stop location choice get_maz_pairs() 1,522,047,900
Trip mode choice 1 taz_skims() 568,231,216
get_maz_pairs() 5,073,493
best transit path 1,014,699
Trip mode choice 2 taz_skims() 568,231,216
get_maz_pairs() 10,146,986
best transit path 2,536,747
Trip mode choice 3 taz_skims() 568,231,216
get_maz_pairs() 12,176,383
best transit path 3,348,505
Trip mode choice 4 taz_skims() 568,231,216
get_maz_pairs() 15,220,479
best transit path 5,073,493
  1. Test Results:

Table 7 is the performance summary of data loading. In this test, population, land use, and walk and bike impedance data are loaded. Although SANDAG model uses 80 skims (70 auto and 10 transit skims), only 6 skims are loaded in this test, including AM SOV, PM SOV, AM local bus, AM premium bus, PM local bus and PM premium skims. Data loading performs well, taking less than 30 seconds.

Table 7: Data Loading Performance

Test Item Runtime (sec)
Load TAZ skims 16.3
Load TAP skims 10.5
Load network LOS 2.9
Total 29.7

Table 8 is a performance summary of each individual method by increasing test size from 10,000 to 50,000,000.

Table 8: Individual Method Performance

Runtime
Test size 10,000 100,000 1,000,000 10,000,000 50,000,000
get_taz() 0.008 0.048 0.722 40.92 1030.5
get_tap() 0.007 0.054 0.808 42.29 1030.3
get_maz() 0.023 0.183 1.695 53.53 1089.8
taz_skims() 0.004 0.034 0.326 3.36 18.8
tap_skims() 0.014 0.135 1.738 94.17 2188.7
get_maz_pairs() 2.867 1.213 5.367 78.57 1142.3
get_maz_tap_pairs() 0.084 0.244 1.928 50.73 1127.4
get_taps_mazs() 0.118 0.341 2.749 28.84 159

Table 9 is a performance summary of each individual method per unit (10000 as one unit) derived from Table 8.

Table 9: Individual Method Performance per Unit (10000)

Runtime/10000
Test size 10,000 100,000 1,000,000 10,000,000 50,000,000
get_taz() 0.008 0.005 0.007 0.041 0.206
get_tap() 0.007 0.005 0.008 0.042 0.206
get_maz() 0.023 0.018 0.017 0.054 0.218
taz_skims() 0.004 0.003 0.003 0.003 0.004
tap_skims() 0.014 0.014 0.017 0.094 0.438
get_maz_pairs() 2.867 0.121 0.054 0.079 0.228
get_maz_tap_pairs() 0.084 0.024 0.019 0.051 0.225
get_taps_mazs() 0.118 0.034 0.027 0.029 0.032

Table 10 is a performance summary of relevant methods by model component.

Table 10: Method Performance by Model Component

Model Component Relevant Method Test Size Runtime (secs) Runtime(hrs)
Mandatory tour mode choice taz_skims() 68,374,080 24.01 0.01
Stop location choice get_maz_pairs() 1,522,047,900 Killed at 68th hr
Trip mode choice scenario 1 taz_skims() 568,231,216 209.3 0.06
get_maz_pairs() 5,073,493 35.6 0.01
best transit path 1,014,699 23391.2 6.50
Total 23636.1 6.57
Trip mode choice scenario 2 taz_skims() 568,231,216 209.3 0.06
get_maz_pairs() 10,146,986 83.4 0.02
best transit path 2,536,747 killed at 12th hr
Total
Trip mode choice scenario 3 taz_skims() 568,231,216 209.3 0.06
get_maz_pairs() 12,176,383 114 0.03
best transit path 3,348,505 NA
Total
Trip mode choice scenario 4 taz_skims() 568,231,216 209.3 0.06
get_maz_pairs() 15,220,479 161.4 0.04
best transit path 5,073,493 NA
Total
  1. Findings:
  • Data loading performs well. Loading household, population, land use, walk and bike impedances, and 6 auto/transit skims took less than 30 seconds.
  • In mandatory tour location choice, taz_skims() method performs well, taking 24 seconds.
  • In stop location choice, the get_maz_pairs() method is a bottleneck. We were not able to complete get_maz_paris() test with test size 1,522,047,900 because the runtime was too long.
  • In trip mode choice, the TVPB procedure is a bottleneck. The TVPB calls took 6.5 hours in the most conservative trip mode scenario. We were not able to complete the TVPB tests for the other three trip mode scenarios because the runtime was too long.
  • As test size increases, the per 10000 unit runtime of get_taz(), get_tap(), get_maz(), tap_skims(), and get_maz_tap_pairs() methods increases significantly.
  • As test size increases, the per 10000 unit runtime of get_maz_tap_pairs() method fluctuates with a unexpected large runtime when the test size is 10,000.
  • As test size increases, the per 10000 unit runtime of taz_skims() and get_taps_mazs() method is stable; Method taz_skims() consistently performs well even when test size is large.
  • The less than ideal performance of tap_skims(), in comparison with taz_skims(), is particularly interesting. If tap_skims() can be implemented in a way similar to taz_skims(), the overall ActivitySim performance could be improved.

ActivitySim

Clone this wiki locally