Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

External regions #858

Closed
wants to merge 8 commits into from
Closed

External regions #858

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
915e8d7
test external regional model examples
jpn-- Apr 23, 2024
3038256
cache buster
jpn-- Apr 23, 2024
3bdc1a4
optional variable doc
jpn-- Apr 24, 2024
b876e9c
fix conda cache dirs
jpn-- Apr 24, 2024
f578e72
Merge branch 'main' into external-regions
jpn-- Apr 24, 2024
9c282f0
Merge branch 'main' into external-regions
jpn-- Apr 24, 2024
e41054f
Merge branch 'main' into external-regions
jpn-- Apr 25, 2024
79cd6a2
Merge branch 'main' into external-regions
jpn-- May 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 46 additions & 63 deletions .github/workflows/core_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ on:
- '*'

env:
CACHE_NUMBER: 0 # increase to reset cache manually
CACHE_NUMBER: 2 # increase to reset cache manually

jobs:
foundation:
Expand All @@ -24,10 +24,10 @@ jobs:
name: linux-64-py${{ matrix.python-version }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: Setup Mambaforge
uses: conda-incubator/setup-miniconda@v2
uses: conda-incubator/setup-miniconda@v3
with:
miniforge-variant: Mambaforge
miniforge-version: latest
Expand All @@ -38,7 +38,7 @@ jobs:
- name: Set cache date for year and month
run: echo "DATE=$(date +'%Y%m')" >> $GITHUB_ENV

- uses: actions/cache@v3
- uses: actions/cache@v4
with:
path: /usr/share/miniconda3/envs/asim-test
key: linux-64-conda-${{ hashFiles('conda-environments/github-actions-tests.yml') }}-${{ env.DATE }}-${{ env.CACHE_NUMBER }}
Expand All @@ -47,19 +47,6 @@ jobs:
- name: Update environment
run: |
mamba env update -n asim-test -f conda-environments/github-actions-tests.yml
mamba install --yes \
"psutil=5.9.5" \
"pydantic=2.6.1" \
"pypyr=5.8.0" \
"pytables=3.6.1" \
"pytest-cov" \
"pytest-regressions=2.5.0" \
"scikit-learn=1.2.2" \
"sharrow>=2.6.0" \
"simwrapper=1.8.5" \
"xarray=2023.2.0" \
"zarr=2.14.2" \
"zstandard=0.21.0"
if: steps.cache.outputs.cache-hit != 'true'

- name: Install activitysim
Expand Down Expand Up @@ -109,12 +96,10 @@ jobs:
include:
- os: macos-latest
label: macOS
prefix: /Users/runner/miniconda3/envs/asim-test
python-version: "3.10"

- os: windows-latest
label: win-64
prefix: C:\Miniconda3\envs\asim-test
python-version: "3.10"

defaults:
Expand All @@ -124,10 +109,10 @@ jobs:
name: ${{ matrix.label }}
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: Setup Mambaforge
uses: conda-incubator/setup-miniconda@v2
uses: conda-incubator/setup-miniconda@v3
with:
miniforge-variant: Mambaforge
miniforge-version: latest
Expand All @@ -138,9 +123,9 @@ jobs:
- name: Set cache date for year and month
run: echo "DATE=$(date +'%Y%m')" >> $GITHUB_ENV

- uses: actions/cache@v3
- uses: actions/cache@v4
with:
path: ${{ matrix.prefix }}
path: ${{ env.CONDA }}/envs
key: ${{ matrix.label }}-conda-${{ hashFiles('conda-environments/github-actions-tests.yml') }}-${{ env.DATE }}-${{ env.CACHE_NUMBER }}
id: cache

Expand Down Expand Up @@ -199,7 +184,6 @@ jobs:
builtin_regional_models:
needs: foundation
env:
mamba-env-prefix: /usr/share/miniconda3/envs/asim-test
python-version: "3.10"
label: linux-64
strategy:
Expand All @@ -221,10 +205,10 @@ jobs:
name: ${{ matrix.region }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: Setup Mambaforge
uses: conda-incubator/setup-miniconda@v2
uses: conda-incubator/setup-miniconda@v3
with:
miniforge-variant: Mambaforge
miniforge-version: latest
Expand All @@ -235,9 +219,9 @@ jobs:
- name: Set cache date for year and month
run: echo "DATE=$(date +'%Y%m')" >> $GITHUB_ENV

- uses: actions/cache@v3
- uses: actions/cache@v4
with:
path: ${{ env.mamba-env-prefix }}
path: ${{ env.CONDA }}/envs
key: ${{ env.label }}-conda-${{ hashFiles('conda-environments/github-actions-tests.yml') }}-${{ env.DATE }}-${{ env.CACHE_NUMBER }}
id: cache

Expand Down Expand Up @@ -301,25 +285,31 @@ jobs:
external_regional_models:
needs: foundation
env:
mamba-env-prefix: /usr/share/miniconda3/envs/asim-test
python-version: "3.10"
label: linux-64
strategy:
matrix:
region:
- prototype_mtc
- prototype_psrc_in_development
include:
- region: Standard 1-Zone Example (MTC)
region-org: ActivitySim
region-repo: activitysim-prototype-mtc
region-branch: extended
- region: Standard 2-Zone Example (SANDAG)
region-org: ActivitySim
region-repo: sandag-abm3-example
region-branch: main
fail-fast: false
defaults:
run:
shell: bash -l {0}
name: ${{ matrix.region }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Checkout ActivitySim
uses: actions/checkout@v4

- name: Setup Mambaforge
uses: conda-incubator/setup-miniconda@v2
uses: conda-incubator/setup-miniconda@v3
with:
miniforge-variant: Mambaforge
miniforge-version: latest
Expand All @@ -330,52 +320,46 @@ jobs:
- name: Set cache date for year and month
run: echo "DATE=$(date +'%Y%m')" >> $GITHUB_ENV

- uses: actions/cache@v3
- uses: actions/cache@v4
with:
path: |
${{ env.mamba-env-prefix }}
${{ env.CONDA }}/envs
~/.cache/ActivitySim
key: ${{ env.label }}-conda-${{ hashFiles('conda-environments/github-actions-tests.yml') }}-${{ env.DATE }}-${{ env.CACHE_NUMBER }}
id: cache

- name: Update environment
run: |
mamba env update -n asim-test -f conda-environments/github-actions-tests.yml
mamba install --yes \
"psutil=5.9.5" \
"pydantic=2.6.1" \
"pypyr=5.8.0" \
"pytables=3.6.1" \
"pytest-cov" \
"pytest-regressions=2.5.0" \
"scikit-learn=1.2.2" \
"sharrow>=2.6.0" \
"simwrapper=1.8.5" \
"xarray=2023.2.0" \
"zarr=2.14.2" \
"zstandard=0.21.0"
if: steps.cache.outputs.cache-hit != 'true'

- name: Install activitysim
# installing without dependencies is faster, we trust that all needed dependencies
# are in the conda environment defined above. Also, this avoids pip getting
# confused and reinstalling tables (pytables).
run: |
python -m pip install -e . --no-deps
python -m pip install . --no-deps

- name: Conda checkup
run: |
mamba info -a
mamba list

- name: Checkout Example
uses: actions/checkout@v4
with:
repository: '${{ matrix.region-org }}/${{ matrix.region-repo }}'
ref: '${{ matrix.region-branch }}'
path: '${{ matrix.region-repo }}'

- name: Test ${{ matrix.region }}
run: |
python -m activitysim test ${{ matrix.region }}
cd ${{ matrix.region-repo }}/test
python -m pytest .

random_seed_generation:
needs: foundation
env:
mamba-env-prefix: /usr/share/miniconda3/envs/asim-test
python-version: "3.10"
label: linux-64
defaults:
Expand All @@ -385,10 +369,10 @@ jobs:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: Setup Mambaforge
uses: conda-incubator/setup-miniconda@v2
uses: conda-incubator/setup-miniconda@v3
with:
miniforge-variant: Mambaforge
miniforge-version: latest
Expand All @@ -399,9 +383,9 @@ jobs:
- name: Set cache date for year and month
run: echo "DATE=$(date +'%Y%m')" >> $GITHUB_ENV

- uses: actions/cache@v3
- uses: actions/cache@v4
with:
path: ${{ env.mamba-env-prefix }}
path: ${{ env.CONDA }}/envs
key: ${{ env.label }}-conda-${{ hashFiles('conda-environments/github-actions-tests.yml') }}-${{ env.DATE }}-${{ env.CACHE_NUMBER }}
id: cache

Expand Down Expand Up @@ -442,7 +426,6 @@ jobs:
estimation_mode:
needs: foundation
env:
mamba-env-prefix: /usr/share/miniconda3/envs/asim-test
python-version: "3.10"
label: linux-64
defaults:
Expand All @@ -451,10 +434,10 @@ jobs:
name: estimation_mode_test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: Setup Mambaforge
uses: conda-incubator/setup-miniconda@v2
uses: conda-incubator/setup-miniconda@v3
with:
miniforge-variant: Mambaforge
miniforge-version: latest
Expand All @@ -465,9 +448,9 @@ jobs:
- name: Set cache date for year and month
run: echo "DATE=$(date +'%Y%m')" >> $GITHUB_ENV

- uses: actions/cache@v3
- uses: actions/cache@v4
with:
path: ${{ env.mamba-env-prefix }}
path: ${{ env.CONDA }}/envs
key: ${{ env.label }}-conda-${{ hashFiles('conda-environments/github-actions-tests.yml') }}-${{ env.DATE }}-${{ env.CACHE_NUMBER }}
id: cache

Expand Down Expand Up @@ -517,15 +500,15 @@ jobs:
run:
shell: bash -l {0}
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0 # get all tags, lets setuptools_scm do its thing
- name: Set up Python 3.10
uses: actions/setup-python@v2
with:
python-version: "3.10"
- name: Install dependencies
uses: conda-incubator/setup-miniconda@v2
uses: conda-incubator/setup-miniconda@v3
with:
miniforge-variant: Mambaforge
miniforge-version: latest
Expand Down
12 changes: 11 additions & 1 deletion docs/dev-guide/using-sharrow.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,13 +193,23 @@ in several examples:
`@np.log1p(size_terms.get(df.alt_dest, df.purpose)) # sharrow: np.log1p(size_terms['sizearray'])`

Here, `size_terms` is a DataFrameMatrix class instance, a special class written into
ActivitySim to facilitate reading from a DataFrame as it it were a 2-d array. As it
ActivitySim to facilitate reading from a DataFrame as if it were a 2-d array. As it
is a special purpose class written in Python, the numba compiler cannot handle it directly.
Fortunately, sharrow provides an alternative: passing the size terms as a xarray `DataArray`.
This has a slightly different interface, so the sharrow and legacy evaluation modes require
different expressions. The switching expression is used to handle the DataFrameMatrix
on the left (before "# sharrow:") and the DataArray on the right.

### Optional Variables

In some cases, a variable may be used where it is available, but is not strictly
necessary for the model to run. For example, a model may have a reference to
mode choice logsums, but the model can still run without them, if it is being used
prior to when logsums are calculated. In this case, the variable can be accessed
using the `get` method, which allows for a default value if the variable is not found.

`@df.get('mode_choice_logsum', 0)`

### Performance Considerations

Sharrow is usually expected to bring significant performance gains to ActivitySim.
Expand Down
Loading