Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disaggregate Accessibilities #5

Merged
merged 133 commits into from
Oct 28, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
133 commits
Select commit Hold shift + click to select a range
b3df946
initial commit with basic working proto pop gen
nick-fournier-rsg Jun 29, 2022
6441c9a
fixed bad dependency range
nick-fournier-rsg Jun 29, 2022
1b8511f
fixed vary_on method to vary all with mapped fields after
nick-fournier-rsg Jul 2, 2022
96fca8e
added check if already dataframe
nick-fournier-rsg Jul 20, 2022
e92ed4a
added dataframe check if the table passed is already a dataframe
nick-fournier-rsg Jul 25, 2022
68ff25e
working proto pop gen. still debugging mandatory work/school model
nick-fournier-rsg Jul 25, 2022
5189fc8
check all table is df
nick-fournier-rsg Jul 25, 2022
a8c2376
workplace runs, next testing school
nick-fournier-rsg Jul 25, 2022
c82504f
began setting up to get all logsums, not just chosen ones
nick-fournier-rsg Jul 27, 2022
b9fa41b
restructured to run as either one-off or model step
nick-fournier-rsg Jul 28, 2022
7838d2f
revert to last merge. Removed dataframe check in iterate, no longer n…
nick-fournier-rsg Aug 24, 2022
286b5c9
extracted logsums, need to inject into pipeline or save output?
nick-fournier-rsg Aug 24, 2022
c835590
working standalone or step model
nick-fournier-rsg Aug 26, 2022
a73e6a4
extractable accessibilities for fixed and nonmandatory
nick-fournier-rsg Aug 30, 2022
194d327
extractable accessibilities for fixed and nonmandatory
nick-fournier-rsg Aug 30, 2022
a237cfd
cleanup and add documentation
nick-fournier-rsg Aug 31, 2022
d5a7bdd
resolve minor conflicts in model __init__ and conda env yml
nick-fournier-rsg Aug 31, 2022
2e993d9
resolved duplicates of old 'example' folders
nick-fournier-rsg Aug 31, 2022
3bb82b1
model uses write_tables functionality
nick-fournier-rsg Sep 1, 2022
6791a43
disaggregate accessibility runs as model step or standalone. Runs as …
nick-fournier-rsg Sep 2, 2022
3022f67
working model test. table initialization not yet working
nick-fournier-rsg Sep 2, 2022
256b778
added logsum short circuit to get logsums and avoid drawing unneccesa…
nick-fournier-rsg Sep 7, 2022
ee3a1f1
working model plus major cleanup
nick-fournier-rsg Sep 7, 2022
69efa7b
working model plus major cleanup
nick-fournier-rsg Sep 7, 2022
c6f26b5
override sample size
nick-fournier-rsg Sep 8, 2022
a26902c
fixed trace output files
nick-fournier-rsg Sep 8, 2022
c877a46
skip_choice default to False
nick-fournier-rsg Sep 8, 2022
765fd74
fixed empty logsums bug
nick-fournier-rsg Sep 8, 2022
5b27210
cleanup redundent parameters
nick-fournier-rsg Sep 8, 2022
a2137c3
fixed list index instead of set
nick-fournier-rsg Sep 8, 2022
df97dd1
coordinated model run files
nick-fournier-rsg Sep 8, 2022
2ba4249
added default skip to false
nick-fournier-rsg Sep 8, 2022
a66366e
added multiprocessing line
nick-fournier-rsg Sep 9, 2022
120c760
began setup for multiprocessing
nick-fournier-rsg Sep 12, 2022
9a663dd
fixed sampling problem. Pipeline for multi processing, NOT WORKING YET
nick-fournier-rsg Sep 12, 2022
753fa0b
deleted run file, not needed anymore
nick-fournier-rsg Sep 12, 2022
8bee834
Merge pull request #592 from ActivitySim/develop
jpn-- Sep 12, 2022
645b3e4
update release instructions
jpn-- Sep 12, 2022
fe65ce4
auto-build docs on release
jpn-- Sep 13, 2022
af8cf94
remind to update switcher
jpn-- Sep 13, 2022
0725861
Merge pull request #602 from camsys/release-instructions
jpn-- Sep 13, 2022
7ffb4d8
need checkout
jpn-- Sep 14, 2022
1e4ffc5
Merge pull request #604 from camsys/release-docbuild
jpn-- Sep 14, 2022
9dd261b
added helper functions for handling suffix args
nick-fournier-rsg Sep 14, 2022
13df8ed
to enable multiprocessing, overhauled pipeline method to include prot…
nick-fournier-rsg Sep 14, 2022
141bb64
working multiprocess, but requires debugging. fails on other models
nick-fournier-rsg Sep 14, 2022
ac3edd3
working multiprocess, but requires debugging. fails on other models
nick-fournier-rsg Sep 14, 2022
6b2181d
cleanup of mp table registering
nick-fournier-rsg Sep 15, 2022
005307d
fixed tracing and slicing issue
nick-fournier-rsg Sep 15, 2022
2c195f0
removed old 'run' model
nick-fournier-rsg Sep 15, 2022
5198916
setup example 2 zones
nick-fournier-rsg Sep 26, 2022
98efe5c
minimum working 2 zone
nick-fournier-rsg Sep 26, 2022
4771bf1
multizone basic working
nick-fournier-rsg Sep 26, 2022
be5d2ca
fixed buggy settings
nick-fournier-rsg Sep 26, 2022
26c57a7
multiprocessing working, but empty final tables
dhensle Sep 27, 2022
749c65c
fixed blank output bug and cleaned up settings to use verbose table n…
nick-fournier-rsg Sep 27, 2022
01b0e5b
Merge branch 'dev_accessibilities' of github.com:nick-fournier-rsg/ac…
nick-fournier-rsg Sep 27, 2022
0902f88
cleanup run scripts a bit
nick-fournier-rsg Sep 27, 2022
00ba5f9
fixed missing base path for full run
nick-fournier-rsg Sep 27, 2022
ea82c39
fixed missing path
nick-fournier-rsg Sep 27, 2022
570b45c
fixed typo
nick-fournier-rsg Sep 27, 2022
9364cce
fixed 1 per taz sampling
nick-fournier-rsg Sep 27, 2022
25d01c7
fixing person merge bug and filtering workers
dhensle Sep 27, 2022
263cca6
Merge branch 'dev_accessibilities' of https://github.com/nick-fournie…
dhensle Sep 27, 2022
9e5053c
fixed duplicate tours!
nick-fournier-rsg Sep 28, 2022
8739d56
updated settings for initialize disagg
nick-fournier-rsg Sep 29, 2022
f41c769
removed obsolete file
nick-fournier-rsg Sep 29, 2022
b36de0f
cleaned up file and moved initialize to tables/...
nick-fournier-rsg Sep 29, 2022
3013f7c
added find nearest node function
nick-fournier-rsg Sep 29, 2022
4a179a9
added initialize disaggregate_accessibility to initialize_households
nick-fournier-rsg Sep 29, 2022
06e7178
moved initialize steps to 'tables/disaggregate_accessibility'
nick-fournier-rsg Sep 30, 2022
8b1d529
updated settings to include initialize disaggregate accessibilities a…
nick-fournier-rsg Sep 30, 2022
991b2c6
moved initialize disagg accessibilities to tables. Created working me…
nick-fournier-rsg Sep 30, 2022
17b3fea
add mp back in
nick-fournier-rsg Sep 30, 2022
cc97d5f
PEP formatting revisions
nick-fournier-rsg Sep 30, 2022
7b4b780
fixed logsums merge on households instead of persons
nick-fournier-rsg Sep 30, 2022
895438e
fixed _accessibility suffix
nick-fournier-rsg Sep 30, 2022
d9c465c
fixed conflict with persons_merged
nick-fournier-rsg Sep 30, 2022
e1b1b0c
updated yaml to use simpler join method
nick-fournier-rsg Sep 30, 2022
ca1c130
Merge remote-tracking branch 'origin/develop' into dev_accessibilities
nick-fournier-rsg Sep 30, 2022
ba27eb3
PEP formatting fixes
nick-fournier-rsg Oct 1, 2022
9c295f9
refreshed example folder from develop
nick-fournier-rsg Oct 1, 2022
56184ad
added missing line at end of file
nick-fournier-rsg Oct 1, 2022
1e78698
black fixes
nick-fournier-rsg Oct 1, 2022
301ac23
black fixes to disaggregate accessibility changes
nick-fournier-rsg Oct 1, 2022
7be7dd8
Merge branch 'ActivitySim:main' into dev_accessibilities
nick-fournier-rsg Oct 3, 2022
188a05f
fixed missing column pipeline error
nick-fournier-rsg Oct 3, 2022
0da4af9
merged disagg accessibilities into mtc_extended and added doc section
nick-fournier-rsg Oct 3, 2022
a2a743c
ran black on disaggregate_accessibility.py
nick-fournier-rsg Oct 3, 2022
ff51013
updated dependencies
nick-fournier-rsg Oct 3, 2022
25ddd57
removing sklearn imports
dhensle Oct 3, 2022
4298141
blacken
dhensle Oct 3, 2022
9fa6d0b
add if none catch
nick-fournier-rsg Oct 3, 2022
4123c86
fixed None suffix default
nick-fournier-rsg Oct 3, 2022
0941d01
moved order of get_table(persons_merged) to avoid pulling prematurely…
nick-fournier-rsg Oct 3, 2022
40a8170
tested and cleaned up rng channels
nick-fournier-rsg Oct 4, 2022
079dcfb
setup injectable suffixes to allow add_size_table as model step not j…
nick-fournier-rsg Oct 4, 2022
2187a4b
removed accessibility output from test
nick-fournier-rsg Oct 4, 2022
d52666e
re blacken py scripts
nick-fournier-rsg Oct 4, 2022
8ab2f65
fixed tracing typo
nick-fournier-rsg Oct 4, 2022
29a9ddc
added variable index name suffix to pass optionally
nick-fournier-rsg Oct 4, 2022
d4886ee
pipeline housekeeping to cleanup any tables, traceables, or channels …
nick-fournier-rsg Oct 5, 2022
5406541
added multiprocess testing
nick-fournier-rsg Oct 5, 2022
6521f56
blacken updates
nick-fournier-rsg Oct 5, 2022
82c36c7
updated test scripts to include MP, problem with vehicle model in mp …
nick-fournier-rsg Oct 5, 2022
ecd53a6
added improved origin sampling, resolved issue with merging with samp…
nick-fournier-rsg Oct 7, 2022
4e01bf2
added sci-kit learn to test depends
nick-fournier-rsg Oct 7, 2022
b6ac7ef
fixed person merging error causing pytest fail, uses inject method to…
nick-fournier-rsg Oct 7, 2022
f53ec77
cleanup comments
nick-fournier-rsg Oct 7, 2022
35d0a5d
fixed pytest to include accessibility table in regress
nick-fournier-rsg Oct 7, 2022
d5042f5
setup for mp test, but needs debugging
nick-fournier-rsg Oct 7, 2022
d5132fc
'blacken'
nick-fournier-rsg Oct 7, 2022
ca317fc
'blacken'
nick-fournier-rsg Oct 7, 2022
935b42d
cleanup example folder
nick-fournier-rsg Oct 7, 2022
b351eb0
fixed pipeline NoneType bug for disagg accessibility table
nick-fournier-rsg Oct 7, 2022
e41fe66
fixed pytest fail on mp, due to exept:True on mp_simulate
nick-fournier-rsg Oct 11, 2022
34e08b3
'blacken'
nick-fournier-rsg Oct 11, 2022
87c251f
added weighted k-means method
nick-fournier-rsg Oct 13, 2022
aeff8e8
created run script for multiple sampling scenarios
nick-fournier-rsg Oct 13, 2022
93544fd
blacken changes
nick-fournier-rsg Oct 13, 2022
47d1e6b
blacken
nick-fournier-rsg Oct 13, 2022
9a9c9cf
fixed copy script
nick-fournier-rsg Oct 17, 2022
a291027
blacken sampling script
nick-fournier-rsg Oct 17, 2022
8180d7d
fixed n_zone when integer
nick-fournier-rsg Oct 17, 2022
5960f40
fixed typo
nick-fournier-rsg Oct 17, 2022
6584f7b
update sampling script
nick-fournier-rsg Oct 25, 2022
dada5cd
'blacken'
nick-fournier-rsg Oct 25, 2022
1096cfb
fixed replacement sample bug
nick-fournier-rsg Oct 25, 2022
0d0a7c3
updated documentation
nick-fournier-rsg Oct 25, 2022
17c6b21
more flexible scenario testing
nick-fournier-rsg Oct 26, 2022
7dde020
blacken
nick-fournier-rsg Oct 26, 2022
914fa57
Merge pull request #4 from nick-fournier-rsg/dev_accessibilities
dhensle Oct 28, 2022
9150754
Merge branch 'rsg_phase7_dev' into disagg_for_merge
dhensle Oct 28, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 53 additions & 1 deletion .github/workflows/deployment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,58 @@ jobs:
repository_url: https://test.pypi.org/legacy/
verbose: true

docbuild:
needs: test-built-dist
if: github.event_name == 'release'
name: ubuntu-latest py3.9
runs-on: ubuntu-latest
defaults:
run:
shell: bash -l {0}
steps:
- uses: actions/checkout@v3
- uses: actions/download-artifact@v3
with:
name: releases
path: dist
- name: Set up Python 3.9
uses: actions/setup-python@v2
with:
python-version: 3.9
- name: Install dependencies
uses: conda-incubator/setup-miniconda@v2
with:
miniforge-variant: Mambaforge
miniforge-version: latest
use-mamba: true
environment-file: conda-environments/docbuild.yml
python-version: 3.9
activate-environment: docbuild
auto-activate-base: false
auto-update-conda: false
- name: Install activitysim
run: |
python -m pip install dist/activitysim-*.whl
- name: Conda checkup
run: |
conda info -a
conda list
echo REPOSITORY ${{ github.repository }}
echo REF ${{ github.ref }}
echo REF_NAME ${{ github.ref_name }}
- name: Build the docs
run: |
cd docs
make clean
make html
- name: Push to GitHub Pages
uses: peaceiris/actions-gh-pages@v3.8.0
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
# Token is created automatically by Github Actions, no other config needed
publish_dir: ./docs/_build/html
destination_dir: ${{ github.ref_name }}

upload-to-pypi:
needs: test-built-dist
if: github.event_name == 'release'
Expand All @@ -89,4 +141,4 @@ jobs:
with:
user: __token__
password: ${{ secrets.PYPI_TOKEN }}
verbose: true
verbose: true
112 changes: 47 additions & 65 deletions HOW_TO_RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
01. Check that the branch you intend to release is passing tests on Travis.
If it's not passing there you should not release it.

00. Start from a completely clean conda environment
and git repository. Assuming you have `conda` installed, you can do so
by starting where ActivitySim is not yet cloned (e.g. in an empty
00. Start from a completely clean conda environment
and git repository. Assuming you have `conda` installed, you can do so
by starting where ActivitySim is not yet cloned (e.g. in an empty
directory) and running:
```sh
conda create -n TEMP-ASIM-DEV python=3.9 git gh -c conda-forge --override-channels
Expand All @@ -19,7 +19,7 @@

00. Per project policy, code on the main branch should have been released,
but if you are *preparing* a release then the code should be on the `develop`
branch. Switch to that branch now, and make sure it is synced to the
branch. Switch to that branch now, and make sure it is synced to the
version on GitHub:
```sh
git switch develop
Expand All @@ -28,45 +28,46 @@

00. Update your Conda environment for testing. We do not want to use an
existing environment on your machine, as it may be out-of-date
and we want to make sure everything passes muster using the
and we want to make sure everything passes muster using the
most up-to-date dependencies available. The following command
will update the active environment (we made this to be `TEMP-ASIM-DEV`
will update the active environment (we made this to be `TEMP-ASIM-DEV`
if you followed the directions above).
```sh
conda env update --file=conda-environments/activitysim-dev.yml
```
If you add to the ActivitySim dependencies, make sure to also update
the environments in `conda-environments`, which are used for testing
and development. If they are not updated, these environments will end
If you add to the ActivitySim dependencies, make sure to also update
the environments in `conda-environments`, which are used for testing
and development. If they are not updated, these environments will end
up with dependencies loaded from *pip* instead of *conda-forge*.

00. Run pycodestyle to ensure that the codebase passes all style checks.
00. Run `black` to ensure that the codebase passes minimal style checks.
This check should only take a few seconds. These checks are also done on
Travis and are platform independent, so they should not be necessary to
GitHub Actions and are platform independent, so they should not be necessary to
replicate locally, but are listed here for completeness.
```sh
pycodestyle .
black --check --diff .
```

00. Run the regular test suite on Windows. Travis tests are done on Linux,
but most users are on Windows, and the test suite should also be run
on Windows to ensure that it works on that platform as well. If you
00. Run the regular test suite on Windows. Most GitHub Actions tests are done on Linux,
Linux (it's faster to start up and run a new clean VM for testing) but most
users are on Windows, and the test suite should also be run on Windows to
ensure that it works on that platform as well. If you
are not preparing this release on Windows, you should be sure to run
at least through this step on a Windows machine before finalizing a
release.
A few of the tests require pre-created data that is not included in the
repository directly, but rather recreated on the fly before testing. The
regular test suite takes some time to run, between about half an hour and
at least through this step on a Windows machine before finalizing a
release.

A few of the tests require pre-created data that is not included in the
repository directly, but rather recreated on the fly before testing. The
regular test suite takes some time to run, between about half an hour and
two hours depending on the specs of your machine.
```sh
python activitysim/examples/placeholder_multiple_zone/scripts/two_zone_example_data.py
python activitysim/examples/placeholder_multiple_zone/scripts/three_zone_example_data.py
pytest .
```

00. Test the full-scale regional examples. These examples are big, too
large to run on Travis, and will take a lot of time (many hours) to
large to run on GitHub Actions, and will take a lot of time (many hours) to
download and run.
```sh
mkdir tmp-asim
Expand All @@ -76,68 +77,41 @@
call run_all_examples.bat
```
These tests will run through the gamut even if some of them crash, so
if you don't sit and watch them go (please don't do this) you'll need
if you don't sit and watch them go (please don't do this) you'll need
to scan through the results to make sure there are no errors after the
fact.
```sh
python ../activitysim/examples/scan_examples_for_errors.py .
```

00. Test the notebooks in `activitysim/examples/prototype_mtc/notebooks`.
There are also demo notebooks for estimation, but their functionality
There are also demo notebooks for estimation, but their functionality
is completely tested in the unit tests run previously.

00. Use bump2version to tag the release commit and update the
version number. The following code will generate a "patch" release,
incrementing the third value in the version number (i.e. "1.2.3"
becomes "1.2.4"). Alternatively, make a "minor" or "major" release.
The `--list` command will generate output to your console to confirm
that the old and new version numbers are what you expect, before you
push the commit (with the changed version in the code) and tags to
GitHub.
```sh
bump2version patch --list
```

It is also possible to make a development pre-release. To do so,
explicitly set the version number to the next patch plus a ".devN"
suffix:

```sh
bump2version patch --new-version 1.2.3.dev0 --list
```

Then, when ready to make a "final" release, set the version by
explicitly removing the suffix:
```sh
bump2version patch --new-version 1.2.3 --list
```

00. Push the tagged commit to GitHub.
```sh
git push --tags
```

00. For non-development releases, open a pull request to merge the proposed
release into main. The following command will open a web browser for
00. For non-development releases, open a pull request to merge the proposed
release into main. The following command will open a web browser for
you to create the pull request.
```sh
gh pr create --web
```
After creating the PR, confirm with the ActivitySim PMC that the release
is ready before actually merging it.

Once final approval is granted, merge the PR into main. The presence
of the git tags added earlier will trigger automated build steps to
prepare and deploy the release to pypi and conda-forge.

00. Create a "release" on GitHub.
```sh
gh release create v1.2.3
```
The process of creating and tagging a release will automatically
trigger various GitHub Actions scripts to build, test, and publish the
new release to PyPI and conda forge, assuming there are no errors.

For a development pre-release, include the `--prerelease` argument.
As the project's policy is that only formally released code is merged
to the main branch, any pre-release should also be built against a
to the main branch, any pre-release should also be built against a
non-default branch. For example, to pre-release from the `develop`
branch:
```sh
Expand All @@ -147,11 +121,19 @@
--notes "this pre-release is for a cool new feature" \
--title "Development Pre-Release"
```
00. Clean up your workspace, including removing the Conda environment used for
testing (which will prevent you from accidentally using an old

00. Clean up your workspace, including removing the Conda environment used for
testing (which will prevent you from accidentally using an old
environment when you should have a fresh up-to-date one next time).
```sh
conda deactivate
conda env remove -n TEMP-ASIM-DEV
```
```

00. Change the default redirect page for the ActivitySim documentation to point
to the newly released documentation. The redirect page can be edited
[here](https://github.com/ActivitySim/activitysim/blob/gh-pages/index.html).

00. Add the new release to the `switch.json` file. Don't delete the references
for existing old documentation. The switcher can be edited
[here](https://github.com/ActivitySim/activitysim/blob/gh-pages/switcher.json).
2 changes: 2 additions & 0 deletions activitysim/abm/models/__init__.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# ActivitySim
# See full license in LICENSE.txt.

from . import (
accessibility,
disaggregate_accessibility,
atwork_subtour_destination,
atwork_subtour_frequency,
atwork_subtour_mode_choice,
Expand Down
Loading