Merge pull request #76 from tsdataclinic/national_data

National data

.gitignore

@@ -129,7 +129,6 @@ dmypy.json
 .pyre/
 
 .DS_Store
-config_local.json
 ###### create react-app 
 
 # dependencies
@@ -159,11 +158,14 @@ config_local.json
 analysis/cache/
 analysis/src/cache/
 analysis/data/
-analysis/notebooks/cache/
-analysis/notebooks/
+analysis/test_notebooks/cache/
+analysis/test_notebooks/
 /app/public/results/*
 /app/public/results/*/*
 build.tar.gz
+pipeline_errors.txt
+config_local.json
+
 
 *.csv
 *.sql

README.md

@@ -22,12 +22,11 @@ The climate crisis we face requires collective intelligence and creative problem
 
 ### Processed Data Files
 
-Our app relies on two data files that we process using the data sources (listed below). These files are:
+Our app relies on two data files that we process using the data sources (listed below). 
 
-- [Stop Features](public/data/stop_features.geojson): Stop level dervived metrics described in the data dictionary
-- [Hospitals](public/data/hospitals.geojson): Locations of hospitals within included cities
+- Stop Features: Stop level dervived metrics described in the data dictionary
+- Hospitals: Locations of hospitals within included cities
 
-We have also included csv files of the stops and summaries by route in the same [directory](public/data/).
 
 #### Derived Metrics Data Dictionary
 
@@ -36,7 +35,7 @@ We have also included csv files of the stops and summaries by route in the same
 | stop_id | GTFS feed stop id | str | [Apache 2.0](Apache2.0) | GTFS Feeds |
 | stop_name | GTFS feed stop names | str | [Apache 2.0](Apache2.0) | GTFS Feeds |
 | routes_serviced | List of all routes servicing a stop | list | [Apache 2.0](Apache2.0) | GTFS Feeds |
-| flood_risk_category | Score 0/1/2 indicating low/medium/high flood risk around transit stop | int | [CC BY-NC-SA 4.0](CC4.0-BY-NC-SA) | [](https://aws.amazon.com/marketplace/pp/prodview-r36lzzzjacd32?sr=0-1&ref_=beagle&applicationId=AWSMPContessa#overview)[First Street Climate-Adjusted Flood Risk](https://aws.amazon.com/marketplace/pp/prodview-r36lzzzjacd32?sr=0-1&ref_=beagle&applicationId=AWSMPContessa#overview),<br>GTFS Feeds |
+| climate_risk_category | Score 0/1/2 indicating low/medium/high climate risk around transit stop | int | [CC BY-NC-SA 4.0](CC4.0-BY-NC-SA) | [](https://aws.amazon.com/marketplace/pp/prodview-r36lzzzjacd32?sr=0-1&ref_=beagle&applicationId=AWSMPContessa#overview)[First Street Climate-Adjusted Flood Risk](https://aws.amazon.com/marketplace/pp/prodview-r36lzzzjacd32?sr=0-1&ref_=beagle&applicationId=AWSMPContessa#overview),<br>GTFS Feeds |
 | hospital_access_cateogory | Score 0/1/2 indicating low/medium/high hospital access from transit stop | int | [Apache 2.0](Apache2.0) | [Geographic Names Information System National File 2021](https://www.usgs.gov/u.s.-board-on-geographic-names/download-gnis-data),<br>GTFS Feeds |
 | job_access_category | Score 0/1/2 indicating low/medium/high number of jobs around transit stop | int | [Apache 2.0](Apache2.0) | [](https://lehd.ces.census.gov/data/)[LEHD Origin-Destination Statistics](https://lehd.ces.census.gov/data/),<br>GTFS Feeds |
 | vulnerable_worker_category | Score 0/1/2 indicating low/medium/high vulnerability of people working around transit stop | int | [Apache 2.0](Apache2.0) | [LEHD Origin-Destination Statistics](https://lehd.ces.census.gov/data/),<br>[OpenStreetMap](https://www.openstreetmap.org/about),<br>[CDC/ATSDR Social Vulnerability Index](https://www.atsdr.cdc.gov/placeandhealth/svi/data_documentation_download.html),<br>GTFS Feeds |
@@ -56,9 +55,9 @@ To discuss tailored adaptations of TREC to your team/city, please email us at da
 All data accessed as of June 26th, 2023.
 
 - Census tract, block geographies retrieved using [TidyCensus](https://walker-data.com/tidycensus/) with data by [US Census Bureau](https://www.census.gov/) 
-- Flood risk data from [First Street Climate-Adjusted Flood Risk - US Aggregate Data](https://aws.amazon.com/marketplace/pp/prodview-r36lzzzjacd32?sr=0-1&ref_=beagle&applicationId=AWSMPContessa#overview) by [First Street Foundation](https://firststreet.org/), [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)
-- Heat risk data from [First Street Climate-Adjusted Heat Risk - US Aggregate Data](https://aws.amazon.com/marketplace/pp/prodview-juylajmn3mixo?sr=0-1&ref_=beagle&applicationId=AWSMPContessa#overview) by [First Street Foundation](https://firststreet.org/), [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)
-- Fire risk data from [First Street Climate-Adjusted Wildfire Risk - US Aggregate Data](https://aws.amazon.com/marketplace/pp/prodview-crz65ioiwanoc?sr=0-1&ref_=beagle&applicationId=AWSMPContessa#overview) by [First Street Foundation](https://firststreet.org/), [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)
+- Flood Risk Factor from [First Street Climate-Adjusted Flood Risk - US Aggregate Data](https://aws.amazon.com/marketplace/pp/prodview-r36lzzzjacd32?sr=0-1&ref_=beagle&applicationId=AWSMPContessa#overview) by [First Street Foundation](https://firststreet.org/), [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)
+- Heat Risk Factor from [First Street Climate-Adjusted Heat Risk - US Aggregate Data](https://aws.amazon.com/marketplace/pp/prodview-juylajmn3mixo?sr=0-1&ref_=beagle&applicationId=AWSMPContessa#overview) by [First Street Foundation](https://firststreet.org/), [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)
+- Fire Risk Factor from [First Street Climate-Adjusted Wildfire Risk - US Aggregate Data](https://aws.amazon.com/marketplace/pp/prodview-crz65ioiwanoc?sr=0-1&ref_=beagle&applicationId=AWSMPContessa#overview) by [First Street Foundation](https://firststreet.org/), [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)
 - Employee location data from [LEHD Origin-Destination Employment Statistics](https://lehd.ces.census.gov/data/) by [US Census Bureau](https://www.census.gov/)
 - Social vulnerability data from [CDC/ATSDR Social Vulnerability Index](https://www.atsdr.cdc.gov/placeandhealth/svi/data_documentation_download.html) by [Centers for Disease Control and Prevention Agency for Toxic Substances and Disease Registry](https://www.atsdr.cdc.gov/) 
 - Street networks retrieved using [OSMnx](https://github.com/gboeing/osmnx) with data by [OpenStreetMap](https://www.openstreetmap.org/about), [Open Database License, "ODbL" 1.0](http://www.opendatacommons.org/licenses/odbl/) 
@@ -67,8 +66,8 @@ All data accessed as of June 26th, 2023.
 - New York City stormwater flood map data was downloaded from [NYC Department of Environmental Protection](http://nyc.gov/stormwater-map)
 - Hampton Roads sea-level rise planning scenarios were downloaded from [Hampton Roads Geographic Exchange Online](https://www.hrgeo.org/search?q=SLR)
 - Static Transit Feeds accessed via [Transitland](https://www.transit.land/), [API Terms](https://www.transit.land/terms)
-- New York City transit stops data was downloaded from [Metropolitan Transportation Authority](http://web.mta.info/developers/developer-data-terms.html#data) 
-- Hampton Roads transit stops were downloaded from [Hampton Roads Transit](https://gtfs.gohrt.com/)
+
+For list of GTFS feeds used and ther respective terms, refer to the [file](transit_feed_licenses.xlsx).
 
 ### Data Clinic
 [Data Clinic](https://www.twosigma.com/data-clinic/) is the data and tech-for-good arm of [Two Sigma](https://twosigma.com), a financial sciences company headquartered in NYC. Since Data Clinic was founded in 2014, we have provided pro bono data science and engineering support to mission-driven organizations around the world via close partnerships that pair Two Sigma's talent and way of thinking with our partner's rich content-area expertise. To scale the solutions and insights Data Clinic has gathered over the years, and to contribute to the democratization of data, we also engage in the development of open source tooling and data products.