Add Sonoma Data Scraper

README.md

@@ -9,9 +9,159 @@ To install this project, you can simply run `./install.sh` in your terminal. Thi
 ## Running the scraper
 To run the scraper, you can use the run script by typing `sh run_scraper.sh` into your terminal. This will enable the virtual environment and run `scraper.py`. Once again, the virtual environment will not stay active after the script finishes running. If you want to run the scraper without the run script, enable the virtual environment, then run `python3 scraper.py`.
 
-## Running the API
-The best way to run the API right now is to run the command `FLASK_APP="app.py" FLASK_ENV=development flask run;`. Note that this is not the best way to run the scraper at this time.
-
+## Data Model
+The following sections document the differences between the counties in the common data model (see `data_models` directory) which we will see as we begin to get data from them.
+
+### Ages
+
+Please make sure to use the following age brackets for the different counties. Note that the brackets may also vary by whether you are scraping cases or deaths data:
+
+
+#### San Francisco
+##### Cases
+    "age": [
+        {"group": "18_and_under", "raw_count": -1 },
+        {"group": "18_to_30", "raw_count": -1 },
+        {"group": "31_to_40", "raw_count": -1 },
+        {"group": "41_to_50", "raw_count": -1 },
+        {"group": "51_to_60", "raw_count": -1 },
+        {"group": "61_to_70", "raw_count": -1 },
+        {"group": "71_to_80", "raw_count": -1 },
+        {"group": "81_and_older", "raw_count": -1}
+        ]
+##### Deaths
+Data broken down by gender is not available on the json files, only on the dashboard.
+
+
+#### Alameda
+##### Cases
+    "age": [
+        {"group": "18_and_under", "raw_count": -1 },
+        {"group": "18_to_30", "raw_count": -1 },
+        {"group": "31_to_40", "raw_count": -1 },
+        {"group": "41_to_50", "raw_count": -1 },
+        {"group": "51_to_60", "raw_count": -1 },
+        {"group": "61_to_70", "raw_count": -1 },
+        {"group": "71_to_80", "raw_count": -1 },
+        {"group": "81_and_older", "raw_count": -1 },
+        {"group": "Unknown", "raw_count": -1 }
+        ]
+##### Deaths
+Data broken down by gender is not available.
+
+
+#### Sonoma
+##### Cases
+    "age": [
+        {"group": "0_to_17", "raw_count": -1 },
+        {"group": "18_to_49", "raw_count": -1 },
+        {"group": "50_to_64", "raw_count": -1 },
+        {"group": "65_and_older", "raw_count": -1 },
+        {"group": "Unknown", "raw_count": -1 }
+        ]
+##### Deaths
+Data broken down by gender is not available.
+
+
+#### Santa Clara
+##### Cases
+    "age": [
+        {"group": "20_and_under", "raw_count": -1 },
+        {"group": "21_to_30", "raw_count": -1 },
+        {"group": "31_to_40", "raw_count": -1 },
+        {"group": "41_to_50", "raw_count": -1 },
+        {"group": "51_to_60", "raw_count": -1 },
+        {"group": "61_to_70", "raw_count": -1 },
+        {"group": "71_to_80", "raw_count": -1 },
+        {"group": "81_to_90", "raw_count": -1 },
+        {"group": "90_and_older", "raw_count": -1 },
+        {"group": "Unknown", "raw_count": -1 }
+        ]
+##### Deaths
+    "age": [
+        {"group": "20_and_under", "raw_count": -1 },
+        {"group": "21_to_30", "raw_count": -1 },
+        {"group": "31_to_40", "raw_count": -1 },
+        {"group": "41_to_50", "raw_count": -1 },
+        {"group": "51_to_60", "raw_count": -1 },
+        {"group": "61_to_70", "raw_count": -1 },
+        {"group": "71_to_80", "raw_count": -1 },
+        {"group": "81_to_90", "raw_count": -1 },
+        {"group": "90_and_older", "raw_count": -1 }
+        ]        
+
+
+#### San Mateo
+##### Cases
+    "age": [
+        {"group": "0_to_19", "raw_count": -1 },
+        {"group": "20_to_29", "raw_count": -1 },
+        {"group": "30_to_39", "raw_count": -1 },
+        {"group": "40_to_49", "raw_count": -1 },
+        {"group": "50_to_59", "raw_count": -1 },
+        {"group": "60_to_69", "raw_count": -1 },
+        {"group": "70_to_79", "raw_count": -1 },
+        {"group": "80_to_89", "raw_count": -1 },
+        {"group": "90_and_older", "raw_count": -1 }
+        ]  
+##### Deaths
+    age": [
+        {"group": "0_to_19", "raw_count": -1 },
+        {"group": "20_to_29", "raw_count": -1 },
+        {"group": "30_to_39", "raw_count": -1 },
+        {"group": "40_to_49", "raw_count": -1 },
+        {"group": "50_to_59", "raw_count": -1 },
+        {"group": "60_to_69", "raw_count": -1 },
+        {"group": "70_to_79", "raw_count": -1 },
+        {"group": "80_to_89", "raw_count": -1 },
+        {"group": "90_and_older", "raw_count": -1 }
+        ]  
+
+
+#### Contra Costa
+##### Cases
+    age": [
+        {"group": "0_to_20", "raw_count": -1 },
+        {"group": "21_to_40", "raw_count": -1 },
+        {"group": "41_to_60", "raw_count": -1 },
+        {"group": "61_to_80", "raw_count": -1 },
+        {"group": "81_to_100", "raw_count": -1 }
+        ]
+##### Deaths
+Data broken down by gender is not available.
+
+
+#### Marin
+##### Cases and Deaths
+    age": [
+        {"group": "0_to_18", "raw_count": -1 },
+        {"group": "19_to_34", "raw_count": -1 },
+        {"group": "35_to_49", "raw_count": -1 },
+        {"group": "50_to_64", "raw_count": -1 },
+        {"group": "65_and_older", "raw_count": -1 }
+        ]
+
+
+
+#### Solano
+##### Cases and Deaths
+    age": [
+        {"group": "0_to_18", "raw_count": -1 },
+        {"group": "19_to_64", "raw_count": -1 },
+        {"group": "65_and_older", "raw_count": -1 }
+        ]
+
+
+#### Napa
+##### Cases
+    age": [
+        {"group": "0_to_17", "raw_count": -1 },
+        {"group": "18_to_49", "raw_count": -1 },
+        {"group": "50_to_64", "raw_count": -1 },
+        {"group": "Over_64", "raw_count": -1 }
+        ]
+##### Deaths
+Data broken down by gender is not available.
 ## Development
 
 We use CircleCI to lint the code and run tests in this repository, but you can (and should!) also run tests locally.

covid19_sfbayarea/data/__init__.py

@@ -1,6 +1,7 @@
 from typing import Dict, Any
 from . import alameda
 from . import san_francisco
+from . import sonoma
 from . import solano
 
 scrapers: Dict[str, Any] = {
@@ -11,6 +12,6 @@
     'san_francisco': san_francisco,
     # 'san_mateo': None,
     # 'santa_clara': None,
+    'sonoma': sonoma,
     'solano': solano,
-    # 'sonoma': None,
 }