HOTFIX: Rewrite Marin data scraper to use Socrata #167
Conversation
Marin now publishes COVID data in an actual data portal with a real API and changed the dashboard to use Tableau. That broke our scraper (because all the charts are now built in an entirely different way). The best way to fix was to rewrite on top of the Socrata data portal API, which we should happily expect to be much more stable. This also adds caching to the Socrata API so we can make multiple calls to the same URL without actually making multiple HTTP requests. The Marin data is arranged such that a lot of different dimensions of data are combined (unlike most other portals where they are separated), and caching lets us keep the logic straightforward without making unnecessary repeated requests. This also turns out to fix #162 -- unknowns are now included in the data! Fixes #165.
|
This looks really good @Mr0grog! One question I have is whether there's a value in pulling down the "Total Hospitalized" stats. I haven't cross-checked with the state data that the hospital scraper pulls; perhaps we're already covered, although Marin has demographic breakdowns. Just asking because the data is there. Otherwise, I think it's good to go! |
|
@benghancock good point! I think I avoided it because we don’t have any standard around it and adding another timeseries is a lot of extra data which could be an issue for the front-end project. It does seem kinda disappointing to leave out when we have it, though. Since this is a hotfix, I’m going to take your comment as an approval and merge as-is so we’ve got something working, but I’ll file an issue to add hospitalization. You or anyone else can implement. |
Mr0grog
deleted the
hotfix-marin-charts-are-now-tableu-slash-hallelujah-marin-published-raw-data
branch
|
Filed the issue as #170. |
Marin now publishes COVID data in an actual data portal with a real API and changed the dashboard to use Tableau. That broke our scraper (because all the charts are now built in an entirely different way). The best way to fix was to rewrite on top of the Socrata data portal API, which we should happily expect to be much more stable.
This also adds caching to the Socrata API so we can make multiple calls to the same URL without actually making multiple HTTP requests. The Marin data is arranged such that a lot of different dimensions of data are combined (unlike most other portals where they are separated), and caching lets us keep the logic straightforward without making unnecessary repeated requests.
This also turns out to fix #162 -- unknowns are now included in the data!
Fixes #165.