Skip to content

Commit

Permalink
HOTFIX: Rewrite Marin data scraper to use Socrata (#167)
Browse files Browse the repository at this point in the history
Marin now publishes COVID data in an actual data portal with a real API and changed the dashboard to use Tableau. That broke our scraper (because all the charts are now built in an entirely different way). The best way to fix was to rewrite on top of the Socrata data portal API, which we should happily expect to be much more stable.

This also adds caching to the Socrata API so we can make multiple calls to the same URL without actually making multiple HTTP requests. The Marin data is arranged such that a lot of different dimensions of data are combined (unlike most other portals where they are separated), and caching lets us keep the logic straightforward without making unnecessary repeated requests.

Fixes #165.
Fixes #162 (unknowns are now included in the data!).
  • Loading branch information
Mr0grog authored Jan 11, 2021
1 parent 6c4585b commit c225690
Show file tree
Hide file tree
Showing 2 changed files with 283 additions and 282 deletions.
Loading

0 comments on commit c225690

Please sign in to comment.