Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HOTFIX: Point hospitalization data scraper to new CHHS API resource #191

Merged
merged 1 commit into from
Mar 16, 2021

Conversation

benghancock
Copy link
Collaborator

Per data.ca.gov, that CKAN resource has been deprecated and replaced by one provided by CHHS. The values and the structure appear to be mostly the same, so this hotfix mainly just points our code to the new resource.

One note: It looks like the CHHS resource doesn't return as much meta data about each of the fields as the prior CA.gov resource. Here's what the raw data looks like for the fields key:

"fields": [
    {"type": "int", "id": "_id"}, 
    {"type": "text", "id": "county"}, 
    {"type": "text", "id": "todays_date"},
    {"type": "text", "id": "hospitalized_covid_confirmed_patients"}, 
    {"type": "text", "id": "hospitalized_suspected_covid_patients"}, 
    {"type": "text", "id": "hospitalized_covid_patients"}, 
    {"type": "text", "id": "all_hospital_beds"}, 
    {"type": "text", "id": "icu_covid_confirmed_patients"}, 
    {"type": "text", "id": "icu_suspected_covid_patients"}, 
    {"type": "text", "id": "icu_available_beds"}
]

I don't think that's super critical, since we're not really using most of those values on the front end anyhow, but worth being aware of.

In addition, I spotted two other things:

  • After diffing the data from CHHS with the latest pull that we had from the prior CA.gov resource, I noticed that the _id field values are different for same given date, for the given county. Probably to be expected and of no consequence for our purposes.

  • Floats are being represented as strings in the raw data (as one might infer from the "text" type in the field descriptions), which we previously hadn't encountered but I've now added handling for.

This script still needs some logging for automated alerts; I figure that can be handled in a separate PR.

Fixes #187

@benghancock benghancock requested a review from Mr0grog March 16, 2021 04:39
Copy link
Collaborator

@Mr0grog Mr0grog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 looks good to me! Between this and #190, I certainly don’t feel great about the state’s switch to CHHS data, though… 😕

@benghancock benghancock merged commit 24269d8 into sfbrigade:master Mar 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Hospital Data Scraper is Broken
2 participants