VACCINATIONS: Contributing to automations #2085
Replies: 25 comments
-
|
Hi @lucasrodes You've perfectly understood how our systems works! I've now added You're more than welcome to contribute to the automation effort of course! We had a pull-request a few days ago from someone who automated the collection for Denmark. For many countries the data is so scattered across newspaper articles and tweets that it's hard to find a systematic pattern right now, but other countries might already prove more doable (e.g. Italy). |
Beta Was this translation helpful? Give feedback.
-
|
Hi @edomt , In this regard, I have checked the file
My assumptions are:
Not sure if I got it right... Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Again, your assumptions are right :)
|
Beta Was this translation helpful? Give feedback.
-
|
@lucasrodes Note that the collection for Italy is now automated thanks to @bergwald's PR here: #251 My hope is that at some point the vaccination data will also be added to the official Italian repo that we're using for the testing data (https://github.com/pcm-dpc/COVID-19). |
Beta Was this translation helpful? Give feedback.
-
|
@edomt In |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @bergwald — I've now reimplemented the automated collection with that file (sorry your script was so short-lived!). The incremental collection worked perfectly but using the full time series is much safer in case they make retroactive corrections. |
Beta Was this translation helpful? Give feedback.
-
|
You're welcome! @edomt Having full-access to databases like the Italian one is ideal since it lowers the costs of creating and maintaining code. No need to search for obscure XPaths and CSS selectors! Let me know if you need any help with other automation or data-related issues. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks gang! Your reward is assured if/when we ever get to meet up in person... Permit me a quick question on the "stability" of the time series - do you anticipate the structure changing any time soon? I'm building against the existing "corrected" time series and want to ensure that it is relatively stable (as all your other datasets have been) |
Beta Was this translation helpful? Give feedback.
-
|
Hi @DrBazUK! Thanks for the kind words. :) In terms of structure, our most stable file is The public But generally we try to be very careful not to remove or rename columns, because quite a few people are relying on our data for automated scripts, and we don't want to break them. Column order can change though, so columns should always be parsed and processed by name in the header, and not by position. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @edomt - a beer or a Pastis is in certainly in order when all this has passed! Adding columns won't impact my present (limited) use case so I'm happy there 👍 but will need to take into account any changes to the order... something to think about. I'm still a data wrangling neophyte when it comes to Python/Jupyter but am motivated to learn more... |
Beta Was this translation helpful? Give feedback.
-
|
I am not sure if this will help, but for the Kingdom of Bahrain it is possible to automate the results from https://healthalert.gov.bh/en/category/daily-covid-19-report which include clear CSS IDs of the values needed instead of relaying on the Twitter/Instagram post |
Beta Was this translation helpful? Give feedback.
-
|
Hi @alarrayed! Thank you, but I'm not seeing a counter for vaccination doses on that page (there are many other counters though). Can you check? |
Beta Was this translation helpful? Give feedback.
-
|
Automated Bermuda vaccination data pull. Can you please merge it with main branch. Source file : bermuda.py |
Beta Was this translation helpful? Give feedback.
-
|
Automated Western Sahara vaccination data pull. Can you please merge it with main branch. Source file : westernsahara.py |
Beta Was this translation helpful? Give feedback.
-
|
We've started to collect information (rarely made available, though) on doses by manufacturer. Now, each automation script can also output a file in this folder: https://github.com/owid/covid-19-data/tree/master/scripts/scripts/vaccinations/automations/output/by_manufacturer See the Italy script for example: https://github.com/owid/covid-19-data/blob/master/scripts/scripts/vaccinations/automations/batch/italy.py The resulting files are concatenated to produce this output: https://github.com/owid/covid-19-data/blob/master/public/data/vaccinations/vaccinations-by-manufacturer.csv We don't want this to become resource-intensive and painful to maintain, so we'll only add it for countries that publish really clean files with the necessary information, i.e. our cc @kokes |
Beta Was this translation helpful? Give feedback.
-
|
The Philippines Does have a official vaccine tracker. maybe this could be automated? here is the website where i found the tracker: https://www.covid19.gov.ph/ |
Beta Was this translation helpful? Give feedback.
-
|
One of my colleagues identified this as the source for updates about Palau (PLW): http://www.palauhealth.org/2019nCoV_SitRep/MOH-COVID-19 Situation Report.pdf 26 April: We aren't yet clear on how often this updates but possibly daily. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, @DrBazUK, UPDATE: Palau numbers are included in US numbers. |
Beta Was this translation helpful? Give feedback.
-
|
Qatar’s how now includes first and second dose numbers. https://covid19.moph.gov.qa/EN/Pages/Vaccination-Program-Data.aspx It looks like it’s the page you are already using. |
Beta Was this translation helpful? Give feedback.
-
|
FYI - the filepath mentioned at the top of this thread for automation_state.csv now results in a 404 error. The new filepath seems to be this. |
Beta Was this translation helpful? Give feedback.
-
|
The source for Saudi Arabia seems to have stopped updating (https://coronamap.sa/Home/GetVaccineCountryInfo?countryname=Saudi%20Arabia). Previously it was updating every few hours but it hasn't updated since the 15th of May. Up to date vaccinations numbers can be found here: |
Beta Was this translation helpful? Give feedback.
-
|
Hello, can you automate the Philippines from this source? This page posts photos of vaccination update almost everyday. Not sure if you allow fb pages as sources but I'm just asking if it's allowed. Thanks |
Beta Was this translation helpful? Give feedback.
-
|
Hi @TetraBiblos991, we currently only automate data imports from websites publishing the data in plain text or providing downloadable files with the data (CSVs, JSONs, XLS, etc.). We do not support data extraction from images. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I have found a covid tracker that regularly updates the vaccinations administered in the Philippines. I hope this can be automated. |
Beta Was this translation helpful? Give feedback.
-
|
Hello again, sorry for continuous comment. I have now found the official vaccine tracker for the Philippines. It's updating every day. I hope this can be automated. Thanks |
Beta Was this translation helpful? Give feedback.
-
As far as I know, there are currently three approaches to automate vaccination data updates:
The data is manually updated using internal OWID tools. To contribute for manual updates, report new data in VACCINATIONS: how to add new countries or entries to our data #230 or open a new issue.
UPDATE: There is now the file
automation_state.csvwhich lists all countries with automated processes (thanks @edomt)UPDATE: Read Contribute section for more details. ℹ️
Beta Was this translation helpful? Give feedback.
All reactions