Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

It appears that aph.gov.au might be blocking our scraper #1477

Open
mlandauer opened this issue Feb 14, 2024 · 1 comment
Open

It appears that aph.gov.au might be blocking our scraper #1477

mlandauer opened this issue Feb 14, 2024 · 1 comment

Comments

@mlandauer
Copy link
Member

We wrote the following message to aph by filling in the form at https://www.aph.gov.au/Help/Contact_Us

I’m Matthew Landauer, one of the founders of the charity the OpenAustralia Foundation, a library of Parliamentary, Government and Public information, including OpenAustralia (https://www.openaustralia.org.au) and They Vote For You (https://theyvoteforyou.org.au/).

Both of these services depend on Hansard data that we scrape from aph.gov.au and parlinfo.aph.gov.au. We appreciate the ongoing cooperation and support of APH in facilitating access to Hansard running and maintaining those scrapers for the last 15 years. Most of the time this works seamlessly, and from time to time we report errors to APH and work with notifications of errors to correct errors in the records.

We recently discovered a problem that is preventing the Hansard from being scraped. This problem started around the beginning of December 2023.

We are now seeing regular but intermittent 403 (forbidden) errors on scraping parlinfo urls which work fine when checked in the browser. Here is one such url as an example:

https://parlinfo.aph.gov.au/parlInfo/search/display/display.w3p;adv=yes;orderBy=_fragment_number,doc_date-rev;page=0;query=Dataset%3Ahansards,hansards80%20Date%3A21%2F1%2F2024;rec=0;resCount=Default

Is there by any chance any anti-scraping or anti-bot technology that’s recently been put in place that could be interfering with our scrapers? Could you please unblock us? Alternatively, if it's stemming from another intermittent technical problem, we kindly request your guidance. For example, If there's a workaround available, we would greatly appreciate any guidance you can offer.

We’d really appreciate your help trying to resolve this issue

All the best,
Matthew Landauer

I put in some very hacky changes to the parser which makes things a little better but the parser needs to be constantly restarted in openaustralia/openaustralia-parser@906431b

Copy link

stale bot commented Aug 16, 2024

This issue has been automatically marked as stale because there has been no activity on it for about six months. If you want to keep it open please make a comment and explain why this issue is still relevant. Otherwise it will be automatically closed in a week. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants