Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incorporate summaries bulk data #39

Open
waldoj opened this issue Jan 6, 2018 · 1 comment
Open

Incorporate summaries bulk data #39

waldoj opened this issue Jan 6, 2018 · 1 comment
Labels
enhancement

Comments

@waldoj
Copy link
Member

waldoj commented Jan 6, 2018

Right now we're scraping every summary. The legislature has started publishing Summaries.csv on their FTP site, with these headers:

"SUM_BILNO","SUMMARY_DOCID","SUMMARY_TYPE","SUMMARY_TEXT"

Get all summaries from this file, and stop scraping them with summaries.php.
waldoj added a commit that referenced this issue Jan 6, 2018
@waldoj
Take a first crack at implementing bulk summary data
6d9a3f4 
Toward #39. [ci skip]
waldoj added a commit that referenced this issue Jan 6, 2018
@waldoj
Allow the new summaries-gathering code to run
71f4590 
Toward #39.
waldoj added a commit that referenced this issue Jan 7, 2018
@waldoj
Incorporate bug fixes
c85248e 
A bunch of changes based on testing. Toward #39.
waldoj added a commit that referenced this issue Jan 7, 2018
@waldoj
Lower the verbosity of missing bills
e61ca2b 
Turns out this is common. Toward #39.
waldoj added a commit that referenced this issue Jan 7, 2018
@waldoj
Fix a PHP error
2d36282 
Toward #39.
@waldoj waldoj closed this as completed in 7037282 Jan 7, 2018
@waldoj waldoj reopened this Jan 7, 2018
@waldoj
Copy link
Member Author

waldoj commented Jan 7, 2018

Hang on—I'm dubious that the hash-checking system is working here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@waldoj