Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify parser to ingest new CSV format #114

Open
waldoj opened this issue Jul 25, 2016 · 2 comments
Open

Modify parser to ingest new CSV format #114

waldoj opened this issue Jul 25, 2016 · 2 comments
Labels

Comments

@waldoj
Copy link
Member

waldoj commented Jul 25, 2016

As of August 1, the SCC is publishing CSV instead of their gnarly fixed-width format. As of October 31, they will no longer publish the old format. So that provides a 3-month window in which to upgrade Crump to use CSV. That ought to mostly involve simplifying it a great deal.

waldoj added a commit that referenced this issue Aug 30, 2016
waldoj added a commit that referenced this issue Oct 26, 2016
waldoj added a commit that referenced this issue Oct 26, 2016
Change how we parse and output files, now that we know when one file
ends and another one starts. This hasn’t even been tested yet. Toward
#114.
waldoj added a commit that referenced this issue Oct 31, 2016
waldoj added a commit that referenced this issue Oct 31, 2016
waldoj added a commit that referenced this issue Oct 31, 2016
waldoj added a commit that referenced this issue Oct 31, 2016
waldoj added a commit that referenced this issue Oct 31, 2016
@waldoj
Copy link
Member Author

waldoj commented Oct 31, 2016

Next up: modify the file iterator so that, instead of ingesting them naively, they are ingested as CSV. Then modify:

line[name] = current_line[1][start:end].strip()

to instead strip and rename the field, but otherwise leave it alone.

waldoj added a commit that referenced this issue Nov 8, 2016
@waldoj
Copy link
Member Author

waldoj commented Nov 8, 2016

Now the problem is that CSV and JSON are both being streamed into a single file.

waldoj added a commit that referenced this issue Jun 17, 2017
waldoj added a commit that referenced this issue Jun 18, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant