EDF to ballot data model extractor #112
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BallotStyles don't drive the contests.
Fixes BallotLab TrustTheVote-Project#83.
Fixes BallotLab TrustTheVote-Project#86.
Fixes BallotLab TrustTheVote-Project#85. This isn't quite right: fails if a slate has candidates from different parties. Deferring on that case for now.
Mark write-in status as boolean (matching the contents of the EDF). No issue attached.
The error was introduced in the process of folding write-ins into contests. - Remove convoluted extra step that makes it harder to do this correctly. - Candidate names are grouped in arrays. Now do the same for parties. - Use 'is_write_in' for the write-in boolean field.
Handles case where candidates in a slate have different parties.
Fixes BallotLab TrustTheVote-Project#86.
Partly to share fields but primarily for correctness: contests have to be a single list independent of sub-type because that's how EDFs work. Splitting the types into multiple lists would lose the order.
…model. Do some minor code cleanup.
An empty dictionary adds a special case to 'PartyData' construction.
- Unify extraction functions: - Use direct field accesses for properties, index for references. - Yield data dictionaries instead of returning lists. - Add in election data. - Entry point is now 'extract_ballot_data', which returns a list.
The caller only receives an 'ElectionData' back.
- Public API is now 'extract'. - Prefix all internal methods as private. - Prefix utility functions but don't make them methods. - Make the element index an instance variable. - Re-wrap text to fit within line length. Note: In a class method order usually puts public functions at the top. Not doing that here to avoid making the change diff useless.
|
This is what running the extractor over the September test case produces: [
{
"name": "General Election",
"type": "general",
"start_date": "2024-11-05",
"end_date": "2024-11-05",
"ballot_styles": [
{
"id": "precinct_1_downtown",
"scopes": [
"recFIehh5Aj0zGTn6"
],
"contests": [
{
"id": "recsoZy7vYhS3lbcK",
"type": "candidate",
"title": "President of the United States",
"district": "United States of America",
"vote_type": "plurality",
"votes_allowed": 1,
"candidates": [
{
"id": "recQK3J9IJq42hz2n",
"name": [
"Anthony Alpha",
"Betty Beta"
],
"party": [
{
"name": "The Lepton Party",
"abbreviation": "LEP"
}
],
"is_write_in": false
},
{
"id": "reccUkUdEznfODgeL",
"name": [
"Gloria Gamma",
"David Delta"
],
"party": [
{
"name": "The Hadron Party of Farallon",
"abbreviation": "HAD"
}
],
"is_write_in": false
},
{
"id": "recPod2L8VhwagiDl",
"name": [],
"party": [],
"is_write_in": true
}
]
},
{
"id": "recthF6jdx5ybBNkC",
"type": "candidate",
"title": "Gadget County School Board",
"district": "Gadget County",
"vote_type": "n-of-m",
"votes_allowed": 4,
"candidates": [
{
"id": "recJvikmG5MrUKzo1",
"name": [
"Rosashawn Davis"
],
"party": [],
"is_write_in": false
},
{
"id": "recigPkqYXXDJEaCE",
"name": [
"Hector Gomez"
],
"party": [],
"is_write_in": false
},
{
"id": "recbN7UUMaSuOYGQ6",
"name": [
"Glavin Orotund"
],
"party": [],
"is_write_in": false
},
{
"id": "recbxvhKikHJNZYbq",
"name": [
"Sally Smith"
],
"party": [],
"is_write_in": false
},
{
"id": "recvjB3rgfiicf0RP",
"name": [
"Oliver Tsi"
],
"party": [],
"is_write_in": false
},
{
"id": "recYurH2CLY3SlYS8",
"name": [],
"party": [],
"is_write_in": true
},
{
"id": "recI5jfcXIsbAKytC",
"name": [],
"party": [],
"is_write_in": true
},
{
"id": "recn9m0o1em7gLahj",
"name": [],
"party": [],
"is_write_in": true
}
]
},
{
"id": "recIj8OmzqzzvnDbM",
"type": "candidate",
"title": "Contest for Mayor of Orbit City",
"district": "Orbit City",
"vote_type": "plurality",
"votes_allowed": 1,
"candidates": [
{
"id": "recKD6dBvkNhEU4bg",
"name": [
"Spencer Cogswell"
],
"party": [
{
"name": "The Hadron Party of Farallon",
"abbreviation": "HAD"
}
],
"is_write_in": false
},
{
"id": "recTKcXLCzRvKB9U0",
"name": [
"Cosmo Spacely"
],
"party": [
{
"name": "The Lepton Party",
"abbreviation": "LEP"
}
],
"is_write_in": false
},
{
"id": "recqq21kO6HWgpJZV",
"name": [],
"party": [],
"is_write_in": true
}
]
},
{
"id": "recqPa7AeyufIfd6k",
"type": "ballot measure",
"title": "Air Traffic Control Tax Increase",
"district": "Gadget County",
"text": "Shall Gadget County increase its sales tax from 1% to 1.1% for the purpose of raising additional revenue to fund expanded air traffic control operations?",
"choices": [
{
"id": "recysACFx8cgwomBE",
"choice": "Yes"
},
{
"id": "recabXA9jzFYRmGXy",
"choice": "No"
}
]
},
{
"id": "recWjDBFeafCdklWq",
"type": "ballot measure",
"title": "Constitutional Amendment",
"district": "The State of Farallon",
"text": "Do you approve amending the Constitution to legalize the controlled use of helium balloons? Only adults at least 21 years of age could use helium. The State commission created to oversee the State's medical helium program would also oversee the new, personal use helium market.Helium balloons would be subject to the State sales tax. If authorized by the Legislature, a municipality may pass a local ordinance to charge a local tax on helium balloons.\nAll forms of helium are authorized for personal use or commercial balloon usage. Use of deuterium and tritium for energy generation will be licensed by the State commission. Use of deuterium and tritium for uncontrolled fusion reactions is strictly prohibited, and constitutes a Class A felony under State Criminal code section 4.222 K.\n",
"choices": [
{
"id": "rec7mVWjUH6fmDxig",
"choice": "Yes"
},
{
"id": "reccIHOhUfJgJkqS7",
"choice": "No"
}
]
}
]
},
{
"id": "precinct_4_bedrock",
"scopes": [
"recSQ3ZpvJlTll1Ve"
],
"contests": [
{
"id": "recsoZy7vYhS3lbcK",
"type": "candidate",
"title": "President of the United States",
"district": "United States of America",
"vote_type": "plurality",
"votes_allowed": 1,
"candidates": [
{
"id": "recQK3J9IJq42hz2n",
"name": [
"Anthony Alpha",
"Betty Beta"
],
"party": [
{
"name": "The Lepton Party",
"abbreviation": "LEP"
}
],
"is_write_in": false
},
{
"id": "reccUkUdEznfODgeL",
"name": [
"Gloria Gamma",
"David Delta"
],
"party": [
{
"name": "The Hadron Party of Farallon",
"abbreviation": "HAD"
}
],
"is_write_in": false
},
{
"id": "recPod2L8VhwagiDl",
"name": [],
"party": [],
"is_write_in": true
}
]
},
{
"id": "recthF6jdx5ybBNkC",
"type": "candidate",
"title": "Gadget County School Board",
"district": "Gadget County",
"vote_type": "n-of-m",
"votes_allowed": 4,
"candidates": [
{
"id": "recJvikmG5MrUKzo1",
"name": [
"Rosashawn Davis"
],
"party": [],
"is_write_in": false
},
{
"id": "recigPkqYXXDJEaCE",
"name": [
"Hector Gomez"
],
"party": [],
"is_write_in": false
},
{
"id": "recbN7UUMaSuOYGQ6",
"name": [
"Glavin Orotund"
],
"party": [],
"is_write_in": false
},
{
"id": "recbxvhKikHJNZYbq",
"name": [
"Sally Smith"
],
"party": [],
"is_write_in": false
},
{
"id": "recvjB3rgfiicf0RP",
"name": [
"Oliver Tsi"
],
"party": [],
"is_write_in": false
},
{
"id": "recYurH2CLY3SlYS8",
"name": [],
"party": [],
"is_write_in": true
},
{
"id": "recI5jfcXIsbAKytC",
"name": [],
"party": [],
"is_write_in": true
},
{
"id": "recn9m0o1em7gLahj",
"name": [],
"party": [],
"is_write_in": true
}
]
},
{
"id": "recWjDBFeafCdklWq",
"type": "ballot measure",
"title": "Constitutional Amendment",
"district": "The State of Farallon",
"text": "Do you approve amending the Constitution to legalize the controlled use of helium balloons? Only adults at least 21 years of age could use helium. The State commission created to oversee the State's medical helium program would also oversee the new, personal use helium market.Helium balloons would be subject to the State sales tax. If authorized by the Legislature, a municipality may pass a local ordinance to charge a local tax on helium balloons.\nAll forms of helium are authorized for personal use or commercial balloon usage. Use of deuterium and tritium for energy generation will be licensed by the State commission. Use of deuterium and tritium for uncontrolled fusion reactions is strictly prohibited, and constitutes a Class A felony under State Criminal code section 4.222 K.\n",
"choices": [
{
"id": "rec7mVWjUH6fmDxig",
"choice": "Yes"
},
{
"id": "reccIHOhUfJgJkqS7",
"choice": "No"
}
]
},
{
"id": "recqPa7AeyufIfd6k",
"type": "ballot measure",
"title": "Air Traffic Control Tax Increase",
"district": "Gadget County",
"text": "Shall Gadget County increase its sales tax from 1% to 1.1% for the purpose of raising additional revenue to fund expanded air traffic control operations?",
"choices": [
{
"id": "recysACFx8cgwomBE",
"choice": "Yes"
},
{
"id": "recabXA9jzFYRmGXy",
"choice": "No"
}
]
}
]
},
{
"id": "precinct_3_spaceport",
"scopes": [
"rec7dCergEa3mzqxy"
],
"contests": [
{
"id": "recsoZy7vYhS3lbcK",
"type": "candidate",
"title": "President of the United States",
"district": "United States of America",
"vote_type": "plurality",
"votes_allowed": 1,
"candidates": [
{
"id": "recQK3J9IJq42hz2n",
"name": [
"Anthony Alpha",
"Betty Beta"
],
"party": [
{
"name": "The Lepton Party",
"abbreviation": "LEP"
}
],
"is_write_in": false
},
{
"id": "reccUkUdEznfODgeL",
"name": [
"Gloria Gamma",
"David Delta"
],
"party": [
{
"name": "The Hadron Party of Farallon",
"abbreviation": "HAD"
}
],
"is_write_in": false
},
{
"id": "recPod2L8VhwagiDl",
"name": [],
"party": [],
"is_write_in": true
}
]
},
{
"id": "recXNb4zPrvC1m6Fr",
"type": "candidate",
"title": "Spaceport Control Board",
"district": "Aldrin Space Transport District",
"vote_type": "n-of-m",
"votes_allowed": 2,
"candidates": [
{
"id": "recBnJZEgCKAnfpNo",
"name": [
"Harlan Ellis"
],
"party": [],
"is_write_in": false
},
{
"id": "recwNuOnepWNGz67V",
"name": [
"Rudy Indexer"
],
"party": [],
"is_write_in": false
},
{
"id": "recvYvTb9hWH7tptb",
"name": [
"Jane Jetson"
],
"party": [],
"is_write_in": false
},
{
"id": "rec9Eev970VhohqKi",
"name": [],
"party": [],
"is_write_in": true
},
{
"id": "recFiGYjGCIyk5LBe",
"name": [],
"party": [],
"is_write_in": true
}
]
},
{
"id": "recqPa7AeyufIfd6k",
"type": "ballot measure",
"title": "Air Traffic Control Tax Increase",
"district": "Gadget County",
"text": "Shall Gadget County increase its sales tax from 1% to 1.1% for the purpose of raising additional revenue to fund expanded air traffic control operations?",
"choices": [
{
"id": "recysACFx8cgwomBE",
"choice": "Yes"
},
{
"id": "recabXA9jzFYRmGXy",
"choice": "No"
}
]
},
{
"id": "recWjDBFeafCdklWq",
"type": "ballot measure",
"title": "Constitutional Amendment",
"district": "The State of Farallon",
"text": "Do you approve amending the Constitution to legalize the controlled use of helium balloons? Only adults at least 21 years of age could use helium. The State commission created to oversee the State's medical helium program would also oversee the new, personal use helium market.Helium balloons would be subject to the State sales tax. If authorized by the Legislature, a municipality may pass a local ordinance to charge a local tax on helium balloons.\nAll forms of helium are authorized for personal use or commercial balloon usage. Use of deuterium and tritium for energy generation will be licensed by the State commission. Use of deuterium and tritium for uncontrolled fusion reactions is strictly prohibited, and constitutes a Class A felony under State Criminal code section 4.222 K.\n",
"choices": [
{
"id": "rec7mVWjUH6fmDxig",
"choice": "Yes"
},
{
"id": "reccIHOhUfJgJkqS7",
"choice": "No"
}
]
}
]
},
{
"id": "precinct_2_spacetown",
"scopes": [
"recUuJTc3tUIUvgF1"
],
"contests": [
{
"id": "recsoZy7vYhS3lbcK",
"type": "candidate",
"title": "President of the United States",
"district": "United States of America",
"vote_type": "plurality",
"votes_allowed": 1,
"candidates": [
{
"id": "recQK3J9IJq42hz2n",
"name": [
"Anthony Alpha",
"Betty Beta"
],
"party": [
{
"name": "The Lepton Party",
"abbreviation": "LEP"
}
],
"is_write_in": false
},
{
"id": "reccUkUdEznfODgeL",
"name": [
"Gloria Gamma",
"David Delta"
],
"party": [
{
"name": "The Hadron Party of Farallon",
"abbreviation": "HAD"
}
],
"is_write_in": false
},
{
"id": "recPod2L8VhwagiDl",
"name": [],
"party": [],
"is_write_in": true
}
]
},
{
"id": "recthF6jdx5ybBNkC",
"type": "candidate",
"title": "Gadget County School Board",
"district": "Gadget County",
"vote_type": "n-of-m",
"votes_allowed": 4,
"candidates": [
{
"id": "recJvikmG5MrUKzo1",
"name": [
"Rosashawn Davis"
],
"party": [],
"is_write_in": false
},
{
"id": "recigPkqYXXDJEaCE",
"name": [
"Hector Gomez"
],
"party": [],
"is_write_in": false
},
{
"id": "recbN7UUMaSuOYGQ6",
"name": [
"Glavin Orotund"
],
"party": [],
"is_write_in": false
},
{
"id": "recbxvhKikHJNZYbq",
"name": [
"Sally Smith"
],
"party": [],
"is_write_in": false
},
{
"id": "recvjB3rgfiicf0RP",
"name": [
"Oliver Tsi"
],
"party": [],
"is_write_in": false
},
{
"id": "recYurH2CLY3SlYS8",
"name": [],
"party": [],
"is_write_in": true
},
{
"id": "recI5jfcXIsbAKytC",
"name": [],
"party": [],
"is_write_in": true
},
{
"id": "recn9m0o1em7gLahj",
"name": [],
"party": [],
"is_write_in": true
}
]
},
{
"id": "recIj8OmzqzzvnDbM",
"type": "candidate",
"title": "Contest for Mayor of Orbit City",
"district": "Orbit City",
"vote_type": "plurality",
"votes_allowed": 1,
"candidates": [
{
"id": "recKD6dBvkNhEU4bg",
"name": [
"Spencer Cogswell"
],
"party": [
{
"name": "The Hadron Party of Farallon",
"abbreviation": "HAD"
}
],
"is_write_in": false
},
{
"id": "recTKcXLCzRvKB9U0",
"name": [
"Cosmo Spacely"
],
"party": [
{
"name": "The Lepton Party",
"abbreviation": "LEP"
}
],
"is_write_in": false
},
{
"id": "recqq21kO6HWgpJZV",
"name": [],
"party": [],
"is_write_in": true
}
]
},
{
"id": "recXNb4zPrvC1m6Fr",
"type": "candidate",
"title": "Spaceport Control Board",
"district": "Aldrin Space Transport District",
"vote_type": "n-of-m",
"votes_allowed": 2,
"candidates": [
{
"id": "recBnJZEgCKAnfpNo",
"name": [
"Harlan Ellis"
],
"party": [],
"is_write_in": false
},
{
"id": "recwNuOnepWNGz67V",
"name": [
"Rudy Indexer"
],
"party": [],
"is_write_in": false
},
{
"id": "recvYvTb9hWH7tptb",
"name": [
"Jane Jetson"
],
"party": [],
"is_write_in": false
},
{
"id": "rec9Eev970VhohqKi",
"name": [],
"party": [],
"is_write_in": true
},
{
"id": "recFiGYjGCIyk5LBe",
"name": [],
"party": [],
"is_write_in": true
}
]
},
{
"id": "recqPa7AeyufIfd6k",
"type": "ballot measure",
"title": "Air Traffic Control Tax Increase",
"district": "Gadget County",
"text": "Shall Gadget County increase its sales tax from 1% to 1.1% for the purpose of raising additional revenue to fund expanded air traffic control operations?",
"choices": [
{
"id": "recysACFx8cgwomBE",
"choice": "Yes"
},
{
"id": "recabXA9jzFYRmGXy",
"choice": "No"
}
]
},
{
"id": "recWjDBFeafCdklWq",
"type": "ballot measure",
"title": "Constitutional Amendment",
"district": "The State of Farallon",
"text": "Do you approve amending the Constitution to legalize the controlled use of helium balloons? Only adults at least 21 years of age could use helium. The State commission created to oversee the State's medical helium program would also oversee the new, personal use helium market.Helium balloons would be subject to the State sales tax. If authorized by the Legislature, a municipality may pass a local ordinance to charge a local tax on helium balloons.\nAll forms of helium are authorized for personal use or commercial balloon usage. Use of deuterium and tritium for energy generation will be licensed by the State commission. Use of deuterium and tritium for uncontrolled fusion reactions is strictly prohibited, and constitutes a Class A felony under State Criminal code section 4.222 K.\n",
"choices": [
{
"id": "rec7mVWjUH6fmDxig",
"choice": "Yes"
},
{
"id": "reccIHOhUfJgJkqS7",
"choice": "No"
}
]
}
]
}
]
}
] |
|
I need this on the |
stratofax
approved these changes
Sep 15, 2022
|
|
||
| All the current examples are taken from these EDF files: | ||
|
|
||
| - https://github.com/TrustTheVote-Project/NIST-1500-100-103-examples/blob/main/test_cases/june_test_case.json |
There was a problem hiding this comment.
Did you also test the September JSON?
There was a problem hiding this comment.
Yes, the example I posted above is from the September test case
README is out of date. It shouldn't have been in the PR. I forgot to delete it.
There was a problem hiding this comment.
No worries, I drafted this comment earlier today and didn't notice that you had specifically mentioned the Sept. JSON in your comments, above.
| @@ -0,0 +1,44 @@ | |||
| Data to use for BallotLab inputs. The data is extracted from EDF test cases | |||
There was a problem hiding this comment.
I know this is out of date, so any questions here are really just a matter of curiosity.
| file = opts.file | ||
| opts = vars(opts) | ||
|
|
||
| try: |
There was a problem hiding this comment.
This should be a separate file loader function outside of main() so that:
- main() is essentially handling the CLI input and passing it to the file loader
- This file loader could be called from the main ballotmaker CLI
Good enough for now, though!
There was a problem hiding this comment.
Sure. But there's not much to it, we can just copy it out. It's not a separate loader because I don't see the script as being used except to test out results. I presume the rest of BallotMaker will be managing files elsewhere and passing them in. You'll note that BallotDate.Extractor.extract takes a dictionary, not a file or a stream. I/O isn't part of this module.
I'm trying this out now on a branch: modifying ballotmaker demo so it takes a Path parameter and loads the test case from that parameter.
There was a problem hiding this comment.
Cool! Let me know how it works out
| # --- Utilities | ||
|
|
||
|
|
||
| def _text_content(item): |
There was a problem hiding this comment.
You know I'm a big fan of typing your function return values, so please apply this suggestion to all the following functions, too.
There was a problem hiding this comment.
Noted, and will do.
|
|
||
| def _text_content(item): | ||
| """Return joined lines from internationalized text.""" | ||
| assert isinstance(item, InternationalizedText) |
There was a problem hiding this comment.
Is this debugging code? If not, please add a descriptive text message for your AssertionError -- it will also serve to document the purpose of this test.
As you may already know, all assert statements can be disabled at runtime with -0, -00, or PYTHONOPTIMIZE
There was a problem hiding this comment.
Yes, I'm aware of it. It's an old check from the early revisions of this branch. I think I did it because I thought that would be annoying to debug and wanted to halt it when it happens. I deferred on fixing it because the right answer is to use type annotations, and I was out of time. I'll take it out.
| contest = BallotMeasureContestData(**contest) | ||
| elif contest["type"] == ContestType.CANDIDATE.value: | ||
| contest = CandidateContestData(**contest) | ||
| contests.append(contest) |
There was a problem hiding this comment.
Does this mean that Ballot Measure Contests and Candidate Contests could be intermingled in the contests list? Does that matter?
There was a problem hiding this comment.
Not only can they be intermingled they have to be. Splitting them up would lose the order they appear in the EDF.
However we can have our cake and eat it too. See the cached properties BallotStyleData.ballot_measure_contests and BallotStyleData.candidate_contests which extract them back out when you need them.
| @@ -0,0 +1,72 @@ | |||
| import pytest | |||
There was a problem hiding this comment.
Your tests seem quite rigorous, so I'm just going to skim then and not comment. I'll run them, of course!
| Drop either field from result if it isn't present. | ||
| """ | ||
| # Note: party ID is returned to allow de-duplicating parties in callers. | ||
| id_ = candidate.party_id |
There was a problem hiding this comment.
Consider the more descriptive & self-documenting party_id
| result["name"] = name | ||
| if abbreviation: | ||
| result["abbreviation"] = abbreviation | ||
| return result, id_ |
There was a problem hiding this comment.
Consider more descriptive variable names, like party_dict instead of result
| parties = [] | ||
| _party_ids = set() | ||
| if selection.candidate_ids: | ||
| for id_ in selection.candidate_ids: |
There was a problem hiding this comment.
Consider: candidate_id instead of id_
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The extractor now takes into account design changes made to the BallotMaker design diagram. It successfully generates data from the September test case, which I'll add in a comment.
Addresses #110.
The extractor has a script form which you can use to replicate the September data but BallotLab should make use of the library form. Examples follow below.
ElectionDataobject returned from extraction tracks the design diagram (and if anything is inconsistent or invalid between the two it's a bug). This means does not have any rendering fields that you might need for the PDF. Those can be added in. See below for an example.Notes
ballotmaker/datais out-of-date. I'm updating it but I didn't want to block on it. The updated README will come with examples of use.Examples
To run the script:
To use as a library:
To print:
To annotate with fields needed for PDF rendering:
ballotmaker/ballots/contest_data.py. It can go in__post_init__or be added in place by the layout code. My intuition is that the latter is more correct, but we can do whatever makes sense.