Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EDF to ballot data model extractor #112

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
f38ced9
Script to convert NIST EDF data into BallotLab data.
ion-oset Aug 21, 2022
88aa5bc
First pass at candidate contests.
ion-oset Aug 21, 2022
41e188c
Ordered content includes headers; use argparse.
ion-oset Aug 21, 2022
25ff3cf
Flags for keeping write-ins, n-of-m contests.
ion-oset Aug 21, 2022
584bb41
Drop 'keep' flags, add '--debug' for exceptions, more doc comments.
ion-oset Aug 21, 2022
43eb184
Significant refactoring of candidate contests.
ion-oset Aug 21, 2022
24b921f
Add offices and parties to candidate contests.
ion-oset Aug 21, 2022
9f0ab55
Add ballot measure contests.
ion-oset Aug 21, 2022
f1fd991
Add ballot text; don't return offices or parties for candidates.
ion-oset Aug 21, 2022
309db0e
Add README for ballot-lab-data script.
ion-oset Aug 22, 2022
f5090b8
Add party names to candidates.
ion-oset Aug 22, 2022
55b5b38
Add votes allowed on candidates races (even if it's the default).
ion-oset Aug 23, 2022
a262592
Add party abbreviations to candidates.
ion-oset Aug 23, 2022
4a27316
Include '@id's with contests.
ion-oset Aug 23, 2022
15efb15
Use 'ContestSelectionId's instead of a count for write-ins.
ion-oset Aug 23, 2022
4355de7
Add selection IDs to all candidates, not just write-ins.
ion-oset Aug 24, 2022
26deeaf
Properly set up multi-candidate slates/tickets.
ion-oset Aug 24, 2022
fb03e3f
Merge write-ins into candidte contests.
ion-oset Aug 24, 2022
1547160
Multi-candidate slates: Fix dropping all candidates before the last one.
ion-oset Aug 27, 2022
78f933a
Multi-candidate slates: Use party IDs to collapse identical parties.
ion-oset Aug 27, 2022
e658646
Multi-candidate slates: Only collapse party IDs if ALL parties are id…
ion-oset Aug 27, 2022
d48ddf7
Drop null fields of parties; use an empty dictionary if a party is mi…
ion-oset Aug 27, 2022
4d1bfb4
Add IDs to ballot measures contests.
ion-oset Aug 27, 2022
8fd9337
Add IDs to ballot measure contest selections.
ion-oset Aug 29, 2022
6a88c50
Update README for 'ballot-lab-data' script.
ion-oset Aug 30, 2022
929a4be
Merge branch 'development' into 'edf-to-ballot-data'.
ion-oset Sep 9, 2022
827b2c9
Move 'scripts' to 'data' and rename script.
ion-oset Sep 9, 2022
8d6d431
Separate EDF to data into a script and a module.
ion-oset Sep 9, 2022
2bec0c1
Use explicit EDF model imports ballot data extractor.
ion-oset Sep 9, 2022
05fac8f
Simplify naming of ballot data extractor functions.
ion-oset Sep 9, 2022
67d3933
Data model for ballot measure choices.
ion-oset Sep 9, 2022
9921c2d
Data model for ballot measure contests.
ion-oset Sep 10, 2022
4e7283f
Data model for candidate parties.
ion-oset Sep 10, 2022
e099419
Data model for candidate choices.
ion-oset Sep 10, 2022
5dc5ca8
Data model for candidate contests.
ion-oset Sep 10, 2022
a112e03
Simplify test assertions on nested data by comparing dictionaries.
ion-oset Sep 12, 2022
7873833
Data model for ballot styles.
ion-oset Sep 12, 2022
3621f32
Data model for elections.
ion-oset Sep 12, 2022
893af78
Use a base type for contest models.
ion-oset Sep 12, 2022
e03c3bb
Add derived candidate and ballot measure contest properties for ballo…
ion-oset Sep 12, 2022
a7f19c1
Reorder contest fields in ballot data extractor to match ballot data …
ion-oset Sep 12, 2022
341df82
Don't return an empty party dictionary when there is no party.
ion-oset Sep 12, 2022
002e415
Re-combine candidate and ballot measure contests in extractor.
ion-oset Sep 13, 2022
b173842
Re-implement and simplify the ballot extractor interface.
ion-oset Sep 14, 2022
5ace8d6
Move document and index creation into 'report'.
ion-oset Sep 14, 2022
04e0d3e
Make 'ElectionReport' and 'ElementIndex' internal to the extractor.
ion-oset Sep 14, 2022
5350723
Document the entry point of the extractor API.
ion-oset Sep 14, 2022
7e2f748
Turn ballot extractor into a class.
ion-oset Sep 14, 2022
1195709
Rename extractor internal functions start with '_extract'.
ion-oset Sep 14, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions src/electos/ballotmaker/data/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
Data to use for BallotLab inputs. The data is extracted from EDF test cases
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this is out of date, so any questions here are really just a matter of curiosity.

and constrained to the data model Ballot Lab is using.

Output file naming format is `{test-case-source}_{ballot-style-id}.json`.
Note the use of `-` to separate words, and `_` to separate the name parts.

All the current examples are taken from these EDF files:

- https://github.com/TrustTheVote-Project/NIST-1500-100-103-examples/blob/main/test_cases/june_test_case.json
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you also test the September JSON?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the example I posted above is from the September test case

README is out of date. It shouldn't have been in the PR. I forgot to delete it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries, I drafted this comment earlier today and didn't notice that you had specifically mentioned the Sept. JSON in your comments, above.


To run it:

- Install the BallotLab fork and change to the 'edf-data-to-ballot' branch.

git clone https://github.com/ion-oset/BallotLab -b edf-data-to-ballot

- Install the project dependencies:

poetry install

- Run:

python scripts/ballot-lab-data.py <test-case-file> <index of ballot style>

e.g.

python scripts/ballot-lab-data.py june_test_case.json 1

Structure of output:

- Output is a series of contests, grouped by contest type (candidate, ballot
measure)
- Within a contest type order of records is preserved.
- The `VotingVariation` in the EDF is `vote_type` here. It can be filtered.
- `vote_type` of `plurality` is the simplest kind of ballot.
- Ignore `n-of-m` and `ranked-choice` until later.
- Write-ins are integrated into the candidate list.
- The fields were selected to match what is needed for `plurality` candidate
contests and a little extra.
- To add fields or modify them we should modify `extract_{contest type}_contest`.

Notes:

- There are no `Header`s or `OrderedHeader`s in `june_test_case.json`.
43 changes: 43 additions & 0 deletions src/electos/ballotmaker/data/edf-to-ballot-data.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
import argparse
import json
from dataclasses import asdict
from pathlib import Path

from electos.ballotmaker.data.extractor import BallotDataExtractor


def report(data, **opts):
"""Generate data needed by BallotLab."""
extractor = BallotDataExtractor()
ballot_data = extractor.extract(data)
ballot_data = [asdict(_) for _ in ballot_data]
print(json.dumps(ballot_data, indent = 4))


def main():
parser = argparse.ArgumentParser()
parser.add_argument(
"file", type = Path,
help = "Test case data (JSON)"
)
parser.add_argument(
"--debug", action = "store_true",
help = "Enable debugging output and stack traces"
)
opts = parser.parse_args()
file = opts.file
opts = vars(opts)

try:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a separate file loader function outside of main() so that:

  • main() is essentially handling the CLI input and passing it to the file loader
  • This file loader could be called from the main ballotmaker CLI

Good enough for now, though!

Copy link
Collaborator Author

@ion-oset ion-oset Sep 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. But there's not much to it, we can just copy it out. It's not a separate loader because I don't see the script as being used except to test out results. I presume the rest of BallotMaker will be managing files elsewhere and passing them in. You'll note that BallotDate.Extractor.extract takes a dictionary, not a file or a stream. I/O isn't part of this module.

I'm trying this out now on a branch: modifying ballotmaker demo so it takes a Path parameter and loads the test case from that parameter.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! Let me know how it works out

with file.open() as input:
text = input.read()
data = json.loads(text)
report(data, **opts)
except Exception as ex:
if opts["debug"]:
raise ex
print("error:", ex)


if __name__ == '__main__':
main()
Loading