BallotMaker Election Data handling

[stratofax]

I'd like to encapsulate the details of data retrieval from the ballot generation code. This way, we can add new data formats and sources (XML, CSV, SQL, API, etc.) without having to rewrite the ballot generator code. This is a classic example of the Open–closed principle.

What's the best way to implement this in BallotMaker?

I'd like to propose some concrete examples. Right now I'm using dictionaries, but I think it would be better to use dataclasses, because dataclasses:

• can be frozen (made immutable) after they're created
• are versatile enough to support single value fields and arrays of data (like, say, lists of candidates with their ids and parties).
• will enforce a consistent data structure in a way that's not possible with dicts or named tuples.

For example:

The CandidateContest dataclass

I'd like to use a factory pattern to make a new CandidateContest dataclass object for each CandidateContest. This class would contain the contest name, and type as properties, and the list of Contest as collection of Candidate and BallotMeasure objects.

Each ballot would therefore be composed of (in part) an ordered set of CandidateContest data classes, which are in turn composed of a collection of Contest objects.

In addition, we could implement a strategy pattern at runtime to decide (in the future) which data retrieval method we want to use, depending on the data source.

This approach could also apply to Ballots themselves.

[stratofax]

Here's a start to a UML diagram for what this might look like:

BallotMaker Class Diagram

classDiagram
  ElectionData *-- BallotData
  BallotData *-- CandidateContestData
  BallotData *-- BallotMeasureData
  BallotMeasureData *-- ChoiceData
  CandidateContestData *-- CandidateData
  class ElectionData {
    +String election_name
    +String start_date
    +String election_type
    +get_election_data()
   
  }
  class BallotData {
    +String id
    +String ballot_scope
    +List candidate_contests
    +List ballot_measures
    +get_ballot_data()
    +get_candidate_contests()
    +get_ballot_measures()
  }
  class CandidateContestData {
    +String id
    +String title
    +String district
    +Int votes_allowed
    +List candidates
    +get_candidate_contest_data()
  }
  class CandidateData {
    +String id
    +String party
    +String party_abbr
    +Bool is_write_in
    +String name
    +get_candidate_data()
  }
  class BallotMeasureData {
    +String id
    +String title
    +String district
    +String text
    +List choices
    +get_ballot_measure_data()
  }
  class ChoiceData {
    +String id
    +String choice
    +get_choice_data()
  }

Loading

Notes

• CandidateContestData, CandidateData, BallotMeasureData, and ChoiceData fields copied directly from contest_data.py. The get methods are currently handled by __post_init__(self) methods.
• The class BallotData is currently not in use. BallotData.id is basically the ballot_style id. BallotData.ballot_scope is a combination of GPUnits, like "Spacetown Precinct, Orbit City"

[stratofax]

@ion-oset to identify the data we still need to generate multiple ballots, I updated the UML diagram (above). Please take a close look at BallotData and the corresponding bullet point in the Notes below the diagram.

Thoughts or questions?

[ion-oset]

Noting our conversation earlier today, and our current conclusions:

The above diagram is very helpful and pretty close to what we want. The main difference is that we want the *Data objects to aggregate their nested elements, so the types on the fields will be classes. These will be considerably simpler than the EDF classes:

• Strings/integers/booleans for most fields (no enumerations)
• Dataclasses for the class types. Currently using Python standard dataclass classes. Even though they don't do enough type checking and need added code to turn the nested dictionaries into nested objects, that's good enough for now.
• Our target is to get rid of all remaining conversion boilerplate and make it possible for the ballot rendering code to simply get data ready to use from the extraction process.

I got a fair amount of the way today, and we're on track. PR forthcoming in the next couple of days.

[ion-oset]

Updated class diagram

classDiagram
  ElectionData *-- BallotStyleData
  BallotStyleData *-- CandidateContestData
  BallotStyleData *-- BallotMeasureData
  BallotMeasureData *-- BallotChoiceData
  ContestData <|-- BallotMeasureData
  CandidateContestData *-- CandidateChoiceData
  ContestData <|-- CandidateContestData
  CandidateChoiceData *-- PartyData

  class ElectionData {
    +String name
    +String type
    +String start_date
    +String end_date
  }

  class BallotStyleData {
    +GpUnit[] ballot_scopes
    +ContestData[] contests

    +candidate_contests()
    +ballot_measure_contests()
  }

  class ContestData {
    +String  id
    +String  title
    +String  district
  }

  class CandidateContestData {
    +String  vote_type
    +Integer votes_allowed
    +CandidateChoiceData[] candidates
  }

  class CandidateChoiceData {
    +String  id
    +PartyData party
    +Boolean is_write_in
  }

  class PartyData {
    +String name
    +String abbreviation
  }

  class BallotMeasureData {
    +String text
    +BallotChoiceData[] choices
  }

  class BallotChoiceData {
    +String id
    +String choice
  }

Loading

Notes

• Base type ContestData for the real contest data types.
  • Allows some shared code.
  • More importantly BallotStyleData.contests has to be a union of the contest types because contests are a mix of possible contest types, and splitting them would lose their ordering.
• PartyData type is used because it's easier to manage than having it be a dictionary.
• ballot_scopes is a list of the GpUnits the BallotStyle refers to
• BallotStyleData.id is the BallotStyle.ExternalIdentifier.Value. (There is no @id for BallotStyles.)
  • This is an optional field in the NIST SP 1500-100 definition, but it is present in the September test case, and it's likely we should require it for the TTV EDF profile.

[stratofax]

This is a great update, I can't wait to see the code!

[ion-oset]

My opinion is that most of what you are proposing I take for granted as the baseline for designing a library / long-term service like BallotMaker is.

• Designing around classes is better than exposing direct data structures. Data structures matter a lot, but they should be easy to replace when they stop working which means even internally it's better to hide them from direct use. Only when performance requires it should the wrappers be pulled off and the wires exposed.
• Yes, use strategy to abstract out the I/O and factories as needed. That's just good modularity, and makes for less brittle code.
• Yes, use dataclasses for the internals rather than directly accessing dictionaries where possible. They are clearer semantically, less vulnerable to typos or unexpected inputs, and much easier to comprehend when reading, especially code someone else wrote. (But this is a different question than the class design -- see the side reply post.)
• Yes (though you didn't say this explicitly) use different types for BallotMaker than exposing any direct access to the internals of the EDF and PDF representations. Hide nist-data-models and reportlab from the BallotLab API. This is what you are doing with the class diagram, and I think it's the most important part to get straight.

More specific thoughts / questions forthcoming.

[ion-oset]

One design issue I want to clarify: dataclasses were designed explicitly to make representation of bundles of data more convenient. They aren't about abstraction: they explicitly expose their implementation to the user. One can use properties to hide direct access to the attributes, but keeping them restricted still doesn't hide the representation. I think their use is mostly orthogonal to the class design and the abstraction boundaries with the client libraries.

Sorry if I'm stating the obvious. Since you started by talking about open/closed principle and then moved into dataclasses vs dicts though I want to be clear I believe those are separate questions and that it's important that we not tangle them together.

[cwulfman]

I've not used the @dataclass decorator very much; so far I've preferred to use @property decorator and write my own accessor methods. To my mind, it has two advantages: (1) it makes a cleaner separation between attributes, which are specific to the implementation of the class, and properties, which are features of the abstraction; (2) it makes it straightforward to implement lazy instantiation of properties, which is useful when setting a property's value involves expensive computation or IO.

Of the two advantages, (1) is probably more important (to someone coming from a functional background, anyway). Users of a class should never access attributes directly; if they follow that practice, the implementation of the class (or its sub-classing) can be changed without affecting their code.

[ion-oset]

This is mostly academic, but there is

I agree completely with the advantages of properties and abstraction. That was why I was making the distinction: the purposes of dataclasses and properties aren't really the same. The alternative to dataclasses here isn't public classes, it's entirely internal to the ballot making. The alternative is passing around dictionaries.

The use case that brought dataclasses into the standard library (see PEP 557) was making it easier to define classes that are meant for holding data (hence the name 😝). They mostly aren't intended to replace all classes, or the use of properties. They are intended to replace such things as namedtuples or __slots__ and cut down on a lot of boilerplate code when mostly what's wanted is a container type, with perhaps a method or two for behavior.

I personally don't think they are the right thing for a public API, except in certain explicit cases (like say data models built around a JsonSchema). In the case of BallotMaker what we're looking is only how to pass around the subset of data from the EDF needed for drawing the ballots internally, so it really is a bundle of attributes, not properties.

[ion-oset]

Let me expand on the initial diagram by asking: what properties does BallotMaker need to track for candidate contests and ballot measure contests? Here's what we have in code so far:

Both (this is probably BallotData but that depends on the definition of that class)

• Contest title
• Election district/jurisdiction
• Headers? We don't have examples of this in the test case yet.

Candidate contests:

• Vote type (what the standard calls VoteVariation). This field is key: it is the primary determinant of the layout choice, and affects what other fields are needed. So far we've only covered plurality (and as a sub-case majority and possibly super-majority) races. n-of-mandranked choice` both warrant other fields than the ones below.
• Candidate lists. Candidates in NIST Data Models are complex, but these are flattened out to the essentials:
  • candidate names
  • party names
  • ...?
• Write-ins. I think these are arguably a special kind of candidate because they have their own layout.

Ballot measure contests:

• Ballot text
• Vote choices

Let's expand on that - what else is needed?

[cwulfman]

Building on what @ion-oset says, and my comment on properties above: I suggest you take a more iterative approach, and, at least at first, establish sparse abstractions and define properties (methods) to produce values as you discover your implementation needs them. You really don't want to get bogged down with data design at this point: NistLib as already done all that heavy lifting for you.

[stratofax]

@cwulfman I completely agree, I don't want to get bogged down with abstract design activities. At this point, I wanted to convey this simple architecture to @ion-oset so he can get a sense of where I want to go. The real design work is taking place in the code right now.

Unfortunately, I haven't been able to merge my work onto main so Ion can't see my code yet! I hope to remedy that today, as the latest round of enhancements includes a first iteration of these data classes handling the data that @ion-oset provided.

Most importantly, I've created an "adapter" layer between the ballot layout and the data, so we can make changes to how we retrieve the data without having to re-write any layout code.

Sweet!

[cwulfman]

Looking forward to seeing the code, @stratofax !

[ion-oset]

Unfortunately, I haven't been able to merge my work onto main so Ion can't see my code yet

No need to merge anything until it's ready. I've been looking at it both in the PR you made today and in my local clone since our last call.

[ion-oset]

@stratofax: Per our discussion on the call we're developing the BallotMaker data model incrementally, as we make fixes that have been filed as issues. Here's what I've got so far.

The following all seem to have been straightforward, with minor edits:

• Implement Instructions: Vote Count ("Vote for X") #82
• Display Party Abbreviation with Candidate, if available #83
• Track Write-In Candidates #84
• Write Contest IDs on ballot for testing and review #86

This one is already done on the data side. If there's more to it I'm missing, please elaborate.

• Add Ballot Measure Contests (3 column layout) #88

This one is a little complex and I'm still figuring out how to shape it. More details in a reply.

• Implement multi-candidate tickets #85

[ion-oset]

This OfficeIds matches to Office.Name.Text.Content: "President of the United States" -- I assume this is definitive, and overrides Contest.Name, even though the two are identical in this case.

Hmm. I'm not certain, but would guess Office.Name does not override Contest.Name. My reasoning is that it's further down the decision chain: the office title is independent of any election, but contest name has been explicitly chosen for the contest. It might usually match the office in the case where there is an office -- but it might not. For example one could include clarifying information from Office.Description or other data in the Contest.Name and still need the official titles next to the names of the candidates.

Something like this example, using Markdown-ish "layout" for a race with multi-candidate tickets. Note: I'm not saying we should use this, I'm just speculating that it's a legitimate thing to do.

# {President of the United States": Contest.Name}

- [  ] {Candidate1.FullName} for {"President": Office.Name} &  {Candidate2} for {"Vice-President": Office.Name}
- [  ] {Candidate3.FullName} for {"President": Office.Name} &  {Candidate4} for {"Vice-President": Office.Name}

[ion-oset]

Thus, I don't need Contest.OfficeIds, but I do need the looked-up Office.Name.Text.Content for each office.

Good. There's code for that already, I left it out of the output so far. I'll turn it on.

In general what I've been doing is always returning data from the element referred to by the ID. So if ballots need an office the ballot data includes fields of the Office element, and the OfficeIds are just for looking it up. (This is why you'll see index (an ElementIndex) as a parameter in the extraction code.) I haven't been adding the @ids unless you explicitly ask for them. That can change if you prefer otherwise.

[ion-oset]

This @id also appears in four (4) BallotStyle.OrderedContent.OrderedContestSelectionIds -- strangely at the top of the list. I'd assume that the write-ins would appear at the bottom of the list? Is this correct?

It makes sense but I'd also be surprised if we can assume that. We should get the word from @cwulfman and @trustthevote, but unless they say otherwise I vote we leave it in the strange order -- at least until the official EDF is corrected. (I say this despite the fact that I think write-ins don't make sense anywhere but at the end, and that putting them at the top would be very like to have political consequences.)

The order on the ballot can be very jurisdiction dependent, and officials may need to mandate a particular one. I do think there is an order, and I think that it defaults to "respect the order in the EDF unless it's explicitly overridden" - but it's not crystal clear in the spec (at least the parts I looked at). Here's what it says (all emphasis is mine, not the spec's):

Under the description of ContestSelection it allows an explicit order, and defers to the BallotStyle if there is none:

SequenceOrder is included to specify an ordering for the contest selections for purposes of display only. The original ballot ordering can be preserved, however, by using the BallotStyle class.

[Edit: "for purposes of display only" may refer to results not ballots 🤨. If so this makes things less certain.]

and under CandidateSelection it talks about the "ordering scheme in place".

When multiple candidates are referenced for a ticket and the ordering of the candidates is important to preserve, it is expected that the generating application will list the references to Candidate instances according to the ordering scheme in place.

[stratofax]

Please continue to pursue this issue as you see fit. As noted elsewhere, I need to check out for a day or so and focus on rendering ballot PDFs with the excellent dataset you sent over earlier today in your updated gist. Thanks again for getting that over so quickly, plenty of "gist for the mill" as it were.

[cwulfman]

We should get the word from @cwulfman and @trustthevote, but unless they say otherwise I vote we leave it in the strange order -- at least until the official EDF is corrected. (I say this despite the fact that I think write-ins don't make sense anywhere but at the end, and that putting them at the top would be very like to have political consequences.)

Correct -- it is up the the creator of the EDF (in this case, myself), to establish the proper order. I'll do that.

[stratofax]

Order of contests & ballot measures

The order that the candidate contests and ballot measures appear in your JSON output seems to be reversed from what I'd expect. Specifically, local contests and ballot measures are at the top; and national, at the bottom.

I didn't know if this was a result of reading BallotStyle.OrderedContent correctly, which means the order of contests is reversed in the EDF file (like it seems to be with BallotStyle.OrderedContent.OrderedContestSelectionIds).

Regardless of the reason, I wanted to float the idea of adding an explicit "order" counter for candidates and contests into your output. Right now, the order is implicit in how the data appears in the file. This seems fragile to me.

@ion-oset what do you think?

[ion-oset]

The algorithm I use in the code for choosing the order of variables is to do a depth first search on BallotStyle.OrderedContent and store them in the order they are encountered. The premise is that it preserves the order (a) as listed in the OrderedContent and (b) as the IDs of elements are encountered in the file. If that's coming out reversed then that's a bug, and I need to fix it.

Right now, the order is implicit in how the data appears in the file. This seems fragile to me.

Do you intend the order counter to be for debugging purposes or a strict definition? As a debug tool I agree. As a definition, I think the order in the EDF should take precedence over everything else unless we learn otherwise.

Note: There is an explicit property set to create an order to ContestSelections (and their subclasses): ContestSelection.SequenceOrder. I just realized it's talking about results display not ballot display so I'm not sure it plays a role here, but the text surrounding it in the spec implies to me that the default order should be what's written in the EDF.

[ion-oset]

Another debugging idea is to patch a copy of the test case and change the IDs to be human comprehensible so we can verify them more easily. It uses a lot of time chasing down opaque IDs over and over.

[stratofax]

I recalled that you mentioned you were pulling the data in the exact order you encountered it in the EDF, so I was suggesting the order counter both as a way to help debug (if there's a problem, we can track it back to where it's introduced), but also that using the position of data in a file (or a table for that matter) as a way to capture sort order is usually not the most robust solution.

Having the order expressed as an unambigous value means I'd have something to check against when I write the contests out on the ballot -- I guess that's another form of debugging, but it feels more like a form of insurance to me. Since we're pulling data out of the EDF and I'm ultimately breaking it up into separate objects, I'd like to ensure that the explicit data order isn't lost in translation.

As always, I'm eager to hear your suggestions. I think a simple counter value that is saved to the data structure as you pull records out of the EDF in order would be pretty straightforward to implement, but that's really your call! Let me know what you suggest.

[stratofax]

@ion-oset you've certainly given me a lot to think about, and your thinking on these issues has (as usual) also been quite excellent. As of tomorrow morning, I'm going to be incorporating the data you've already sent me into the ballot layout, so I'm not going to be able to engage with some of these questions related to tickets and ordering until Friday at the earliest.

In the meantime, I think you should keep doing what you've been doing, posting questions here and on the respective issues, tagging John and Ciff as needed. You've given me plenty of high-quality data to work with for Friday's ballot update. So, please understand that my lack of response here has nothing to do with a lack of interest, only that I'm going full "code-mode" tomorrow until I can see the new data on the ballot PDF.

So, thanks again for your clear and thoughtful work on chasing down these issues with the data. It's essential and I'll be back to help with it in a day or so.

[cwulfman]

We should all be keeping a copy of the NIST-1500-100 spec under our pillows at night -- or at least we should be bookmarking https://pages.nist.gov/ElectionResultsReporting/ for easy access. NIST 1500-100 has been pretty well thought-out, so you should assume they've already thought of the problem you're encountering and have addressed it. If they haven't, first ask yourself if you might need to reformulate your problem: only then should you start thinking about work-arounds and extensions.

So: look at the definition of BallotStyle. First, note this in the definition:

To preserve any rotation associated with the ballot, it is expected that the generating application will list the occurrences of OrderedContest in the order as on the ballot for the associated geopolitical unit.

This is important: it is expected that the application generating the EDF will list the contests in order. That means the ordering of contests in your PDF is out of your hands: you have to use what you're given. It is up to the application that generates the EDF to order the contests the way the user (election official) wants.

That principle (you have to use what you're given) holds throughout.

Now look at the definition of ContestSelection. A ContestSelection object (or one of its sub-classes, CandidateSelection or BallotMeasureSelection) has an optional SequenceOrder, but you should ignore that: it is to be used only in post-election results display. Instead, the spec says:

The original ballot ordering can be preserved, however, by using the BallotStyle class.

So: the designers of NIST-1500-100 expect the order of contests, and of candidates within them, to be defined by the order in the BallotStyle. You have to play 'em as they lie. (It's worth noting that Python lists and tuples are explictly ordered.)

[cwulfman]

This should be a GitHub issue.

…

On Aug 24, 2022, at 6:28 PM, Neil Johnson ***@***.***> wrote: @ion-oset <https://github.com/ion-oset> You noticed that Contest.OfficeIds only lists one ID for the presidential race, but should list two, for president and vice president, respectively. I agree! Also, it seems that BallotStyle.OrderedContent.OrderedContestSelectionIds are listed in reverse order, with the write-in(s) listed at the top. That doesn't seem correct. What's the best way to put this on the @cwulfman <https://github.com/cwulfman> radar? Slack? A GitHub Issue? — Reply to this email directly, view it on GitHub <#81 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGZRWMQZVEQJJX4OPSJDOLV22OZPANCNFSM57IZFLNQ>. You are receiving this because you were mentioned.