NIST data records library

[ion-oset]

Proposal

Create a library that defines Python data classes for each of the NIST schemas that can be shared between all ElectOS subsystems that are built (initially the ones using Python).

Rationale & Design Goals:

• These data models come up again and again in different projects and are being re-written part way in multiple places. They would benefit from a single code base derived from a single source of truth.
• Treat the schema as the source of truth, because it's part of the standard and can be reviewed and understood independent of the code.
• Be DRY: One code base (per-language) allows easier debugging and testing.
• Extensibility: Generate low-level data-only classes from a standard schema that other code can add behavior to while being able to exchange the data between packages and subsystems.
• Future-proof. This isn't tied to one language. It is also likely that the schema itself will change or be improved. This design intends to avoid lock-in but allow better implementations to verify that their behavior hasn't changed or broken anything.

Schemas

The schemas in question are the ones listed in NIST Publications:

• Election Results Reporting (NIST SP-1500 100) - JSON Schema
• Voter Registration Information (NIST SP-1500 102) - JSON Schema
• Cast Vote Records (NIST SP-1500 103) - JSON Schema

Design decisions

Start with Python

Rationale:

• There are several ElectOS projects written in Python that need access to NIST data records right now.
• Python has a lot of library and tool support to try this out quickly.

Use data classes

"Data classes", aka "records", "plain old data", "structs" are classes with no methods. Functionality can be added to them as class methods or via subclasses, but there's no data attached to a basic record beyond the data extracted from the schema.

Rationale:

• Makes serialization/deserialization easier.
• Don't add what's not needed. Adding behavior by inheritance or composition is easy to do on top of these classes.

Use a code generator

Don't write these classes by hand. Existing generators get us most of the way there. If we need to do so we can we can develop a custom generator that allows us complete control over the output.

Rationale:

• Saves time: hand-written boilerplate for 50+ classes per schema is hard to maintain.
• Resulting code is robust against changes in the schema.
• Allows easier modification of implementation if when dependencies change.
• Change implementation library if there are technical issues.
• Generate equivalent libraries for languages besides Python.

Use the standard's JSON Schema as the source of truth

We don't need to be limited to JSON, or to require it, but as an initial source of truth it has some advantages:

• There are decent generator libraries for JSON Schema.
  • There are also decent generator libraries for XML Schema, though most of them are not Python.
• JSON interchange over APIs will be more obvious to support.
• JSON is a close match for objects in programming languages.
  • XML is a closer match for documents, though it can handle objects.

Example

An example of what this looks like for NIST SP-1500-102 is what's currently in Vanadium Core: vanadium/models/nist

• Record classes
• Enumerations

[cwulfman]

A few comments to start the discussion:

• This is all pretty abstract. It would be great if you could provide an example of how these classes would actually be used, in several ElectOS subsystems. I can imagine a few:
  • read in an ElectionReport (or a CVR); add to it or make changes to it; export it (perhaps in a different format, like XML, or RDF)
  • from an API: process a POST or PUT of data (to, e.g., https://electos.org/app/CandidateContest/ with new or updated information about a contest). You need to be able to parse the HTTP content (probably a JSON string); validate it; and store it in a local database (an SQL store, or an XML store, or an RDF triple store)

What would this look like in practice, using this records library?

[ion-oset]

Agreed. I wanted to get the abstract part written down. I'll attach some examples as soon as I can.