VTP+ Demo: Tech Stack

[stratofax]

I originally posted this to Slack, copied here so we can track this conversation here -- NJ

Before we pick a frontend tech stack, I suggest the following:

1. Select the backend tech stack. @ion-oset recommends FastAPI and I think this is an excellent idea.
2. Sketch out a basic API or data interface to the backend.
3. Create wireframes of the webpages needed for the frontend, perhaps using something like Figma.
4. Then, with all of these elements at least sketched out, select a tech stack for the front end, most likely some kind of JS framework, or vanilla HTML / CSS / JS. Since I think I’ll be doing most of the coding here (it is my day job, when I’m not writing Python) I’d like to have some input, but I don’t have any existing favorites or agenda besides keeping it simple.

As far as the backend tech stack, some questions to answer:

• Once the backend gets data from a voter, how does it prepare the data to be consumed by VTP+? This is truly the heart of the system, and we want to get this right.
• Do we need to write to persistent storage, or can everything be stored in the server’s memory?
• Do we need to validate data on the backend? How about the frontend? What kind of validation?

Answering these questions will help us evaluate the best choice for a backend server.

[ttvp-admin]

Neil- I could not find this equivalent thread in Slack; on what channel (presumably an OSET Slack account)? No big deal as I can easily track here too. Gregory Miller Co-Founder, COO (*He/Him*) OSET Institute, Inc. <http://www.osetinstitute.org> | TrustTheVote Project <http://www.trustthevote.org> | ElectOS <https://electos.org/> | Podcast <https://deadmendontvote.org/> 530 Lytton Avenue, 2nd Floor Palo Alto, CA 94301 Main: 650.600.1450 Mobile: 503.703.5150

…

On Wed, Jan 11, 2023 at 6:05 AM Neil Johnson ***@***.***> wrote: *I originally posted this to Slack, copied here so we can track this conversation here -- NJ* Before we pick a frontend tech stack, I suggest the following: 1. Select the backend tech stack. @ion <https://github.com/ion> recommends FastAPI and I think this is an excellent idea. 2. Sketch out a basic API or data interface to the backend. 3. Create wireframes of the webpages needed for the frontend, perhaps using something like Figma 4. Then, with all of these elements at least sketched out, select a tech stack for the front end, most likely some kind of JS framework, or vanilla HTML / CSS / JS. Since I think I’ll be doing most of the coding here (it is my day job, when I’m not writing Python) I’d like to have some input, but I don’t have any existing favorites or agenda besides keeping it simple. As far as the backend tech stack, some questions to answer: - Once the backend gets data from a voter, how does it prepare the data to be consumed by VTP+? This is truly the heart of the system, and we want to get this right. - Do we need to write to persistent storage, or can everything be stored in the server’s memory? - Do we need to validate data on the backend? How about the frontend? What kind of validation? Answering these questions will help us evaluate the best choice for a backend server. — Reply to this email directly, view it on GitHub <#51>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADFYC2WEL5BR5J36MGLSOCTWR24Y5ANCNFSM6AAAAAATYCQLAM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[stratofax]

Hi Greg,

This is actually the preferred place to discuss the VTP+ demo tech stack, for the exact reasons you cited about Slack. We're really trying to move the conversation away from Slack and onto GitHub where it can benefit even more contributors, beyond the small group on a Slack channel or discussion.

It's great to have you in the mix!

[ttvp-admin]

Yeah, sure I am fine with that, especially if the intent is to make this a public conversation vs. OSET Institute internal; I misunderstood the intent. Let this conversation flourish here in the wild for sure. There is a related concern that I've no idea how to address, that I can take up offline with you. Cheers Greg Gregory Miller Co-Founder, COO (*He/Him*) OSET Institute, Inc. <http://www.osetinstitute.org> | TrustTheVote Project <http://www.trustthevote.org> | ElectOS <https://electos.org/> | Podcast <https://deadmendontvote.org/> 530 Lytton Avenue, 2nd Floor Palo Alto, CA 94301 Main: 650.600.1450 Mobile: 503.703.5150

…

On Thu, Jan 12, 2023 at 7:18 AM Neil Johnson ***@***.***> wrote: Hi Greg, This is actually the preferred place to discuss the VTP+ demo tech stack, for the exact reasons you cited about Slack. We're really trying to move the conversation away from Slack and onto GitHub where it can benefit even more contributors, beyond the small group on a Slack channel or discussion. It's great to have you in the mix! — Reply to this email directly, view it on GitHub <#51 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADFYC2QELBONJFWE4IIA7T3WSAOCTANCNFSM6AAAAAATYCQLAM> . You are receiving this because you commented.Message ID: <TrustTheVote-Project/VoteTrackerPlus/repo-discussions/51/comments/4668735 @github.com>

[windoverwater]

Some initial first-round thoughts on the tech stack:

• let's just use FastAPI - https://fastapi.tiangolo.com/
• it seems like the VTP python code needs a shim (IO) class to shepherd the current text (utf8) to JSON or something that FastAPI can then handle. Note that some of the print statements are already just calling json.dump
• note - there is only one user interactive program so far and I think is the only one needed for the demo, and that is the vote.py and cast_ballot.py scripts (vote.py calls cast_ballot.py in the backend). Thats should be the only program that makes repeated calls into the backend
• of the remaining tool.poetry.scripts (poetry_pyproject.toml) I think only show_contests.py, tally_contests.py, and verify_ballot_receipt,py need front-end support

[stratofax]

My quick thoughts on this:

• FastAPI is an excellent choice, and I think it's also the simplest option that does what we need it to do.
• In fact, I think we can use FastAPI to set up API endpoints using the built-in web server, which means we don't need an intermediate JSON file to recieve data from the client.
• If vote.py and cast_ballot.py are what we need to use to create a frontend for voters, let's build an API for these first.
• Do we need front-end support for these other scripts, or could we run them from the existing CLI for purposes of tabluating the demo?

[windoverwater]

Here is a summary of the scripts that voters will want to run via their phones (on the backend in their specific git workspace):

• vote
• verify-ballot-receipt
• tally-contests
• show-contest

Note that the vote script calls the accept-ballot and cast-ballot scripts and that accept-ballot calls the merge-contest script.

The scripts are located here: https://github.com/TrustTheVote-Project/VoteTrackerPlus/tree/master/src/vtp

[windoverwater]

Notes from the George Howell in-person Neil-Sandy meeting. Note - perhaps with bad form I am taking the liberty to fill in a few details that we did not actually discuss only for better understanding of these notes. As such, please do not interpret ALL of the below as agreed-to decisions - it is possible that Neil and I misunderstood what the other was saying. So to catch possible instances of that early, I am filling out some detail which in fact might need to be redacted. So please be awake when reading this.

1. The George Howell Cafe has good coffee - the flat white was exceptional

2. Overview of the current idea of the actual demo:

• Every half hour, a 20 minute demo starts. Using the same software, CVR data, hardware, etc., first there is a RCV demo which focuses on RCV, and there there is VTP demo which focuses on VTP.
• the participants basically more or less perform the same actions on their cell phones, it is just that the storyline/narrative focuses on either RCV or VTP. The voters still execute more or less the same sequence of touch points.
• the hardware is probably a linux box and a router/firewall/WiFi AP with a disconnected WAN port

3. The (Linux) server 'starts' post an execution of 'setup-vtp-demo' which currently creates the local git remote repos and a couple git workspaces for a few instances of the 'VTP scanner app'. setup-vtp-demo currently needs GitHub access but it can be done beforehand.

4. More demo VTP details:

• each voter when they connect to the server will need the server to create a git clone (workspace) of the VTP mock demo repo. Each voter than interacts only with that clone via FastAPI from the client side into the server side and from there into the VTP layer.
• in the nominal workflow, the voter will first execute the 'vote' VTP command. The vote command is the only VTP interactive command.
• the other two commands that the voter will execute are verify-ballot-receipt and tally-contests. They might also want to execute show-contest. But these three commands are not interactive.
• unfortunately the VTP scripts call VTP scripts by creating python subprocesses and not by import'ing the scripts. This was deemed ok for the time being - if there is enough time prior to the demo these can be changed to an import idiom.
• a few VTP scripts currently output JSON, but most output text and read text from STDIN. VTP needs a FastAPI shim layer for all IO.

5. Client details:
   There was general agreement to avoid big frameworks and instead leverage the python ??? package to generate the javascript, CSS, etc files for the client side. The decision is more or less based on:

• the 'big' frameworks are effectively a learning curve tax following by a performance and accessibility and usability tax
• the client side is small enough that a framework is not necessary
• the team has expertise with python ??? and no one wants to learn yet another framework and the details of how to get around its limitations

Also, given the nature of the VTP local install and the fact that the (linux) server runs under a single UID, that single UID will have the same active poetry VTP environment for all client side voter connections - only the git clone of the mock git repo will be unique.

6. Action Items

• we need to start working a wire frame (screenshots whatever) for the client side. We will try figma first to see if that works.
• we did not talk about this, but IMHO we still need an initial stub out of a test harness so that we can start putting some tests together
• we decided to start meeting regularly on Tuesdays at 10:30 at various coffee places
• we need to set up a second weekly zoom meeting where Ion and Cliff can possibly attend. @ion-oset @cwulfman can we meet sometime on Thursday?
• I need to learn FastAPI so to start adding the IO shim layer to the existing VTP. @ion-oset - can you help with this?

The existing https://github.com/TrustTheVote-Project/VTP-RCV-demo seems ok to start accepting client and server side code.

Miss anything or misrepresent anything?

[stratofax]

Looks good, thanks for going into detail here. A few notes:

• Item 1, Coffee Quality: I had the cappuccino, so I can't comment on the flat white -- but from my experience, I think it was probably at least as good as what I had.
• Item 2, Demo Overview: when you say "router" I think you mean "WiFi hotspot." This means the demo is accessible within the radius of this local WiFi LAN, but will not be available to the rest of the Internet.
• Item 5, Client/Server details: I'd actually call this item "Client Details," because it covers the discussion we had about developing the client, using HTML, CSS and JavaScript. I suggested that we don't use a JS framework (like React, Next.js, or Angular) for the reasons that Sandy notes above (learning curve and performance tax, etc.). Please note there is no direct connection to or dependence on Python or any related libraries. All communication between the client and the server will happen via a well-defined API endpoint that resides on the linux box. This means that the client doesn't care if the web server is running Python or any other language. The client will only need to read data from and send data to this API endpoint, likely in some kind of JSON blob.
• Item 6, Action Items: I sent Sandy an invite to a OSET Figma project. Let me know if you'd like access too!

[windoverwater]

Thanks! Edited.

[ttvp-admin]

This is a hugely helpful summary of the meeting, thanks! Greg Gregory Miller Co-Founder, COO (*He/Him*) OSET Institute, Inc. <http://www.osetinstitute.org> | TrustTheVote Project <http://www.trustthevote.org> | ElectOS <https://electos.org/> | Podcast <https://deadmendontvote.org/> 530 Lytton Avenue, 2nd Floor Palo Alto, CA 94301 Main: 650.600.1450 Mobile: 503.703.5150

[ttvp-admin]

One important point here: It is imperative that I understand as early as possible the BOM for this demo in May; if there are items of HW or SW that we need to have on hand for demo purposes, I need visibility for budget purposes ASAP. We can (and should) take that thread offline. Thanks! GAM

…

On Fri, Jan 27, 2023 at 2:51 PM Neil Johnson ***@***.***> wrote: Looks good, thanks for going into detail here. A few notes: - Item 1, Coffee Quality: I had the cappuccino, so I can't comment on the flat white -- but from my experience, I think it was probably at least as good as what I had. - Item 2, Demo Overview: when you say "router" I think you mean "WiFi hotspot." This means the demo is accessible within the radius of this local WiFi LAN, but will not be available to the rest of the Internet. - Item 5, Client/Server details: I'd actually call this item "Client Details," because it covers the discussion we had about developing the client, using HTML, CSS and JavaScript. I suggested that we *don't* use a JS framework (like React, Next.js, or Angular) for the reasons that Sandy notes above (learning curve and performance tax, etc.). Please note there is no direct connection to or dependence on Python or any related libraries. All communication between the client and the server will happen via a well-defined API endpoint that resides on the linux box. This means that the client doesn't care if the web server is running Python or any other language. The client will only need to read data from and send data to this API endpoint, likely in some kind of JSON blob. - Item 6, Action Items: I sent Sandy an invite to a OSET Figma project. Let me know if you'd like access too! — Reply to this email directly, view it on GitHub <#51 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADFYC2QAVJLZUGYUJ42NA33WURGP5ANCNFSM6AAAAAATYCQLAM> . You are receiving this because you commented.Message ID: <TrustTheVote-Project/VoteTrackerPlus/repo-discussions/51/comments/4801330 @github.com>

[windoverwater]

Funny you should ask this - created a private repo, https://github.com/TrustTheVote-Project/VTP-private-discussions, so to be able to host private discussions. Moving this to https://github.com/TrustTheVote-Project/VTP-private-discussions/discussions/2

Is this 'offline' enough?

[windoverwater]

7. Server Details

Along the lines of running the live demo, we probably want to be able to manually test the whole set-things-up, execute, and tear-things-down - if not automate it all with tests :-). Here are some initial thoughts regarding what we may need/want. At the moment I am thinking these could either be python scripts or makefile entries.

• Initialize the polling center (the demo hardware):

  • run a setup-vtp-demo from scratch into a new directory location
  • start the web server with vote.py connections blocked

• Open the polls

  • allow vote.py connections to proceed

• Close the polls

  • stop new vote.py connections
  • shut down all active vote.py connections
  • flush all pending CVR merges
  • report on client connection statistics

• Shutdown the polling center

  • stop the server
  • log some demo data

In theory the above would be executed once every 30 minutes, alternatively starting a RCV demo and then a VTP demo (on the same data, hardware, etc).

[stratofax]

For the sake of simplicity, wouldn't it be easier to start and stop the server to open and close the polls? In other words, the intialze and open steps are combined, as are the close and shutdown steps. Two commands instead of four. What do you think?

[stratofax]

As discussed in our in-person meeting, the outline above is not intended to represent separate commands, but instead to document the process of starting and stopping the election at a high level. So, I agree with the outline of steps as it's currently described.

[windoverwater]

Notes from the Haute/Nero Coffee in-person Neil-Sandy 20230131 meeting.

Coffee at Haute was good but IMHO George Howell is just better. After about an hour we moved to Nero . FWIIW I find Nero's coffee to be unfortunately inconsistent depending on the barista - and usually not as good as Haute.

The meeting began with Neil debriefing me on prior OSET technology that is or could be relevant to VTP. This was very informative and took a while and was very helpful. Not recording those notes here since this is a public VTP discussion board. We then moved to Nero's.

After reviewing the "7. Server Details" above we eventually decided to try understanding that and the VTP demo workflow in terms of FastAPI endpoints. And to do this prior to tackling some initial wire framing via [[https://www.figma.com/file/z6cAkP3ZNJRgZYHAhKWiA8/VTP%2B-Demo?node-id=0%3A1&t=og8zKERF62uQ5M97-0]]. In the end we came up with the following initial 5 endpoints:

• Pre-demo steps
  • This occurs with the demo laptop connected to the internet and having GitHub access etc
  • From within the chosen VTP poetry python environment AND chosen mock demo git repo (and git submodules), run setup-vtp-demo
    • setup-vtp-demo can do what it does now (it currently creates 4 scanner repos/workspaces and that is ok for now)

0. Initialize the demo on-site - assumes no internet connection

   • this happens at the fair
   • start the web server
   • test the web server

1. First API endpoint

   • phone connects to web server and requests a unique connection ID
   • returns a unique connection ID

2. Second API endpoint

   • given a unique connection ID, phone requests a ballot
   • returns a ballot

3. Third API endpoint

   • given a unique connection ID, uploads a ballot
     • We decided for the moment that the (JS) client side will validate the contest selections and directly support voter self-adjudication of the ballot. The VTP python code currently also supports this. However, the data that the VTP python code returns to the client via endpoint 3 contains the information necessary for the client side (JS) to directly self adjudication. Another solution is to create another endpoint that manages the self adjudication of the ballot so that no JS needs to be written/debugged/tested to support this. In hindsight, I think I now vote for an additional endpoint to use the existing python self adjudication code.
     • Question - is it ok for the VTP python code to continue to validate the completed ballot and to throw an exception back to the client side if the completed ballot is non-compliant post JS self-adjudication?

4. Fourth API endpoint

   • supports calling the python script verify-ballot-receipt with various switches. Each different python switch would map to a different button
   • it is this endpoint which is of high interest regarding the VTP demo. This is the endpoint that basically demonstrates E2EV. As such we probably want to plan some time optimizing the UX experience for this - the better it is, the more compelling VoteTracker+ will be for the voter.
   • also requires a connection ID

5. Fifth API endpoint

   • supports calling the python script tally-contests with various switches. Each different python switch might map to a different button
   • it is this endpoint which is of high interest to the RCV folks as it is this one where the voter can see how their ranked voted is counted across the rounds of rank choice voting. As such we probably want to plan some time optimizing the UX experience for this - the better it is, the more compelling RCV should be for the voter.

Other action items and decisions/observations:

1. Rename the 'local-remote' git repo in all documentation to the 'tabulation' repo - the name local-remote is confusing.

2. Maybe finesse two different names for the current VTP scanner app repos/workspaces to either scanner repos/workspaces when talking about voting center VTP deployments and client repos/workspaces when talking about the RCV/VTP demo.

3. We decided to allow participants in the demo to vote as many times as they would like so to gain experience with RCV and VTP and to allow multiple people to use one phone. This might be a bad decision - in retro spec it seems like it. I think we want as few things as possible to confuse the voter (even though in real life they will not be voting by phone). However, to prevent multiple votes the solution would need to work across multiple browsers - it may be too much work for the demo.

4. The presenter needs to make certain that people understand that in a real election, the voter's secret number (the row offset) will not be observable by any other person, election official or voter, except in the existing case of accessibility needs where someone else may already have access to the selections depending on the specific election infrastructure.

[windoverwater]

Moved this to https://github.com/TrustTheVote-Project/VTP-web-api/blob/main/docs/DesignNotes.md