BL Grant Proposal RFP #3 -- Reference Implementation Test Harness

[Therecanbeonlyone1969]

Background:

As part of proving conformance of a Baseline Protocol Implementation (BPI) with the Baseline Protocol Specification (currently in draft release v 0.1), the entity trying to prove conformance to the Baseline Protocol specification -- all three sections -- must have a systematic way to prove conformance to each requirement. This is most easily achieved through an automated test harness which is the focus of this RFP.

Goal:

In order to make it easier for implementers to prove conformance, the goal of this RFP is to create an automated test suite or test harness for implementers to prove the conformance of their implementation with the Baseline Protocol Specification. Note, that it is NOT the goal of this RFP to create a full end-to-end integration test suite since those are implementation-specific.

RFP Done Criteria:

Minimal Done Criteria:

• Each requirement marked R must have a unit test that can be run against one or more of the existing reference implementations. Note that the reference implementations must not pass the unit test.
• Each unit test for a requirement marked R must have:
  • One or more test fixtures that describe the unit test and its expected behavior, that specify input data for the unit test in one or more schemas, and the expected output of the unit test. in one or more schemas. Not both input and outputs will contain required data elements, and may contain also optional data elements
  • A well-documented software wrapper in at least 1 common development language (javascript, rust, etc) that allows an implementer to invoke either an external API call (compliant with the API standard) or a separate external API that allows to directly access an internal implementation endpoint that invokes the functionality specified in the requirement, and return a unit-test result in the specified output schema(s).

Requirments marked as D or conditional requirements may be included in the test harness with the same Done Criteria as for the R requirements.

Note, that implementers might make implementation choices such as for specific zero-knowledge prover schemes that are not covered by the test harness. In that case, it is incumbent to the implementer to add those unit tests to the test harness to ensure that the test harness stays up to date and allows more and more integration with different implementation choices.

Budget: $100,000

[ognjenkurtic]

@GoldenBit0 Since this goes hand in hand with the work core devs are doing on the SRI, we might be good candidates to take on this task as well. Can we talk about this during our next SRI meeting?

[GoldenBit0]

@ognjenkurtic let's discuss this week!

[GoldenBit0]

Beneficial Skill Set:

• 2 - 3 years of Java script or type script experience. Go or Rust work too
• Experience with standard cryptographic libraries e.g. libsodium or cryptojs and cryptographic primities such as hashing, digital signatures, encryption/decryption, merkle trees
• Experience with REST APIs, ideally also websockets
• Experience with data base administration and, ideally, also table/data model design
• 1 - 2 years of QA test automation experience including relevant frameworks e.g. cypress and others

Nice-to-have:

• Experience with zero-knowledge frameworks such as zk-snarks, zk-starks, bullet proofs, committment schemes
• Expereince with peer-to-peer messaging frameworks such as libp2p, swarm, NATS, AMQP

[kthomas]

FYI this PR should be taken into consideration at the next TSC call: ethereum-oasis-op/baseline#503

[JamieXML]

Hi, this is Jamie, spoeaking for OASIS.
Let me mention a few process and license considerations for the TSC and PGB to discuss. Thank you to Sonal, who gave me a great understanding of what's desired here. Because this is a competitive process, and analogous to a hiring, there are a few steps we advise you to take, so that the outcome is generally accepted as open and fair by all participants.

• This is an RfP, using funds and (I assume) your official status, so we recommend you take steps to make sure the opportunity / RfP is open, in the sense of circulated to multiple plausible contractors / bidders. Maybe that's been done? I am concerned about the small number of "bites" of which I have hear, so far. As I mentioned to Sonal, OASIS is happy to help circulate this -- but is there any documentation of your expectations, other than this page, so far?
• Has a deadline been set for responses? Unless there's more of which I am unaware, i think you may need some structure about your dates for opening, closing bids, and choosing one. Several parties
• I agree with KT that this should be an explicit TSC topic. Based on your governance, we'd expect that reviewing and awarding of bids is an action specifically conducted by the TSC, and then, depending on what the EEA OP PGB has agreed to delegate, with either (1) a recommendation reported up to the PGB for action, or (2) the final award reported to the PGB timely.
• When you engage a party, for pay, to do technical work, there should be a form of short contractor agreement in place for your protection, to scope expectations, confirm the award's terms, and obtain proper licensing of the delivered product. OASIS committees do a fair amount of this, I am happy to share a form or help with one. It need not be long, two pages might do it. But the main issue at this stage is that any bidder should expect to work under a short written agreement.
• One of the issues that will come out of a short contract is, who exactly are you contracting with? We recommend against brokered deals where you do business with party A (influence but no duties), who brings in their pal party B (doing the work but mostly answers to Party A, but not you). It's best that your RfP go direct: ti the Party B doing the work.
• You likely also will want official exit criteria listed -- in the process sense of "you deliver these three things, and then TSC/someone reviews it as a conformant deliverable, and then you get paid."
• On licensing let me mention a few specific considerations, which I recommend you (TSC or the RfP process runner) consider:

1. Baseline project's output itself is under CC0, but FYI, the testbed code is not required to be the same in order to work with the C) material.
2. You'd be better protected if you specify what license you want from the contractor. We recommend choosing an open source license with patent claim protection. (E.g., Apache) Bluntly, that way, your code contractor doesn't get to sue you or other users later. There are other options as well if you wish to discuss.
3. The short contract also should provide (and this is an OASIS rule, for your protection) that the right to use the name "Baseline" to officially designate anything, including an official testbed harness, remains with the Project. It should be clearly stated that engaging a contractor does not entitle them to rev and fork Baseline the spec, or give your endorsement to stuff.