[examples] Reshare over replicated log #1796

clabby · 2025-10-03T19:46:08Z

Overview

Adds an example for resharing over a replicated log, rather than using a trusted arbiter (a. la. commonware-vrf).

makes progress on #1708 - to also reconfigure after performing a reshare, we'll need to have the epocher example in.

cloudflare-workers-and-pages · 2025-10-03T19:46:46Z

Deploying monorepo with Cloudflare Pages

Latest commit:	`3c94581`
Status:	✅ Deploy successful!
Preview URL:	https://cc148058.monorepo-eu0.pages.dev
Branch Preview URL:	https://cl-reshare-example.monorepo-eu0.pages.dev

View logs

patrick-ogrady

The core cryptography is used correctly but suggest a much tighter coupling with block height (will make this much simpler/easier to follow -> we can't be "optimistic" with this, so we shouldn't design the code assuming we want to).

examples/reshare/src/application/actor.rs

examples/reshare/src/dkg/actor.rs

patrick-ogrady · 2025-10-03T22:44:53Z

examples/reshare/src/setup.rs

+
+// --- `serde` helpers ---
+
+mod serde_hex {


We should use commonware_utils::hex?

If we need to reach for serde for some reason, we should expand the scope of what utils can do.

commonware_utils::hex / commonware_utils::from_hex are incompatible with serde without additional wrapping. You have to type-erase your hex data as a String, and you end up with something like:

#[derive(Serialize, Deserialize)] pub struct MySerdeStuff { /// A `Public<MinSig>` that's type-erased as a hex string pub poly: String, } // 2-hop serialization let poly: Poly<MinSig> = ...; let t = MySerdeStuff { poly: hex(poly.encode().as_ref()) }; let ser_t = serde_json::to_string(&t); // ... // 2-hop deserialization let deser_t: MySerdeStuff = serde_json::from_string(&ser_t).unwrap(); let poly_bytes = from_hex(&deser_t.poly).unwrap(); let poly = Poly::<MinSig>::decode(&poly_bytes).unwrap();

Normally it's common to have some serde feature flags lying around in crates so that you can directly serialize / deserialize without intermediate steps or type erasure, i.e.:

#[derive(Serialize, Deserialize)] pub struct MySerdeStuff { pub poly: Public<MinSig>, } let poly: Poly<MinSig> = ...; let t = MySerdeStuff { poly }; // one-hop serialization let ser_t = serde_json::to_string(&t); // one-hop deserialization let deser_t: MySerdeStuff = serde_json::from_string(&ser_t).unwrap();

See for example how we did it over in kona.

examples/reshare/src/dkg/actor.rs

patrick-ogrady · 2025-10-06T00:25:55Z

examples/reshare/src/dkg/actor.rs

+                &mut receiver,
+            );
+
+            while let Some(message) = self.mailbox.next().await {


I'm tempted to suggest merging this with application because we only ever interact with it (what I've been using as the "gut check" for when an actor should be merged/standalone)?

I think the dkg logic could remain in its own function but seems clearer to have this run inside the application instead of as another actor that is "driven" by blocks?

We can debate tomorrow.

Yeah, could certainly do that. I mainly separated them as a best-practice for the example; Hopefully, we can steer people away from doing expensive computation in the application control loop (soon, that'll be fool proof with the post-coding abstraction.)

Anything that can slow down building (and by virtue, consensus) == bad imo

^ i.e., the way I think about "application" is that it's an incredibly hot control loop; In a well designed CW app, it seems like you should never be able to hold it up for more than ~a few ms (and all computation that's being done there should be critical progress to moving to the next view.)

fair retort

examples/reshare/src/dkg/actor.rs

examples/reshare/src/application/actor.rs

patrick-ogrady · 2025-10-06T00:30:21Z

examples/reshare/src/application/actor.rs

+        let genesis_digest = genesis.digest();
+        let built = Arc::new(Mutex::new(None));
+
+        while let Some(message) = self.mailbox.next().await {


Looking forward to the actor-less abstraction 👀

examples/reshare/src/dkg/types.rs

examples/reshare/src/dkg/manager.rs

examples/reshare/src/validator.rs

patrick-ogrady

All I could do before internet cut out. Will keep reviewing.

patrick-ogrady · 2025-10-13T18:36:01Z

examples/reshare/src/application/actor.rs

+                            }
+
+                            // Ask the DKG actor for a result to include
+                            let reshare = context


I still don't feel great about this timeout here.

The intention with the consensus-process dropping the response channel when it no longer needed something was to have a clean abstraction for the application to build ASAP whatever it had and then return. If dkg doesn't have a commitment, it should just be able to return no?

dkg::Actor can be doing work when the request for a commitment comes in, and may not be able to immediately return None.

I originally had the work that DkgManager does in a separate thread to make sure that the control loop for the dkg::Actor could always issue snappy responses, though it led to the code being a bit less clear to follow (from your initial review @ #1796 (comment)).

Now, since actions are dispatched inline, it's possible that the reshare process could hold up block building without this timeout.

monorepo/examples/reshare/src/dkg/actor.rs

Lines 279 to 299 in 9aa01bd

match relative_height.cmp(&(BLOCKS_PER_EPOCH / 2)) {

Ordering::Less => {

// Continuously distribute shares to any players who haven't acknowledged

// receipt yet.

manager.distribute(epoch).await;

// Process any incoming messages from other dealers/players.

manager.process_messages(epoch).await;

}

Ordering::Equal => {

// Process any final messages from other dealers/players.

manager.process_messages(epoch).await;

// At the midpoint of the epoch, construct the deal outcome for inclusion.

manager.construct_deal_outcome(epoch).await;

}

Ordering::Greater => {

// Process any incoming deal outcomes from dealing contributors.

manager.process_block(epoch, block).await;

}

}

examples/reshare/src/application/actor.rs

patrick-ogrady · 2025-10-13T18:40:25Z

examples/reshare/src/application/actor.rs

+                                return;
+                            }
+
+                            // Verify the block


We need to check this before the re-proposal logic. Otherwise, someone could go right to the last height and re-propose (skipping the inner heights).

If we checked this first, we'd never be able to check the re-propose case.

block.height != parent.height + 1 + block.parent != parent.digest() are always true when re-proposing, which would cause verification to always fail.

Note the order of checks here:

If re-proposing (parent commitment == proposed commitment), the block's height must be the last height in the epoch to validate.

Else, the block must:

Have a height that increases by 1 relative to its parent.

Have its parent digest set to the digest of the parent.

Be in the same epoch as the round from the consensus Context.

Because you cannot re-propose without building on top of a previously verified parent, and prior to the last height, the standard rules are followed, it should not be possible to skip the inner heights here.

tl;dr - this saves us:

monorepo/examples/reshare/src/application/actor.rs

Line 180 in 9aa01bd

if block.height == get_last_height(round.epoch()) {

examples/reshare/src/application/supervisor.rs

patrick-ogrady · 2025-10-13T18:46:58Z

examples/reshare/src/application/supervisor.rs

+    }
+}
+
+impl<V: Variant, P: PublicKey> p2p::Coordinator for Supervisor<V, P> {


Given we are changing consensus participants but not peers, I wonder if we should separate this?

Good idea - we can if you'd prefer! Not a huge deal in the example; Easy enough to construct a supervisor for the sake of the P2P resolver that marshal needs.

patrick-ogrady · 2025-10-13T18:47:56Z

examples/reshare/src/dkg/actor.rs

+
+        if inactive_participants.len() < self.num_participants_per_epoch {
+            // Choose some random active participants to also be players if there are not enough.
+            let mut rng = StdRng::seed_from_u64(0);


use context's rng?

Is there a way to do seeded RNG with Context? We need this to be pseudo-random so that all participants grab the exact same set of initial players.

patrick-ogrady · 2025-10-13T18:50:58Z

examples/reshare/src/validator.rs

+    let (mut network, mut oracle) = discovery::Network::new(context.with_label("network"), p2p_cfg);
+
+    // Register all possible peers
+    oracle.register(0, peer_config.all_peers()).await;


Given we don't update oracle in subsequent epochs, we should remove the p2p oracle impl from supervisor? (see earlier comment)

#1796 (comment)

examples/reshare/src/validator.rs

patrick-ogrady · 2025-10-13T19:00:17Z

examples/reshare/src/validator.rs

+                    }
+
+                    // If ends with contiguous_height, ensure it is at least required_container
+                    if metric.ends_with("_processed_height") {


We should ensure this runs over a few epochs (it may already but just calling out to be sure)

Yeah it does! We've got test_1k that goes over 10 epochs.

codecov · 2025-10-14T01:17:45Z

Codecov Report

❌ Patch coverage is 58.97436% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.20%. Comparing base (9c7abdc) to head (3c94581).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
storage/src/store/mod.rs	0.00%	16 Missing ⚠️

@@            Coverage Diff             @@
##             main    #1796      +/-   ##
==========================================
+ Coverage   92.10%   92.20%   +0.09%     
==========================================
  Files         307      308       +1     
  Lines       79648    81273    +1625     
==========================================
+ Hits        73362    74938    +1576     
- Misses       6286     6335      +49

Files with missing lines	Coverage Δ
consensus/src/threshold_simplex/engine.rs	`96.92% <100.00%> (+4.73%)`	⬆️
cryptography/src/bls12381/dkg/arbiter.rs	`95.58% <ø> (ø)`
storage/src/store/mod.rs	`88.14% <0.00%> (-2.36%)`	⬇️

... and 10 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9c7abdc...3c94581. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

clabby added the example An example of how to use the library code label Oct 3, 2025

clabby force-pushed the cl/reshare-example branch from 2889336 to d5ed983 Compare October 3, 2025 19:46

clabby force-pushed the cl/reshare-example branch 2 times, most recently from afc14ba to 3a4b100 Compare October 3, 2025 19:50

clabby requested a review from patrick-ogrady October 3, 2025 19:56

clabby added this to Tracker Oct 3, 2025

clabby moved this to Ready for Review in Tracker Oct 3, 2025

clabby requested a review from BrendanChou October 3, 2025 19:56

clabby force-pushed the cl/reshare-example branch from 3a4b100 to 06c99fe Compare October 3, 2025 20:33

patrick-ogrady requested changes Oct 3, 2025

View reviewed changes

patrick-ogrady mentioned this pull request Oct 4, 2025

[bls12381/dkg] Remove Duplicate Commitment Check #1806

Merged

clabby force-pushed the cl/reshare-example branch 3 times, most recently from e24f8cd to 19f84ee Compare October 5, 2025 19:45

clabby self-assigned this Oct 5, 2025

clabby force-pushed the cl/reshare-example branch from 19f84ee to 151e66d Compare October 5, 2025 20:12

patrick-ogrady reviewed Oct 6, 2025

View reviewed changes

examples/reshare/src/dkg/actor.rs Outdated Show resolved Hide resolved

patrick-ogrady reviewed Oct 6, 2025

View reviewed changes

examples/reshare/src/application/actor.rs Show resolved Hide resolved

patrick-ogrady reviewed Oct 6, 2025

View reviewed changes

examples/reshare/src/dkg/types.rs Show resolved Hide resolved

patrick-ogrady reviewed Oct 6, 2025

View reviewed changes

examples/reshare/src/dkg/manager.rs Outdated Show resolved Hide resolved

patrick-ogrady reviewed Oct 6, 2025

View reviewed changes

examples/reshare/src/validator.rs Show resolved Hide resolved

clabby force-pushed the cl/reshare-example branch 4 times, most recently from 61297c7 to c883fdb Compare October 7, 2025 14:35

clabby marked this pull request as draft October 8, 2025 01:50

clabby force-pushed the cl/reshare-example branch from 9b4aa44 to 6dd7e29 Compare October 8, 2025 01:50

clabby force-pushed the cl/reshare-example branch 11 times, most recently from 7e40088 to 9aa01bd Compare October 13, 2025 17:03

clabby mentioned this pull request Oct 13, 2025

[docs] Add blog post for reshare example #1875

Draft

clabby added 7 commits October 13, 2025 13:29

[examples] Reshare over replicated log

dd061a3

move to new ack/share API

75e1fc1

add epoching

949e846

rebase

c8047ab

Adjust DkgManager to allow for optional playing/dealing

6f6ff30

reconfigure amongst different participants

04aa97b

tests + cleanups

30ccc48

patrick-ogrady reviewed Oct 13, 2025

View reviewed changes

examples/reshare/src/validator.rs Outdated Show resolved Hide resolved

patrick-ogrady reviewed Oct 13, 2025

View reviewed changes

clabby force-pushed the cl/reshare-example branch 2 times, most recently from 51a7d14 to 4b5b815 Compare October 13, 2025 19:59

[wip] crash recovery

3c94581

clabby force-pushed the cl/reshare-example branch from 4b5b815 to 3c94581 Compare October 13, 2025 23:55

	match relative_height.cmp(&(BLOCKS_PER_EPOCH / 2)) {
	Ordering::Less => {
	// Continuously distribute shares to any players who haven't acknowledged
	// receipt yet.
	manager.distribute(epoch).await;

	// Process any incoming messages from other dealers/players.
	manager.process_messages(epoch).await;
	}
	Ordering::Equal => {
	// Process any final messages from other dealers/players.
	manager.process_messages(epoch).await;

	// At the midpoint of the epoch, construct the deal outcome for inclusion.
	manager.construct_deal_outcome(epoch).await;
	}
	Ordering::Greater => {
	// Process any incoming deal outcomes from dealing contributors.
	manager.process_block(epoch, block).await;
	}
	}

[examples] Reshare over replicated log #1796

Are you sure you want to change the base?

[examples] Reshare over replicated log #1796

Uh oh!

Conversation

clabby commented Oct 3, 2025

Overview

Uh oh!

cloudflare-workers-and-pages bot commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying monorepo with Cloudflare Pages

Uh oh!

patrick-ogrady left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clabby Oct 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clabby Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

patrick-ogrady left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Oct 14, 2025

cloudflare-workers-and-pages bot commented Oct 3, 2025 •

edited

Loading

clabby Oct 5, 2025 •

edited

Loading

clabby Oct 6, 2025 •

edited

Loading