Implement the ZODA Coding Scheme #1802

cronokirby · 2025-10-04T01:56:44Z

c.f. https://eprint.iacr.org/2025/034

Like Reed-Solomon coding, but with a guarantee, after checking your own shards, that you will be able to reconstruct the data, given enough shards from other people.

patrick-ogrady · 2025-10-06T18:06:47Z

coding/src/zoda.rs

@@ -0,0 +1,510 @@
+use crate::{


When the time is write, it would be great to add a long-form description of what is going on here (and how it differs from ZODA as-written)

EXACTLY what I was hoping for. Will review and provide feedback 👍

patrick-ogrady · 2025-10-06T18:09:09Z

coding/src/zoda.rs

+/// The actual number of required samples for a given n * samples and k * samples
+/// will be less.
+fn required_samples_upper_bound(n: usize, k: usize) -> usize {
+    (128.0 / -(1.0 - k as f64 / (n + k) as f64).log2()).ceil() as usize


We must use BigRational (or something like it) to ensure this number is computed the same on all architectures.

cloudflare-workers-and-pages · 2025-10-07T00:13:32Z

Deploying monorepo with Cloudflare Pages

Latest commit:	`4d47a1c`
Status:	✅ Deploy successful!
Preview URL:	https://5de6ca51.monorepo-eu0.pages.dev
Branch Preview URL:	https://cronokirby-zoda.monorepo-eu0.pages.dev

View logs

patrick-ogrady · 2025-10-07T22:54:55Z

justfile


 # Run all fuzz tests in a given directory
-fuzz fuzz_dir max_time='60' nightly_version='+nightly':
+fuzz fuzz_dir max_time='60' nightly_version='+nightly' max_mem='4000':


Why does this use so much memory (it looks like it is failing fuzz CI for OOM)?

==8964== ERROR: libFuzzer: out-of-memory (used: 4009Mb; limit: 4000Mb) To change the out-of-memory limit use -rss_limit_mb=<N> Live Heap Allocations: 4087589494 bytes in 9177 chunks; quarantined: 18386202 bytes in 812404 chunks; 21986 other chunks; total chunks: 843567; showing top 95% (at most 8 unique contexts) 4043309056 byte(s) (98%) in 964 allocation(s) #0 0x5634fe51df44 (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0x9e2f44) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #1 0x5634fe68d4a3 (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xb524a3) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #2 0x5634fe65ccae (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xb21cae) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #3 0x5634fe6638b6 (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xb288b6) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #4 0x5634fe57238c (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xa3738c) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #5 0x5634fe59a984 (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xa5f984) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #6 0x5634fe563cbb (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xa28cbb) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #7 0x5634fe58c364 (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xa51364) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #8 0x5634fe5682ae (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xa2d2ae) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #9 0x5634fe575c08 (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xa3ac08) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #10 0x5634fe5a06b4 (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xa656b4) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #11 0x5634fe5a28b3 (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xa678b3) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #12 0x5634fe5a64a8 (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xa6b4a8) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #13 0x5634fe5a0eeb (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xa65eeb) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #14 0x5634fe5adcb0 (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xa72cb0) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #15 0x5634fe5b5309 (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xa7a309) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #16 0x5634fe5b641a (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xa7b41a) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #17 0x5634fe5b7387 (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xa7c387) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #18 0x5634fe5cd866 (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xa92866) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #19 0x5634fe5e7636 (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0xaac636) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) #20 0x7f8b4982a1c9 (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e) #21 0x7f8b4982a28a (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 274eec488d230825a136fa9c4d85370fed7a0a5e) #22 0x5634fe47c1a4 (/home/runner/work/monorepo/monorepo/target/x86_64-unknown-linux-gnu/release/zoda+0x9411a4) (BuildId: 9d0c1ff1858fd873085f0329d2c8c658427029f7) MS: 1 CopyPart-; base unit: adc83b19e793491b1c6ea0fd8b46cd9f32e592fc 0xa,0xa, \012\012 artifact_prefix='coding/fuzz/artifacts/zoda/'; Test unit written to coding/fuzz/artifacts/zoda/oom-71853c6197a6a7f222db0f1978c7cb232b87c5ee Base64: Cgo= SUMMARY: libFuzzer: out-of-memory

patrick-ogrady · 2025-10-07T22:56:56Z

coding/src/zoda.rs

+//! From that participant's perspective, they've received a completely random
+//! choice of rows. The other participants' rows are less random, since they're
+//! guaranteed to not overlap. However, no guarantee on the randomness of the other
+//! rows is required: each sample is large enough to guarantee that the data


I suspect this will be the primary claim people poke at. I wonder if there is some way we can add some intuition around a malicious dealer not being able to circumvent this?

coding/src/zoda.rs

patrick-ogrady · 2025-10-07T23:01:32Z

coding/src/lib.rs

        test_suite::<NoCoding<Sha256>>();
    }
+
+    #[test]


I wonder how hard it would be to compare an off-the-shelf KZG impl here? Seems like that would be unreal...

utils/src/rational.rs

coding/src/field.rs

patrick-ogrady · 2025-10-07T23:46:52Z

coding/fuzz/src/lib.rs

-        let data_len = u.int_in_range(0..=u32::MAX)?; // data.len() <= u32:Max
-        let data = u.bytes(data_len as usize)?.to_vec();
-        let shuffle_bytes = u.bytes(8)?.to_vec();
+        let data = u.arbitrary::<Vec<u8>>()?;


I suspect we'll save a good bit of memory if we better cap this (unless I'm missing something)?

Not sure why this worked before tbh...

The issue is more so how the other parameters affect the number of samples you need, and the minimum size of each shard, versus the data itself. The way the fuzzing suite works is that the fuzzer won't just generate a big vector that exceeds the memory budget, but it's not able to peer into the code to avoid doing things with consequences later on.

I'm sure you've already noted this but we just "skip" configurations that are too hairy in other fuzzers. I suppose we could do that here for now?

This shows that the shuffling logic works at least.

We'll need this for the recovery algorithm.

No restriction on shuffle length, and we actually check things from the perspective of one participant.

Much cleaner now!

ZODA can use a lot of memory (on one machine) if you have a lot of shards, because each shard has data proportional in size to the number of shards, so you get a quadratic size when simulating things on one machine.

codecov · 2025-10-13T23:24:48Z

Codecov Report

❌ Patch coverage is 90.12876% with 115 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.19%. Comparing base (e2917c4) to head (4d47a1c).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
coding/src/zoda.rs	80.05%	67 Missing ⚠️
coding/src/poly.rs	93.82%	29 Missing ⚠️
coding/src/field.rs	91.57%	15 Missing ⚠️
coding/src/lib.rs	97.33%	2 Missing ⚠️
utils/src/rational.rs	98.11%	2 Missing ⚠️

@@            Coverage Diff             @@
##             main    #1802      +/-   ##
==========================================
- Coverage   92.20%   92.19%   -0.01%     
==========================================
  Files         308      311       +3     
  Lines       81072    82096    +1024     
==========================================
+ Hits        74750    75687     +937     
- Misses       6322     6409      +87

Files with missing lines	Coverage Δ
coding/src/reed_solomon/mod.rs	`92.67% <ø> (+3.15%)`	⬆️
coding/src/lib.rs	`88.39% <97.33%> (-5.25%)`	⬇️
utils/src/rational.rs	`98.36% <98.11%> (-0.35%)`	⬇️
coding/src/field.rs	`91.57% <91.57%> (ø)`
coding/src/poly.rs	`93.82% <93.82%> (ø)`
coding/src/zoda.rs	`80.05% <80.05%> (ø)`

... and 2 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e2917c4...4d47a1c. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

patrick-ogrady reviewed Oct 6, 2025

View reviewed changes

cronokirby force-pushed the cronokirby/ZODA branch from 0e2ed7e to e5840e7 Compare October 7, 2025 00:13

cronokirby marked this pull request as ready for review October 7, 2025 00:16

cronokirby changed the title ~~WIP: ZODA~~ Implement the ZODA Coding Scheme Oct 7, 2025

cronokirby force-pushed the cronokirby/ZODA branch from e5840e7 to 3a4a477 Compare October 7, 2025 20:53

patrick-ogrady reviewed Oct 7, 2025

View reviewed changes

coding/src/zoda.rs Outdated Show resolved Hide resolved

patrick-ogrady reviewed Oct 7, 2025

View reviewed changes

utils/src/rational.rs Outdated Show resolved Hide resolved

patrick-ogrady reviewed Oct 7, 2025

View reviewed changes

coding/src/field.rs Outdated Show resolved Hide resolved

patrick-ogrady reviewed Oct 7, 2025

View reviewed changes

cronokirby force-pushed the cronokirby/ZODA branch 6 times, most recently from 92caff3 to 080a7df Compare October 13, 2025 17:46

cronokirby added 10 commits October 13, 2025 15:01

[coding] Create empty ZODA implementation

393489c

[coding] Implement ZODA without checksum columns

59411af

This shows that the shuffling logic works at least.

[coding] Goldilocks field addition

345149f

[coding] Goldilocks field multiplication

5753b3d

[coding] Goldilocks: use const wherever possible

0b3023b

[coding] Field: add root of unity

e3a6825

[coding] move field to its own module

3937d9b

[coding] Implement NTT and inverse NTT

b229dd0

[coding] Make NTT generic over IndexMut

3df6f69

We'll need this for the recovery algorithm.

[coding] Implement vanishing polynomial calculation

a6eb6e8

cronokirby added 21 commits October 13, 2025 15:01

[coding] Implement reed solomon recovery

a62f2fd

[coding] Rename ntt => poly

854969a

[coding] Implement stream conversion from u64 -> F

c2fd078

[coding] ZODA encoding

48f00a3

[coding] Implement ZODA

8e6449b

[coding] Formatting

dfdf135

[coding] Upgrade fuzz test

41bfc48

No restriction on shuffle length, and we actually check things from the perspective of one participant.

[coding] Refactor generic test suite

67d83df

Much cleaner now!

[coding] Fix edge case with ZODA sample calculation

42fa78e

[coding] Adjust fuzz tests to stay under 4000 MB

874bd9b

ZODA can use a lot of memory (on one machine) if you have a lot of shards, because each shard has data proportional in size to the number of shards, so you get a quadratic size when simulating things on one machine.

[coding] Share checksum between Shards with Arc

664c814

[coding] Implement Codec for ZODA

aa5f991

[coding] ZODA: More robust reshard checks

d25c47f

[coding] ZODA: actually implement error handling

1b1c8ad

[coding] Formatting

14909ee

[coding] Fix remaining lints

6dec86a

[coding] Document ZODA

564b68d

[utils] Add lg2 function to rational

1d58f5e

[coding] Use BitMap, and rational number calculation

6c79485

[coding] ZODA: Review pass 1

1542c01

[coding] Reduce fuzz parameters

4d47a1c

cronokirby force-pushed the cronokirby/ZODA branch from 080a7df to 4d47a1c Compare October 13, 2025 22:01

Implement the ZODA Coding Scheme #1802

Are you sure you want to change the base?

Implement the ZODA Coding Scheme #1802

Conversation

cronokirby commented Oct 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloudflare-workers-and-pages bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying monorepo with Cloudflare Pages

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Oct 13, 2025

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cronokirby commented Oct 4, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented Oct 7, 2025 •

edited

Loading