Skip to content

Commit 1ede65b

Browse files
nuke-web3nuke-web3
authored
Verifiable Encryption Doc & Diagrams (#19)
1 parent 6711267 commit 1ede65b

2 files changed

Lines changed: 178 additions & 20 deletions

File tree

‎README.md‎

Lines changed: 62 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,9 @@
11
# Private Data Availability Proxy
22

3-
A [Celestia Data Availability (DA)](https://celestia.org) proxy, enabling use of the [canonical JSON RPC](https://node-rpc-docs.celestia.org/) but intercepting and [***verifiably*** encrypting](./doc/verifiable_encryption.md) sensitive data before submission on the public DA network, and enable decryption on retrieval.
3+
A [Celestia Data Availability (DA)](https://celestia.org) proxy, enabling use of the [canonical JSON RPC](https://node-rpc-docs.celestia.org/) but intercepting and [**_verifiably_** encrypting](./doc/verifiable_encryption.md) sensitive data before submission on the public DA network, and enable decryption on retrieval.
44
Non-sensitive calls are unmodified.
55

6-
Verifiable encryption is presently enabled via an [SP1 Zero Knowledge Proof (ZKP)](https://docs.succinct.xyz/docs/sp1/what-is-a-zkvm), with [additional proof systems planned](./doc/verifiable_encryption.md)
6+
Verifiable encryption is presently enabled via an [SP1 Zero Knowledge Proof (ZKP)](https://docs.succinct.xyz/docs/sp1/what-is-a-zkvm), with [additional proof systems planned](./doc/verifiable_encryption.md#future-work-and-research-directions)
77

88
**Jump to a section:**
99

@@ -30,22 +30,17 @@ It's possible to change these, but requires upstream involvement:
3030

3131
> Please [open an issue](https://github.com/celestiaorg/pda-proxy/issues) if you have any requests!
3232
33-
## Architecture
33+
## Interact
3434

35-
![Verifiable Encryption Diagram](./doc/assets/verifiable-encryption.drawio.svg)
35+
First you need to [configure](#configure) your environment and nodes.
3636

3737
The PDA proxy depends on a connection to:
3838

3939
1. Celestia Data Availability (DA) Node to:
4040
- Submit and retrieve (verifiable encrypted) blob data.
4141
1. (Optional) [Succinct prover network](https://docs.succinct.xyz/docs/sp1/generating-proofs/prover-network) as a provider to generate Zero-Knowledge Proofs (ZKPs) of data existing on Celestia.
4242
_See the [ZKP program](./zkVM/sp1/program-chacha) for details on what is proven._
43-
44-
## Interact
45-
46-
First you need to [configure](#configure) your environment and nodes.
47-
48-
Then any HTTP1 client works to send [Celestial JSON RPC](https://docs.celestia.org/how-to-guides/submit-data#submitting-data-blobs-to-celestia) calls to the proxy:
43+
Then any HTTP1 client works to send [Celestia JSON RPC](https://docs.celestia.org/how-to-guides/submit-data#submitting-data-blobs-to-celestia) calls to the proxy:
4944

5045
```sh
5146
# Proxy running on 127.0.0.1:26657
@@ -75,6 +70,56 @@ cd scripts
7570

7671
Celestia has many [API client libraries](https://docs.celestia.org/how-to-guides/submit-data#api) to build around a PDA proxy.
7772

73+
### Request Flow
74+
75+
#### Encrypt `blob.Submit`
76+
77+
```mermaid
78+
sequenceDiagram
79+
participant JSON RPC Client
80+
participant PDA Proxy
81+
participant Celestia Node
82+
JSON RPC Client->>+PDA Proxy: blob.Submit(blobs, options)<br>{AUTH_TOKEN in header}
83+
PDA Proxy->>PDA Proxy: Job Processing...<br>{If no DB entry, start new zkVM Job}
84+
PDA Proxy->>-JSON RPC Client: Response{"Call back"}
85+
PDA Proxy->>PDA Proxy: ...Job runs to completion...
86+
JSON RPC Client->>+PDA Proxy: blob.Submit(blobs, options)<br>{AUTH_TOKEN in header}
87+
PDA Proxy->>PDA Proxy: Query Job DB<br>Done!<br>{Job Result cached}
88+
PDA Proxy->>Celestia Node: blob.Submit(V. Encrypt. blobs, options)
89+
Celestia Node->>PDA Proxy: Response{Inclusion Block Height}
90+
PDA Proxy->>-JSON RPC Client: Response{Inclusion Block Height}
91+
```
92+
93+
#### (Try Decrypt) `blob.[Get|GetAll]`
94+
95+
```mermaid
96+
sequenceDiagram
97+
participant JSON RPC Client
98+
participant PDA Proxy
99+
participant Celestia Node
100+
101+
JSON RPC Client->>+PDA Proxy: blob.Get(height, namespace, commitment)
102+
PDA Proxy->>Celestia Node: <Passthrough>
103+
Celestia Node->>PDA Proxy: Response{namespace,data,<br>share_version,commitment,index}
104+
PDA Proxy->>PDA Proxy: *Try* deserialize & decrypt
105+
PDA Proxy->>-JSON RPC Client: *Success* -> Response{...,decrypted bytes,...}
106+
PDA Proxy->>JSON RPC Client: *Failure* -> <Passthrough>
107+
```
108+
109+
#### Transparent Proxy (all other calls)
110+
111+
```mermaid
112+
sequenceDiagram
113+
participant JSON RPC Client
114+
participant PDA Proxy
115+
participant Celestia Node
116+
117+
JSON RPC Client->>+PDA Proxy: Request{<Anything else>}<br>{AUTH_TOKEN in header}
118+
PDA Proxy->>Celestia Node: <Passthrough>
119+
Celestia Node->>PDA Proxy: <Passthrough>
120+
PDA Proxy->>-JSON RPC Client: Response{<Normal API response}
121+
```
122+
78123
## Operate
79124

80125
**_TODO: notice on single job at a time_**
@@ -88,12 +133,12 @@ To build and run, see [developing instructions](#develop)
88133

89134
### Requirements
90135

91-
TODO
92-
93136
1. A machine to run with a _minimum_ of:
94137

95-
- L4 NVIDIA GPU
96-
- GB RAM
138+
- NVIDIA GPU with 20GB+ of VRAM (Tested on [L4](https://www.nvidia.com/en-us/data-center/l4/))
139+
- Must support CUDA 12+
140+
- 4+ CPU cores
141+
- 16GB+ RAM
97142
- Ports accessible (by default):
98143
- service listening at `TODO`
99144
- Light client (local or remote) over `26658`
@@ -136,6 +181,8 @@ As we don't want to embed huge files, secrets, and dev only example static files
136181

137182
1. Setup a DNS to point to your instance with email and domain.
138183
1. Create and update an `.env` (see [config](#configure).
184+
1. Select a base OS image to run on the host that includes [CUDA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/index.html) or install it manually.
185+
- See: [CUDA Container Toolkit install instructions](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) and [AWS NVIDIA docs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-nvidia-driver.html) (or your cloud host's docs for GPU base OS images).
139186
1. Run [./scripts/setup_remote_host.sh](./scripts/setup_remote_host.sh) or otherwise see the scripts to manually configure similarly.
140187
1. **ONLY for development & testing!** copy the unsafe example TLS files from [./service/static](./service/static) to `app/static` on the host
141188
- You should use:
@@ -249,7 +296,4 @@ The images are built and published for [releases](https://github.com/celestiaorg
249296

250297
## Acknowledgments
251298

252-
Based heavily on:
253-
254-
- <https://github.com/eigerco/lumina/tree/main/rpc>
255-
- <https://github.com/celestiaorg/eq-service>
299+
Based heavily on <https://github.com/celestiaorg/eq-service>.

‎doc/verifiable_encryption.md‎

Lines changed: 116 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,118 @@
11
# Verifiable Encryption
22

3-
TODO describe and expand on [research here](https://docs.google.com/document/d/1XZyuOxdMm5INcHwQZOZ8ALRk_YkvicNwQHSfOVs8hoM/)
4-
./verifiable_encryption.md
3+
### _A New Primitive Empowering Private Data Availability_
4+
5+
> “Don’t trust. Verify.”
6+
7+
This document introduces **Verifiable Encryption (VE)** and explores how it enables **Private Data Availability (PDA)** - a transformative new primitive for secure, decentralized systems.
8+
9+
## Verifiable Encryption
10+
11+
Traditional encryption ensures that ciphertext reveals **nothing** about the underlying plaintext.
12+
This guarantees privacy, but at a cost: **you can’t verify anything about the data without decrypting it**.
13+
14+
This creates a trust bottleneck - **you must fully trust those who hold decryption keys** to act honestly.
15+
But what if we could shift some of that trust to verification?
16+
17+
**Verifiable Encryption (VE)** makes this possible.
18+
By introducing constraints on:
19+
20+
- the plaintext,
21+
- the encryption algorithm,
22+
- and the keys used,
23+
24+
VE allows **public verifiability** of claims about the data **without decrypting it**.
25+
That means anyone can check that encrypted data meets specific criteria - without learning what that data is.
26+
27+
## Data Availability
28+
29+
Many protocols rely on **Data Availability (DA)** for safety and liveness guarantees.
30+
In adversarial conditions - such as censorship or outages - it is critical that data be **publicly available**.
31+
32+
However, not all data should be exposed to the world.
33+
34+
Some datasets are **too sensitive** for full transparency.
35+
The challenge: **How can we ensure critical data is available, yet only selectively disclosed under prearranged conditions**?
36+
37+
## The Power of VE + PDA
38+
39+
By combining **Verifiable Encryption** with **Private Data Availability**, we unlock a powerful new primitive: **auditable yet private data**.
40+
41+
With integration into **existing or novel Key Management Systems (KMS)**, VE allows one to define:
42+
43+
- **who can access** decryption keys,
44+
- **under what conditions**,
45+
- and **what can be verified** without access.
46+
47+
This means:
48+
49+
- Anyone (users, smart contracts, off-chain agents) can verify that encrypted data is available and satisfies certain properties.
50+
- Only authorized parties can decrypt and access the sensitive contents.
51+
52+
## Use Cases
53+
54+
We’ve outlined a few use cases below - but would love to hear your ideas too!
55+
💡 [Open an issue](https://github.com/celestiaorg/pda-proxy/issues) to share feature requests or novel applications of VE and PDA.
56+
57+
### _Programmable Privacy for Web3 dApps_
58+
59+
VE and PDA align closely with the principles of [local-first access control](https://www.inkandswitch.com/keyhive/notebook/), enabling **secure collaboration** across decentralized applications.
60+
61+
In a world where chain data is globally replicated and indexed, **encryption at rest** becomes essential for access control and selective disclosure.
62+
63+
#### Example Applications
64+
65+
- **PDA as a database** for collaborative dApps with fine-grained access control.
66+
- **Private rollups** with programmable cryptography, enabling [obfuscated state](https://0xparc.org/blog/programmable-cryptography-1).
67+
- **Private bridging and escrow** sending verifibly correctm but private messages around web2 and/or web3 apps.
68+
- **Drop-in support** for existing DA users via a [proxy service](../README.md), simplifying migration to PDA.
69+
70+
### _Trustless Data Markets_
71+
72+
With VE, PDA, and escrow contracts you can construct protocols to build trustless exchange of data access
73+
See the [Stock0](https://dorahacks.io/buidl/14098) media market hackathon project for some great inspiration!
74+
75+
Here is a [diagram inspired by them](https://docs.google.com/presentation/d/1qq1QXSBcThOjaQ2OcEyS8cwNyAHs3SnC76YrBMAYENk) of an example setup of inputs for a market:
76+
77+
```mermaid
78+
flowchart LR
79+
Data["Data to be Sold"] --> zkVM_Algo["zkVM(tranform media)"]
80+
zkVM_Algo -- "proven data transform w/ VE anchor" --> Contract["Marketplace on <dApp chain>"]
81+
Data -- "VE data" --> Celestia["Celestia"]
82+
Celestia -- "header" --> Blobstream["Blobstream on <dApp chain>"]
83+
Blobstream <-- "verify VE anchor and DA" --> Contract
84+
```
85+
86+
### _Verifiable Private Backups_
87+
88+
> NOTE: Celestia does _not_ guarantee that data will be avalible forever!
89+
> See [the docs on retrievability](https://docs.celestia.org/learn/retrievability#data-retrievability-and-pruning-in-celestia-node) for the latest safe assumptions to use.
90+
91+
With PDA, sensitive data can be publicly published in encrypted form, with **predefined methods for recovery** - without revealing its contents.
92+
93+
This unlocks a new class of **verifiable, resilient backups**.
94+
95+
#### Example Applications
96+
97+
- **Auditable storage**: Confidential datasets can be verified to exist and be recoverable, while remaining hidden from the public.
98+
99+
- **Disaster recovery**: Critical encrypted data is guaranteed to be retrievable using known decryption methods, ensuring survivability without sacrificing privacy.
100+
101+
## Example Architecture
102+
103+
The **anchor** acts as a bridge, connecting **any protocol** to a **proof** that some _private_ data was made available.
104+
105+
![Verifiable Encryption Diagram](./assets/verifiable-encryption.drawio.svg)
106+
107+
## Future Work and Research Directions
108+
109+
While VE for PDA is still evolving, the potential is enormous.
110+
Current implementations have limitations, but these are rapidly being addressed by:
111+
112+
- Enabling performance improvements, **confidential compute**, and **scalable parallelization** of PDA workflows.
113+
- **Hybrid systems** combining:
114+
- Trusted Execution Environments (TEEs),
115+
- Multi-Party Computation (MPC),
116+
- and Zero-Knowledge Proofs (ZKPs),
117+
118+
For deeper insights, see our ongoing [research discussion here](https://docs.google.com/document/d/1XZyuOxdMm5INcHwQZOZ8ALRk_YkvicNwQHSfOVs8hoM/).

0 commit comments

Comments
 (0)