Private DA Middleware Proxy

⚠ NOTICE Work in Progress! NOT ready for use!/ ⚠

TODO

• blob.Submit encrypts before proxy submission
• blob.Get proxy result is decrypted before responding
• state.Balance & state.AccountAddress & state.Transfer passthrough proxy
• Fully compliant passthrough proxy for all non-encrypted data API calls (more than Blob and State?)

---

A Celestia client for the Blob and State JSON RPC namespaces enabling sensitive data to be verifiably encrypted before submission on the (public) network, and enable decryption on retrieval. Non-sensitive calls are unmodified.

Verifiable encryption is enabled via an SP1 Zero Knowledge Proof (ZKP).

Jump to a section:

• Send requests this service: Interact
• Spin up an instance of the service: Operate
• Build & troubleshoot: Develop

Known Limitations

At time of writing, as it should be possible to change these limitations:

• Assumes that there is a single blob per transaction, no logic to handle multiple blobs.
• Verifiable Encryption is one-at-a-time, and takes minutes to complete each request.
  • If the service is processing a job, it will respond with busy and (may?) add that to a queue.
  • {Other proxy calls should still be responsive}

Possible to change these, but requires upstream involvement:

• Max blob size on Celestia is presently ~2MB
• Upstream jsonrpsee en/decryption middleware feature into lumina?

Architecture

---
config:
  look: handDrawn
  theme: dark
  layout: elk
---
flowchart TD
 subgraph VE_Container["zkVM or TEE"]
        Encrypt["Encryption<br>(ChaCha, AES, ...)"]
  end
 subgraph VE_Out_Seal["Verifiable Encrypted Data"]
        VE_Out["Encrypted Data"]
        VE_Hash["Input Data Anchor<br>(Hash)"]
        VE_Seal["Seal/Attestation"]
  end
 subgraph P_Container["zkVM or TEE"]
        Program["Program<br>(DEX, Sequencer)"]
  end
 subgraph P_Out_Seal["Verifiable Program Output"]
        P_Out["Program Output"]
        P_Hash["Input Data Anchor<br>(Hash)"]
        P_Seal["Seal/Attestation"]
  end
 subgraph Chain["Blockchain<br>{Ethereum}"]
        App_Contracts["App Contract"]
        Blobstream["Blobstream"]
  end
    In["{Confidential}<br>Input Data"] --> Encrypt & Program
    Encrypt --> VE_Out_Seal
    Program --> P_Out_Seal
    VE_Hash == Identical === P_Hash
    Key(["{Confidential}<br>Encryption Key"]) -- Provided as input<br>or embedded ---> Encrypt
    VE_Out_Seal --> DA["DA<br>{Celestia}"]
    DA --> Blobstream
    P_Out_Seal --> App_Contracts
    Blobstream -. "Confirm DA for<br>Encrypted Data" .-> App_Contracts
    Chain -. Chain State .-> Program
    Key_Recover["Key Recovery Process<br>(MPC, Threshold, Etc.)"] -.-> Key
    App_Contracts@{ shape: doc}
    Blobstream@{ shape: doc}
    In@{ shape: docs}
    DA@{ shape: cyl}
    Key_Recover@{ shape: h-cyl}
    Key@{ shape: dbl-circ}

Loading

The PDA proxy depends on a connection to:

1. Celestia Data Availability (DA) Node to:
   • Submit and retrieve (verifiable encrypted) blob data.
2. (Optional) Succinct prover network as a provider to generate Zero-Knowledge Proofs (ZKPs) of data existing on Celestia. See the ZKP program for details on what is proven.

Interact

First you need to configure your environment and nodes.

Then any HTTP1 client works to send Celestial JSON RPC calls to the proxy:

# Proxy running on 127.0.0.1:26657
# See: <https://mocha.celenium.io/blob?commitment=S2iIifIPdAjQ33KPeyfAga26FSF3IL11WsCGtJKSOTA=&hash=AAAAAAAAAAAAAAAAAAAAAAAAAFHMGnPWX5X2veY=&height=4499999>

source .env
# blob.Get
curl -H "Content-Type: application/json" -H "Authorization: Bearer $CELESTIA_NODE_WRITE_TOKEN" -X POST \
     --data '{ "id": 1, "jsonrpc": "2.0", "method": "blob.Get", "params": [ 4499999, "AAAAAAAAAAAAAAAAAAAAAAAAAFHMGnPWX5X2veY=", "S2iIifIPdAjQ33KPeyfAga26FSF3IL11WsCGtJKSOTA="] }' \
     $PDA_SOCKET
# blob.GetAll
curl -H "Content-Type: application/json" -H "Authorization: Bearer $CELESTIA_NODE_WRITE_TOKEN" -X POST \
     --data '{ "id": 1, "jsonrpc": "2.0", "method": "blob.GetAll", "params": [ 4499999, [ "AAAAAAAAAAAAAAAAAAAAAAAAAFHMGnPWX5X2veY=" ] ] }' \
     $PDA_SOCKET
# blob.Submit
# Note: send "{}" as empty `tx_config` object, so the node uses it's default key to sign & submit to Celestia
# Also for testing we explicitly allow --insecure
curl -H "Content-Type: application/json" -H "Authorization: Bearer $CELESTIA_NODE_WRITE_TOKEN" -X POST \
     --data '{ "id": 1, "jsonrpc": "2.0", "method": "blob.Submit", "params": [ [ { "namespace": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAMJ/xGlNMdE=", "data": "DEADB33F", "share_version": 0, "commitment": "aHlbp+J9yub6hw/uhK6dP8hBLR2mFy78XNRRdLf2794=", "index": -1 } ], { } ] }' \
     https://$PDA_SOCKET \
     --verbose \
     --insecure
    # ^^^^ DO NOT use insecure TLS in real scenarios!

Celestia has many API client libraries to build around a PDA proxy.

Operate

TODO: notice on single job at a time

• single GPU 100% used per job
• presently no way to scale on multi-GPU

Most users will want to pull and run this service using Docker or Podman via container registry, see running containers.

To build and run, see developing instructions

Requirements

TODO

1. A machine to run with a minimum of:

   • L4 NVIDIA GPU
   • GB RAM
   • Ports accessible (by default):
     • service listening at TODO
     • Light client (local or remote) over 26658
     • (Optional) Succinct prover network over 443

2. A whitelisted key in your env for use with the Succinct prover network Key - requested here.

3. A Celestia Light Node installed & running accessible on localhost, or elsewhere. Alternatively, use an RPC provider you trust.

   • Configure and fund a Celestia Wallet for the node to sign and send transactions with.
   • Generate and set a node JWT with write permissions and set in .env for the proxy to use.

Configure

Required and optional settings are best configured via a .env file. See example.env for configurable items.

cp example.env .env

# edit .env

Using containers

The images are available:

TODO

# ghcr:
docker pull ghcr.io/celestiaorg/pda-proxy:latest

# Docker hub:
docker pull celestiaorg/pda-proxy:latest

Don't forget you need to configure your environment.

Setup Host

As we don't want to embed huge files, secrets, and dev only example static files, you will need to place them on the host machine in the following paths:

1. Create or use scp to copy a known good .env to /app/.env on the host machine (see config.
2. Setup a DNS to point to your instance, update .env with email and domain
3. Run ./scripts/setup_lets_encrypt.sh or otherwise configure TLS certs & keys.
   1. ONLY for testing! copy the unsafe example TLS files from ./service/static to app/static on the host
      • You should use:

      TLS_CERTS_PATH=/app/static/sample.pem
      TLS_KEY_PATH=/app/static/sample.rsa

4. Run ./scripts/init_celestia_docker.sh to initialize a local Celestia Node with persistent storage.
   • Update .env to use the correct CELESTIA_NODE_WRITE_TOKEN

Running containers

With the correct setup of the host, you can startup both the proxy and local celestia node with:

docker compose --env-file /app/.env up -d

Or manually just the proxy itself:

just docker-run

Develop

First, some tooling is required:

1. Rust & Cargo - install instructions
2. SP1 zkVM Toolchain - install instructions
3. Protocol Buffers (Protobuf) compiler - official examples contain install instructions
4. (Optional) Just - a modern alternative to make installed
5. NVIDIA compiler & container toolkit https://docs.succinct.xyz/docs/sp1/generating-proofs/hardware-acceleration#software-requirements

Then:

1. Clone the repo

   git clone https://github.com/your-repo-name/pda-proxy.git
   cd pda-proxy

2. Choose a Celestia Node

   • See the How-to-guides on nodes to run one yourself, or choose a provider & set in env.
   • NOTE: You must have the node synced back to the oldest possible height you may encounter in calling this service for it to fulfill that request.

3. Configure required env variables

4. Build and run the service

   # NOT optimized, default includes debug logs printed
   just run-debug
   # Optimized build, to test realistic performance w/ INFO logs
   just run-release

There are many other helper scripts exposed in the justfile, get a list with:

# Print just recipes
just

Containerization

Docker and Podman are configured in Dockerfile to build an image with that includes a few caching layers to minimize development time & final image size -> publish where possible. To build and run in a container:

# Using just
just docker-build
just docker-run


# Manually

## Build
[docker|podman] build -t eq_service .

## Setup
source .env
mkdir -p $PDA_DB_PATH

## Run (example)
[docker|podman] run --rm -it -v $PDA_DB_PATH:$PDA_DB_PATH --env-file .env --env RUST_LOG=eq_service=debug --network=host -p $PDA_PORT:$PDA_PORT pda_proxy

Importantly, the DB should persist, and the container must have access to connect to the DA light client (likely port 26658) and Succinct network ports (HTTPS over 443).

The images are built and published for releases - see running containers for how to pull them.

License

MIT License

Acknowledgments

Based heavily on:

• https://github.com/paritytech/jsonrpsee/blob/master/examples/examples/proc_macro.rs
• https://github.com/eigerco/lumina/tree/main/rpc