perf(engine): batch multiproof messages

[yongkangc]

Problem

When state updates arrive faster than proof computation, each message triggers a separate proof task, causing redundant trie traversals for the same accounts.

Solution

Batch consecutive same-type messages before dispatching to workers.

Batching Logic

Dispatch-then-batch:

1. Dispatch first message immediately — Don't wait; start proof computation right away
2. Accumulate while workers process — Drain channel for more same-type messages (same source for StateUpdate)
3. Process accumulated batch — Merge targets and dispatch as single work item

Different-type messages encountered during drain are stored in pending_msg and processed in the next iteration to preserve ordering.

Changes

• Add process_multiproof_message() with dispatch-then-batch logic
• Add pending_msg field to preserve message ordering across iterations

Expected Impact

• Fewer proof computations under load (N messages merged into 1 batch)
• Lower pending queue, less overhead

Benches

• Mainnet: perf(engine): batch multiproof messages  #20066 (comment)

---

Detailed flow diagrams

fixes #19168

[Copilot]

Pull request overview

This PR introduces message batching optimization for the multiproof task to improve performance, while simultaneously fixing a critical ordering issue. The previous approach of sending non-matching messages back to the channel moved them to the end of the queue, breaking StateUpdate ordering required by ProofSequencer. The solution stores pending messages locally and handles them immediately after batch processing.

Key changes:

• Add batching logic for consecutive PrefetchProofs messages to reduce processing overhead
• Add batching logic for consecutive StateUpdate messages to merge state updates efficiently
• Fix message ordering by using local pending_msg buffer instead of re-sending to channel

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[yongkangc]

Flow Diagrams

Before: No Batching

Channel: [StateUpdate1, StateUpdate2, StateUpdate3, PrefetchProofs1]

Loop:
  recv StateUpdate1   -->  dispatch proof #1
  recv StateUpdate2   -->  dispatch proof #2
  recv StateUpdate3   -->  dispatch proof #3
  recv PrefetchProofs1  -->  dispatch proof #4

Result: 4 messages, 4 proof tasks, 4 trie traversals

After: With Batching

Channel: [StateUpdate1, StateUpdate2, StateUpdate3, PrefetchProofs1]
State:   pending_msg = None

Loop iteration 1:
  pending_msg = None, so select recv StateUpdate1
  |
  +-- process_multiproof_message(StateUpdate1)
        |
        +-- try_recv() StateUpdate2, same type, MERGE
        +-- try_recv() StateUpdate3, same type, MERGE  
        +-- try_recv() PrefetchProofs1, different type
        |         |
        |         +-- pending_msg = Some(PrefetchProofs1)
        |
        +-- dispatch ONE merged batch as proof #1

Loop iteration 2:
  pending_msg = Some(PrefetchProofs1), skip select
  |
  +-- process_multiproof_message(PrefetchProofs1)
        |
        +-- dispatch proof #2

Result: 4 messages, 2 proof tasks, 2 trie traversals

Message Type Handling

Message Type	Batching
StateUpdate	Merge: combine state fields
PrefetchProofs	Merge: extend targets
FinishedStateUpdates	No batch (signal)
EmptyProof	No batch (forward as-is)

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 8 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[yongkangc]

Mainnet Bench on reth4 1000 blocks

[yongkangc]

this is extracted to process_multiproof_message

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

crates/engine/tree/src/tree/payload_processor/multiproof.rs:642

• The documentation states "number of messages merged" but due to the batch size recording issue (see previous comments), the actual values recorded are inconsistent. When the first message is dispatched immediately and no additional messages are batched, it records 1. However, when additional messages are batched after the first immediate dispatch, it only records the count of the additional messages, not including the first one.

Update the documentation to clarify the actual semantics, or fix the recording logic to be consistent.

    /// Histogram of sparse trie update durations.
    pub sparse_trie_update_duration_histogram: Histogram,
    /// Histogram of sparse trie final update durations.
    pub sparse_trie_final_update_duration_histogram: Histogram,

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mediocregopher]

nit: it looks like chunk got renamed to outcome, if we're still returning number of chunks here then chunks or num_chunks would be a clearer name I think

[mediocregopher]

same nit here as with the prewarm case

[mediocregopher]

Did you do some testing that confirmed that doing this is impactful? My intuition says that it wouldn't matter a whole lot, since it's just the results of one tx, more likely than not it would only occupy a single worker while the rest are waiting for further batched messages.

[mediocregopher]

It'd be nice to pull out a lot (or all) of these &muts into a separate struct, like MultiproofBatchCtx or something. It seems like a lot are just for metrics, if so maybe two structs, MultiproofBatchCtx and MultiproofBatchMetrics.