How to build a custom EL on reth with minimal footprint

[roberts-pumpurs]

We are building a L1 blockchain that would use reth as its EL. We need ways to "decrement" user balances based on stuff that happened on the CL. After discussions on Telegram, @mattsse recommended to base our solution on the Reth-Optimism approach by introducing a new transaction type with custom processing logic that only the CL could inject into the mempool.

Our ideal integration with reth would be:

• all of our custom stuff about the custom txs & its handling would be hosted on our repo, and we'd use reth as a library (eg via cargo git deps)
• use reth-ethereum implementations wherever possible

After a lot of experimentation last week, here's the summary of where we currently stand:

• Attempt 1: copying over the custom-node example to our repo and making adjustments. The custom-node example in Reths repo just proxies types and implementations to reth-optimism, making this approach a fools errand. Not all the files from custom-node example were actually used in the actual implementation; eg the ExtendedTxEnvelope was unused anywhere outside of the extended_op_tx_envelope.rs module. Essentially, the example is insufficient/unfinished to be used for our needs, hard-to-understand compile errors made this approach unfeasible, as I was hoisting out a lot of code into our repo, just to make some adjustments & trait implementations. This turned out to be a massive undertaking. Because the deep, nested layers of generic meta-programming, to me it was never really clear what other trait must be implemented or which constraints are needed. We'd need someone on the reth team as a guide for this approach. Altho this would give us the smallest surface area and the least amount of code that we'd need to own and maintain. The breaking point was after I introduced our own custom node primitive struct.
• Attempt 2: copying over the ethereum subfolder and making adjustments, renaming types with our prefixes. The goal was to have a minimum eth-compliant node, with its default implementations inlined in our repository, and then we'd just make alterations where necessary. This approach faced the following problems: ethereum folder is not a self-contained Ethereum implementation of all the necessary traits & types. I could not properly use our own PrimitiveTypes. We ran into problems where rust_ethereum_primitives were hard constraints on some trait implementations (like LoadReceipt) that were not even located under the ethereum folder. This cascaded into the following loop: get compile error about X not satisfying trait constraints (because of some custom primitive we've renamed), copying over the reference implementation and making alterations so it uses the new primitive definition, repeating. As an end result, I was again hoisting over unknown amount of code from crates that lived outside of the ethereum subfolder (from reth core implementations), and making adjustments so it'd use my primitives. Also, such an approach is not documented and seemed like another fool's errand because there was no end in sight of how much stuff I'd have to copy & alter, and whether whatever I was copying was actually a necessary change, and if it would even work once all was copied.
• Attempt 3: In this approach, I tried copying over optimism subfolder because it has hoisted out more code than the ethereum folder (eg custom txpool crate is self-contained there) - it seemed like a more complete version. Also, because our custom eth EL would ideally add a new custom transaction type and work on users balances, the Optimism code would be a very close fit for what we need. Goal being: rather than having "ethereum + our changes", I'd have to do "optimism - unnecessary stuff". But this approach also has its own issues. Namely, optimism heavily depends on optimism-alloy crates (with which I have no experience and I wouldn't know what extra stuff would I be bringing into my dependency chain), and other crates within reth with optimism or op feature flags. As an end result, after hoisting out all of optimism folder, renaming the crates & types to our own, the dependencies would still point to reth crates with op feature flags (eg reth-primitives-traits with op feature), and these crates would then depend on the reth-optimism-primitives & friends rather than the forked version I had in my repo. Only 2 options available: use [crates-io.patch] to force these internal dependencies to use my custom crates that I've forked from optimism folder, or start hoisting out all of these reth-core dependencies into my own repo and adding changes there, and eliminating all usage of op feature flags. Either way both approaches seem either brittle or like an insane amount of work. Especially if I'll then have to cut out a lot of logic that optimism does (eg sequencer comms) and other stuff that I may not even be aware of.

CONCLUSION (TLDR): A common pattern in all of my attempts is that I end up copying over large amounts of code from reth core crates (stuff that did not live under the examples/custom-node, nor ethereum, nor optimism subfolders). And making adjustments to those. Eg when hoisting out reth-optimism I was suddenly an owner of 16k LOC that I did not even write, and still could not get my solution to compile. Arguably a skill issue on my end tho 😄 . I am just highlighting my experience, hats off to you for writing a generic Ethereum EL.

QUESTIONS:

1. I was wondering if the best approach to using reth now would be: forking reth, creating my own sub-folder for my-custom-chain (eg copying over the optimism folder) and keeping all of the changes there, optionally adding feature flags to reth core crates (in a similar fashion like op feature flags). Then the only maintenance burden would be my own subfolder and keeping my reth fork in sync.
2. Is there a feasible alternative I could attempt? What would you suggest?
3. Do you suspect that I'll have to create my own alloy forks (like optimism did)? Ideally I'd like to avoid forking even more foreign code, but I am not yet too familiar with all of the alloy/optimisms/reth integration points and maybe this should be my first step?
4. Is there any work on reth I could contribute & upstream to make my own integration smoother?

[mattsse]

hey @roberts-pumpurs

thanks for the write up.

A common pattern in all of my attempts is that I end up copying over large amounts of code from reth core crates

this is currently still the case yes, the optimism folder would be the blueprint for
creating a new ethereum based chain with invasive internal changes such as:
a) tx types
b) evm
c) rpc

We definitely have some work to do and are focusing on this next to makes this easier and reduce boiler plate as much as possible, e.g. by relaxing some bounds on the default eth/op types.

from experience, introducing feature flags is a nightmare to maintain, but it would be the fastest way to get something modified. while we're trying to make some modifications possible directly in reth.

My recommendation rn would probably be some kind of hybrid approach, where you start your chain as new standalone repo/crate but use a fork of reth as dependencies, this way, if you encounter a limitation you could flag this to us so we can address but in the meantime you could modify something in place, e.g. via feature flags.

Is there any work on reth I could contribute & upstream to make my own integration smoother?

what worked well in the past is a cycle of

1. report limitation
2. we/you address limitation, e.g. by making something configurable

this is basically how the opstack support evolved

In the meantime we'll be making simplifcations to the common:
extend reth with 1 tx type use case.

The custom-node example is also still actively being worked on and got some upgrades over the last few days, we want to use this to find limitations and improve UX specifically for this use case.

I ack that this currently requires a lot of boilerplate, but gives you the most flexibility in the long run.

[mattsse]

One thing I forgot to share is some experience working on both eth and opstack at the same time,
we were working towards pectra and ishtmus at the same time, and because opstack has it's own implementations or is able to customize the eth default components we didn't encounter much friction at all and adding opstack specific changes was pretty painless overall

[roberts-pumpurs]

from experience, introducing feature flags is a nightmare to maintain, but it would be the fastest way to get something modified.

Why exactly were feature flags hard to maintain?

[roberts-pumpurs]

Currently, I am considering working directly within the reth fork, as I need to prioritise a working solution first-hand. I don't plan on building reth-eth or reth-optimism from the repo, only my own node, meaning that I can skip the feature flag management. In the meantime, when I notice that I need to "add changes to reth core crates", I can still flag this to you, or I can review the diff afterwards. I think this will achieve the same result without me having to manage two distinct codebases.

[hai-rise]

@roberts-pumpurs Do you already have a clear scope of modifications? If the initial scope is small, working directly on a fork may work, as there's less pain when you need to rebase or migrate. If not, using Reth as a library may be a better early investment. It makes your code structure much clearer for scale. If you make the switch late, translating the fork code into the library consumer may not be free as well. The former may change key types directly, while the latter mostly implements anew and traits. You may end up rewriting/throwing lots of fork code. The suggested hybrid method also works well for us, especially for feature branches as we wait for upstream support.

Interestingly, we didn't really face boilerplate for Reth-as-a-library (with the only exception of mempool traits asking for too many methods, some are for blobs which we don't even support as an L2). I'd say you can make very meaningful modifications even at under 1000 lines of code. A custom node module should be under 100 lines. You can start with all Op* types, then one-by-one switch in your own chain types.

The generics and compiler errors may be intimidating at first, but familiarity should grow quickly. I feel you may lose momentum every time you restart from scratch, though. Again, starting small and working towards each incremental compilable checkpoint may help.

Finally, feel free to post code, questions, compile errors, etc., which may lead to more actionable advice.

[gakonst]

Just ACKing that this is explicitly a design goal of the Reth SDK work -- so we'll make it work. Thanks for flagging it. Please get more of your friends to share their feedback / design directions in this thread.

[JesseTheRobot]

As a brief update here: we ended up hijacking legacy transactions with custom data payloads & system specific validation rules (i.e make sure custom data payloads are only accepted in certain cases). you can see the implementation here