Flatten bytes class hierarchy #40

lu-pinto · 2025-03-04T15:58:29Z

PR description

This PR simplifies the Tuweni class hierarchy of the bytes project and attempts to get some performance gains by reducing memory churn, specially for copying byte arrays and Bytes instances. Bytes have now a much flatter class hierarchy that will hopefully improve inlining and reduce time on itable lookups which was identified as one of the problems in Besu for low performance. The implementation of Bytes used at runtime is now highly biased towards ArrayWrappingBytes which makes it easier for JIT to inline and optimize.

Also mutable operations wil be moved into MutableBytes exclusively to show intention in copies. This avoids copies on sequences of logic operations, e.g. MutableBytes.create(10).or(Bytes.fromHex("0x11")).shiftRight(2) no further copies occur after initial instance creation.
Other Bytes subtypes should be immutable from now on to boost performance.

Note

Adds new v2 modules for DevP2P (incl. Discovery v5), RLP/SSZ encodings, IO/TLS utilities, and unsigned big-integer types with comprehensive tests.

New v2 libraries and protocols:
- DevP2P (discv4/v5): Adds Endpoint, ENR parsing/validation (EthereumNodeRecord), packet/message types, peers/repository/routing table, and a Vert.x-based Discovery v5 service with handshake, session encryption (AES-GCM/HKDF), and topic table.
- Encoding/Serialization: Introduces v2 RLP (reader/writer/bytes/bytebuffer) and SSZ (reader/writer, fixed/variable lists, hash tree root) implementations with comprehensive tests.
- Crypto/IO Utilities: Adds Base32/Base58/Base64(URL-safe) helpers; TLS utilities with fingerprint repositories and trust managers.
- Big integers: Adds v2 UInt32, UInt64, UInt256, UInt384 with arithmetic/bit ops and tests.
- Extensive tests across new modules ensuring encode/decode, protocol messages, discovery interactions, and cryptographic primitives.

^{Written by Cursor Bugbot for commit ad67e19. This will update automatically on new commits. Configure here.}

github-actions · 2025-03-04T15:58:41Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

lu-pinto · 2025-03-04T16:00:07Z

I have read the CLA Document and I hereby sign the CLA

lu-pinto · 2025-03-04T16:00:26Z

recheck

lu-pinto · 2025-05-21T13:05:07Z

After this comment I will make this PR ready to be pulled in without stopping any development on Tuweni v1 as I'm leaving the current org.apache.tuweni namespace untouched.

I will squash all changes so far and will create an entire new package org.apache.tuweni.v2 which contains the new class hierarchy for Tuweni along with all of its dependents.

Signed-off-by: Luis Pinto <[email protected]>

lu-pinto · 2025-05-22T12:51:40Z

recheck

Signed-off-by: Luis Pinto <[email protected]>

macfarla

do we need to add a consensys copyright statement? Are both v1 and v2 tests running?

bytes/src/jmh/java/org/benchmark/VirtualCall.java

bytes/src/main/java/org/apache/tuweni/v2/bytes/ArrayWrappingBytes.java

bytes/src/test/java/org/apache/tuweni/bytes/ConcatenatedBytesTest.java

bytes/build.gradle

bytes/src/jmh/java/org/benchmark/ProfiledBenchmark.java

bytes/src/main/java/org/apache/tuweni/v2/bytes/ArrayWrappingBytes.java

github-actions · 2025-06-18T05:17:03Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

github-actions · 2025-09-19T05:16:50Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

github-actions · 2025-10-09T05:17:01Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

github-actions · 2025-10-23T05:17:06Z

Closed as inactive. Feel free to reopen if this PR is still being worked on.

cursor · 2025-10-30T01:22:57Z

units/src/main/java/org/apache/tuweni/v2/units/bigints/UInt256.java

+      result[i] = this.ints[i] & other;
+    }
+    return new UInt256(result);
+  }


Bug: Bitwise Operation Errors in UInt Classes

The and, or, and xor methods in both UInt256 and UInt384 classes contain a byte indexing error when operating on Bytes or Bytes32 parameters. The byte offset calculation incorrectly uses the ints array index (i) instead of the bytes parameter index (j), leading to incorrect bitwise results and potential IndexOutOfBoundsException.

Additional Locations (4)

units/src/main/java/org/apache/tuweni/v2/units/bigints/UInt384.java#L457-L464

units/src/main/java/org/apache/tuweni/v2/units/bigints/UInt384.java#L487-L494

units/src/main/java/org/apache/tuweni/v2/units/bigints/UInt384.java#L519-L526

units/src/main/java/org/apache/tuweni/v2/units/bigints/UInt256.java#L637-L644

cursor · 2025-10-30T01:22:57Z

devp2p/src/main/kotlin/org/apache/tuweni/v2/devp2p/Packet.kt

+    fun create(keyPair: SECP256K1.KeyPair, now: Long): Packet {
+      val expiration = expirationFor(now)
+      val sigHash = createSignature(
+        PacketType.ENRRESPONSE,


Bug: Packet Signature Mismatch Causes Verification Failures

The ENRRequestPacket.create() method uses PacketType.ENRRESPONSE for the packet signature. This is inconsistent with the encode() method's use of PacketType.ENRREQUEST and will cause signature verification failures for ENRRequest packets.

cursor · 2025-10-30T01:22:57Z

crypto/src/main/java/org/apache/tuweni/v2/crypto/Hash.java

+      try {
+        SHA256Hash.Hash result = SHA256Hash.hash(shaInput);
+        try {
+          return SHA256Hash.hash(shaInput).bytes();


Bug: Double Hashing Causes Memory Leak

The sha2_256 methods (both byte[] and Bytes overloads) compute the hash twice when Sodium is available. The first hash object is stored and destroyed, but a new hash object is created and returned from a second computation, which is then leaked.

Additional Locations (1)

crypto/src/main/java/org/apache/tuweni/v2/crypto/Hash.java#L106-L107

github-actions · 2025-11-13T05:17:16Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

cursor · 2025-11-13T10:51:59Z

bytes/src/main/java/org/apache/tuweni/v2/bytes/Bytes32.java

+      checkElementIndex(offset, bytes.length);
+    }
+    checkLength(bytes, offset);
+    return new ArrayWrappingBytes(bytes, offset, bytes.length);


Bug: Array Wrapper Overruns Bounds: Incorrect Length

The fromArray method creates an ArrayWrappingBytes with bytes.length as the length parameter, which attempts to wrap more bytes than available when offset > 0. This causes the wrapper to extend beyond the array bounds. The length should be SIZE (32 bytes) to correctly wrap exactly 32 bytes starting from the offset, matching the method's documented behavior and the validation performed by checkLength.

cursor · 2025-11-13T10:51:59Z

bytes/src/main/java/org/apache/tuweni/v2/bytes/ByteBufferWrappingBytes.java

+  @Override
+  public void appendTo(ByteBuffer byteBuffer) {
+    byteBuffer.put(this.byteBuffer);
+  }


Bug: Append Ignores Slice Boundaries

The appendTo method puts the entire byteBuffer into the target buffer, ignoring the offset and size() of this wrapper. When a ByteBufferWrappingBytes represents a slice (constructed with offset and length), this appends the wrong bytes. The method transfers from the buffer's current position/limit rather than the intended slice range defined by offset and size().

IMO it's not a bug

Signed-off-by: Luis Pinto <[email protected]>

…-hierarchy

cursor · 2025-11-14T16:54:19Z

bytes/src/main/java/org/apache/tuweni/v2/bytes/Bytes32.java

+  private static final int SIZE = 32;
+
+  /** A {@code Bytes32} containing all zero bytes */
+  public static final Bytes ZERO = fromByte((byte) 0);


Bug: Type Mismatch: ZERO Constant Threatens Integrity

The ZERO constant is declared as type Bytes instead of Bytes32, causing a type inconsistency. The javadoc states it's a Bytes32 containing all zero bytes, but the actual type doesn't match. This breaks type safety and could cause issues when code expects a Bytes32 instance but receives a generic Bytes reference.

cursor · 2025-11-14T16:54:19Z

bytes/src/main/java/org/apache/tuweni/v2/bytes/Bytes48.java

+  public static final int SIZE = 48;
+
+  /** A {@code Bytes48} containing all zero bytes */
+  public static final Bytes ZERO = fromByte((byte) 0);


Bug: ZERO Constant: Type Inconsistency Threatens Safety

The ZERO constant is declared as type Bytes instead of Bytes48, causing a type inconsistency. The javadoc states it's a Bytes48 containing all zero bytes, but the actual type doesn't match. This breaks type safety and could cause issues when code expects a Bytes48 instance but receives a generic Bytes reference.

cursor · 2025-11-14T16:54:19Z

bytes/src/main/java/org/apache/tuweni/v2/bytes/MutableBytes.java

+      byte[] newBytesArray = new byte[otherSize];
+      System.arraycopy(bytesArray, 0, newBytesArray, otherSize - size, size);
+      bytesArray = newBytesArray;
+      size = otherSize;


Bug: Immutability Contract Violated: Stale Hashcodes

The and, or, and xor methods mutate the size field after object construction when resizing the internal array. Since size is inherited from the immutable parent Bytes class and used for hashcode caching, this mutation breaks the immutability contract and can cause cached hashcodes to become stale, leading to incorrect behavior in hash-based collections.

github-actions bot added a commit that referenced this pull request Mar 4, 2025

@lu-pinto has signed the CLA in #40

5e70cbe

lu-pinto force-pushed the flatten-Bytes-class-hierarchy branch 2 times, most recently from 81cb3e2 to 46f5de9 Compare March 6, 2025 13:33

lu-pinto force-pushed the flatten-Bytes-class-hierarchy branch from 0625d27 to 2dbda56 Compare March 12, 2025 16:36

lu-pinto closed this Apr 4, 2025

lu-pinto reopened this Apr 4, 2025

github-actions bot locked and limited conversation to collaborators Apr 4, 2025

lu-pinto marked this pull request as draft April 8, 2025 15:23

lu-pinto force-pushed the flatten-Bytes-class-hierarchy branch from 1b4e159 to 8a24b0e Compare May 21, 2025 12:54

lu-pinto added 5 commits May 22, 2025 13:34

Implement Tuweni v2

ebaa8bd

Signed-off-by: Luis Pinto <[email protected]>

mv org.apache.tuweni.bytes.v2 to org.apache.tuweni.v2.bytes

65f217b

Signed-off-by: Luis Pinto <[email protected]>

move all other touched files to org.apache.tuweni.v2 package

ace7910

Signed-off-by: Luis Pinto <[email protected]>

bring in old tuweni v1 after move

279acd6

Signed-off-by: Luis Pinto <[email protected]>

fix failure in DefaultDiscoveryV5ServiceTest about reusing ports

1c641ff

Signed-off-by: Luis Pinto <[email protected]>

lu-pinto force-pushed the flatten-Bytes-class-hierarchy branch from 8a24b0e to 1c641ff Compare May 22, 2025 12:49

add missing package-info.java files

5aba562

Signed-off-by: Luis Pinto <[email protected]>

lu-pinto marked this pull request as ready for review May 22, 2025 13:25

Consensys unlocked this conversation May 22, 2025

fix ConnectTwoServersTest - missed package changes and port bindings

dff599a

Signed-off-by: Luis Pinto <[email protected]>

lu-pinto force-pushed the flatten-Bytes-class-hierarchy branch 2 times, most recently from cc29c50 to dff599a Compare May 22, 2025 13:43

lu-pinto mentioned this pull request May 22, 2025

Adopt new Tuweni API changes Consensys/discovery#192

Draft

macfarla reviewed Jun 3, 2025

View reviewed changes

bytes/src/jmh/java/org/benchmark/VirtualCall.java Outdated Show resolved Hide resolved

bytes/src/main/java/org/apache/tuweni/v2/bytes/ArrayWrappingBytes.java Outdated Show resolved Hide resolved

bytes/src/test/java/org/apache/tuweni/bytes/ConcatenatedBytesTest.java Show resolved Hide resolved

ahamlat reviewed Jun 3, 2025

View reviewed changes

macfarla added performance and removed Stale labels Aug 28, 2025

lu-pinto added the Stale label Aug 29, 2025

macfarla removed the Stale label Sep 3, 2025

lu-pinto added performance Stale and removed performance Stale labels Sep 4, 2025

github-actions bot added the Stale label Sep 19, 2025

macfarla removed the Stale label Sep 25, 2025

github-actions bot added the Stale label Oct 9, 2025

github-actions bot closed this Oct 23, 2025

macfarla reopened this Oct 30, 2025

macfarla removed the Stale label Oct 30, 2025

cursor bot reviewed Oct 30, 2025

View reviewed changes

github-actions bot added the Stale label Nov 13, 2025

cursor bot reviewed Nov 13, 2025

View reviewed changes

lu-pinto mentioned this pull request Nov 13, 2025

Potential native libsodium memory leakage in Tuweni Crypto #42

Open

lu-pinto added 3 commits November 13, 2025 14:30

Move bound checks for Bytes implementation out of constructors

578e6bb

Signed-off-by: Luis Pinto <[email protected]>

fix bug in and/or/xor operations with UInt256/UInt384

10e6e81

Signed-off-by: Luis Pinto <[email protected]>

Fix memleak in Hash and bug in devp2p Packet creation

a4def2a

Signed-off-by: Luis Pinto <[email protected]>

lu-pinto force-pushed the flatten-Bytes-class-hierarchy branch from ad67e19 to a4def2a Compare November 13, 2025 14:31

github-actions bot removed the Stale label Nov 14, 2025

lu-pinto added 2 commits November 14, 2025 16:50

Miscelaneous fixes

6d79491

Signed-off-by: Luis Pinto <[email protected]>

Merge remote-tracking branch 'upstream/main' into flatten-Bytes-class…

0929d61

…-hierarchy

cursor bot reviewed Nov 14, 2025

View reviewed changes

Flatten bytes class hierarchy #40

Are you sure you want to change the base?

Flatten bytes class hierarchy #40

Uh oh!

Conversation

lu-pinto commented Mar 4, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description

Uh oh!

github-actions bot commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lu-pinto commented Mar 4, 2025

Uh oh!

lu-pinto commented Mar 4, 2025

Uh oh!

lu-pinto commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lu-pinto commented May 22, 2025

Uh oh!

macfarla left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 18, 2025

Uh oh!

github-actions bot commented Sep 19, 2025

Uh oh!

github-actions bot commented Oct 9, 2025

Uh oh!

github-actions bot commented Oct 23, 2025

Uh oh!

cursor bot Oct 30, 2025

Choose a reason for hiding this comment

Bug: Bitwise Operation Errors in UInt Classes

Uh oh!

cursor bot Oct 30, 2025

Choose a reason for hiding this comment

Bug: Packet Signature Mismatch Causes Verification Failures

Uh oh!

cursor bot Oct 30, 2025

Choose a reason for hiding this comment

Bug: Double Hashing Causes Memory Leak

Uh oh!

github-actions bot commented Nov 13, 2025

Uh oh!

cursor bot Nov 13, 2025

Choose a reason for hiding this comment

Bug: Array Wrapper Overruns Bounds: Incorrect Length

Uh oh!

cursor bot Nov 13, 2025

Choose a reason for hiding this comment

Bug: Append Ignores Slice Boundaries

Uh oh!

lu-pinto Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

cursor bot Nov 14, 2025

Choose a reason for hiding this comment

Bug: Type Mismatch: ZERO Constant Threatens Integrity

Uh oh!

cursor bot Nov 14, 2025

Choose a reason for hiding this comment

Bug: ZERO Constant: Type Inconsistency Threatens Safety

Uh oh!

cursor bot Nov 14, 2025

Choose a reason for hiding this comment

Bug: Immutability Contract Violated: Stale Hashcodes

Uh oh!

Reviewers

Assignees

Labels

Projects

lu-pinto commented Mar 4, 2025 •

edited by cursor bot

Loading

github-actions bot commented Mar 4, 2025 •

edited

Loading

lu-pinto commented May 21, 2025 •

edited

Loading