Skip to content

Conversation

@tarasbob
Copy link

@tarasbob tarasbob commented Dec 2, 2025

Fixes performance degradation when OTLP tracing is enabled by reducing the volume of exported spans.

Problem:
When OTLP tracing is enabled with the default DEBUG filter level, the engine was creating and exporting thousands of spans per second on busy networks. Each transaction generates multiple debug spans:

  • 1 span in execute_metered per transaction
  • 3 spans in prewarming per transaction (recv, prewarm, outcome)

This high volume causes significant overhead:

  • Memory allocation and channel contention in the batch exporter
  • CPU cycles for span serialization
  • Network bandwidth to the OTLP collector
  • Potential OOM from span buffering

Solution:
Downgrade high-frequency per-transaction spans from debug_span! to trace_span! in:

  • crates/engine/tree/src/tree/metrics.rs: execute_metered loop
  • crates/engine/tree/src/tree/payload_processor/prewarm.rs: prewarming loops

Impact:

  • Per-transaction spans are now excluded from default OTLP export
  • Performance returns to normal on busy networks
  • Deep per-transaction debugging still available with --tracing-otlp.filter=trace
  • Orchestration-level spans remain at DEBUG for operational visibility

Testing:

  • All 96 tests in reth-engine-tree pass
  • Project compiles without warnings

…el (paradigmxyz#20058)

Fixes performance degradation when OTLP tracing is enabled by reducing
the volume of exported spans.

Problem:
When OTLP tracing is enabled with the default DEBUG filter level, the
engine was creating and exporting thousands of spans per second on busy
networks. Each transaction generates multiple debug spans:
- 1 span in execute_metered per transaction
- 3 spans in prewarming per transaction (recv, prewarm, outcome)

This high volume causes significant overhead:
- Memory allocation and channel contention in the batch exporter
- CPU cycles for span serialization
- Network bandwidth to the OTLP collector
- Potential OOM from span buffering

Solution:
Downgrade high-frequency per-transaction spans from debug_span! to
trace_span! in:
- crates/engine/tree/src/tree/metrics.rs: execute_metered loop
- crates/engine/tree/src/tree/payload_processor/prewarm.rs: prewarming loops

Impact:
- Per-transaction spans are now excluded from default OTLP export
- Performance returns to normal on busy networks
- Deep per-transaction debugging still available with --tracing-otlp.filter=trace
- Orchestration-level spans remain at DEBUG for operational visibility

Testing:
- All 96 tests in reth-engine-tree pass
- Project compiles without warnings
@mediocregopher
Copy link
Collaborator

cc @shekhirin

@mattsse mattsse requested a review from shekhirin December 6, 2025 07:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

2 participants