feat: Add consistent batch request metrics tracking across all QoS services #485
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Addresses memory exhaustion issues causing 12GB RAM OOM crashes: 1. Add 100MB request body size limits (supports Solana's ~75MB blocks) 2. Cap endpoint observations per request (uses MaxConcurrentRelaysPerRequest) 3. Reduce WebSocket observation channel buffer from 1000 to 50 4. Add hydrator graceful shutdown with context cancellation 5. Add 30s timeouts to hydrator operations
Additional fixes completing the OOM prevention release: - 2.3 Session rollover: Add context for graceful shutdown of block height monitor - 2.4 Observation goroutines: Add 30s timeout to prevent indefinite hanging - 2.5 time.After leak: Replace with time.NewTimer + defer Stop() - 2.6 WebSocket cleanup: Close client connection if endpoint connection fails
Per JSON-RPC 2.0 spec (https://www.jsonrpc.org/specification), responses with null IDs are valid for error cases when the server couldn't parse the request ID. This is documented in Section 5 - Response object: "If there was an error in detecting the id in the Request object (e.g. Parse error/Invalid Request), it MUST be Null." Changes: - Update validateResponseIDs to treat null ID responses as "wildcards" that can match unmatched request IDs - Update createResponseObservations to skip null ID responses gracefully with debug logging instead of error logging - Downgrade "could not find request for response ID" from error to warn
Preserves original HTTP status codes from backend endpoints instead of transforming them based on JSON-RPC error codes. This allows clients to receive accurate HTTP status information (e.g., 429 Too Many Requests, 503 Service Unavailable) from backend services. Changes: - Update RequestQoSContext interface to include httpStatusCode parameter - Modify all QoS implementations (EVM, Solana, Cosmos, NoOp) to capture and propagate HTTP status codes - Change protocol/shannon to pass through non-2xx responses instead of returning errors - Make qos.HTTPResponse fields public for cross-package access
jorgecuesta
force-pushed
the
fix/batch-metrics-tracking
branch
from
f8c7739 to
6c4664c
Compare
…rvices Adds `is_batch_request` label and batch size histogram to all QoS services: Cosmos: - Add `is_batch_request` label to requestsTotal metric - Add `cosmos_batch_request_size` histogram Solana: - Add `GetRequestMethods()` method to interpreter for batch support - Add `is_batch_request` label to requestsTotal metric - Add `solana_batch_request_size` histogram - Update PublishMetrics to iterate through methods like EVM/Cosmos EVM: - Add `evm_batch_request_size` histogram (already had is_batch_request) This enables consistent batch request visibility across all services: - Filter batch vs single requests in Prometheus - Analyze batch size distribution patterns - Capacity planning based on batch request patterns
jorgecuesta
force-pushed
the
fix/batch-metrics-tracking
branch
from
6c4664c to
77df29e
Compare
Contributor
|
change were contained on fix/memory-optimization |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
is_batch_requestlabel to Cosmos and Solana metrics (EVM already had it)*_batch_request_size) to all QoS services (EVM, Cosmos, Solana)GetRequestMethods()method to Solana interpreter for batch supportChanges by Service
is_batch_requestLabelNew Metrics
path_evm_batch_request_size- Distribution of EVM batch request sizespath_cosmos_batch_request_size- Distribution of Cosmos batch request sizespath_solana_batch_request_size- Distribution of Solana batch request sizesTest plan
make test_unit)make go_lint)