[Protocol][Shannon][Observability] Protocol shannon HTTP timeout metrics (#383)

## Summary

Replace simple HTTP relay function with comprehensive HTTP client
featuring request tracing and metrics collection

### Primary Changes:

- Introduced new httpClientWithTracing struct with detailed HTTP request
lifecycle monitoring and atomic counters for performance metrics
- Replaced sendHttpRelay function with SendHTTPRelay method that
includes comprehensive error categorization and request tracing
- Added HTTP transport optimization with configurable connection
pooling, timeout settings, and keep-alive configuration

### Secondary Changes:

- Updated requestContext and Protocol structs to include httpClient
field for centralized HTTP client management
- Added detailed logging with timing breakdowns for DNS lookup,
connection establishment, TLS handshake, and response times
- Implemented graceful shutdown functionality with connection cleanup
and statistics reporting

## Issue

Missing details on HTTP timeouts when sending relay requests to
endpoints.

- Issue or PR: #{ISSUE_OR_PR_NUMBER}

## Type of change

Select one or more from the following:

- [x] New feature, functionality or library
- [ ] Bug fix
- [ ] Code health or cleanup
- [ ] Documentation
- [ ] Other (specify)

## QoS Checklist

### E2E Validation & Tests

- [ ] `make path_up`
- [ ] `make test_e2e_evm_shannon`

### Observability

- [ ] 1. `make path_up`
- [ ] 2. Run the following E2E test: `make
test_request__shannon_relay_util_100`
- [ ] 3. View results in LocalNet's [PATH Relay Grafana
Dashboard](http://localhost:3003/d/relays/path-service-requests)

## Sanity Checklist

- [x] I have updated the GitHub Issue `assignees`, `reviewers`,
`labels`, `project`, `iteration` and `milestone`
- [ ] For docs, I have run `make docusaurus_start`
- [ ] For code, I have run `make test_all`
- [ ] For configurations, I have updated the documentation
- [x] I added `TODO`s where applicable

---------

Co-authored-by: Daniel Olshansky <[email protected]>

Author: adshmh

observation/protocol/shannon.pb.go

@@ -45,7 +45,8 @@ message ShannonRequestError {
 enum ShannonEndpointErrorType {
   SHANNON_ENDPOINT_ERROR_UNSPECIFIED = 0;
   // endpoint internal error: not recognized.
-  SHANNON_ENDPOINT_ERROR_INTERNAL = 1;
+  // DEPRECATED: Use more specific error types below (HTTP_UNKNOWN, RAW_PAYLOAD_UNKNOWN, UNKNOWN)
+  SHANNON_ENDPOINT_ERROR_INTERNAL = 1 [deprecated = true];
   // endpoint config error: e.g. DNS lookup error, TLS certificate error.
   SHANNON_ENDPOINT_ERROR_CONFIG = 2;
   // endpoint timeout on responding to relay request.
@@ -95,6 +96,13 @@ enum ShannonEndpointErrorType {
 
   // RelayRequest was canceled by PATH intentionally.
   SHANNON_ENDPOINT_REQUEST_CANCELED_BY_PATH = 33;
+
+  // New error types for better classification (added without renumbering existing ones)
+  // Error was recognized but specific classification failed.
+  // Only general category: HTTP, Raw Payload, General, is known.
+  SHANNON_ENDPOINT_ERROR_HTTP_UNKNOWN = 34;
+  SHANNON_ENDPOINT_ERROR_RAW_PAYLOAD_UNKNOWN = 35;
+  SHANNON_ENDPOINT_ERROR_UNKNOWN = 36;
 }
 
 // ShannonSanctionType specifies the duration type for endpoint sanctions

proto/path/protocol/shannon.proto

@@ -70,6 +70,9 @@ type requestContext struct {
 	// - Tracks RelayMinerError data from the current relay response for reporting.
 	// - Set by trackRelayMinerError method and used when building observations.
 	currentRelayMinerError *protocolobservations.ShannonRelayMinerError
+
+	// HTTP client used for sending relay requests to endpoints while also capturing various debug metrics
+	httpClient *httpClientWithDebugMetrics
 }
 
 // HandleServiceRequest:
@@ -244,7 +247,14 @@ func (rc *requestContext) sendRelay(payload protocol.Payload) (*servicetypes.Rel
 	headers := buildHeaders(payload)
 
 	// Send the HTTP relay request
-	httpRelayResponseBz, err := sendHttpRelay(ctxWithTimeout, rc.selectedEndpoint.url, signedRelayReq, headers)
+	httpRelayResponseBz, err := rc.httpClient.SendHTTPRelay(
+		ctxWithTimeout,
+		hydratedLogger,
+		rc.selectedEndpoint.url,
+		signedRelayReq,
+		headers,
+	)
+
 	if err != nil {
 		// Endpoint failed to respond before the timeout expires.
 		// Wrap the net/http error with our classification error