feat: client side load balancing for user transformations

[lvrach]

Description

This PR is attempting to address an imbalance issue observed in production. The rudder-server client tends to open more connections proportionally to older UT or ha-proxy pods. Not allowing the downstream system to scale horizontally.

The PR introduces two alternative HTTP clients to address that:

1. recycled: A new client is created every x minutes (configurable). This allows TCP connections to be reestablished in a more balanced way. However:
   • This approach doesn't guarantee perfect balancing but doesn't allow the system to deteriorate enough.
   • There is an overhead in connection creation and clean-up.
   • There a possibility of resource leak
2. httplb : load-balances TCP connections between available endpoint. Endpoint discovery is done via DNS and requires a headless service in k8s to work. It refreshes newly added or removed pods using Transformer.Client.ttl. It uses the same transport as the standard client and the same configuration.
3. stdlib: standard HTTP library, what we have today.

Clients can be selected from Transformer.Client.type config, by default stdlib is used.

Testing

• recycled was heavily tested in load-test environment under heavy load for a long time. It manage to maintain good enough latencies for a dedicated user transformer.

• httplb is also tested in the same environment, but with mixed results:

  • It maintains perfect balance among all pods.
  • It causes ut pod HPA to scale up and down constantly, fine-tuning HPA might be required
  • Also we want to verify how well it balances between different processes of UT.

Note: due to node.js clustering, UT is spawning multiple processes, all subscribed to the same port.

Here you can see the comparison between recycled (left) and httplb (right):

Assuming HPA stability is fixed, httplb would fair much better than recycled.

Linear Ticket

https://linear.app/rudderstack/issue/PIPE-1720/client-side-load-balancing-for-transformations

resolves PIPE-1720

Security

• The code changed/added as part of this pull request won't create any security issues with how the software is being used.

[codecov]

Codecov Report

Attention: Patch coverage is 32.20339% with 40 lines in your changes missing coverage. Please review.

Project coverage is 74.82%. Comparing base (e504d75) to head (1cce0fe).
Report is 1 commits behind head on release/1.40.x.

Files with missing lines	Patch %	Lines
processor/transformer/transformer.go	47.50%	21 Missing ⚠️
utils/sysUtils/httpclient.go	0.00%	19 Missing ⚠️

Additional details and impacted files

@@                Coverage Diff                 @@
##           release/1.40.x    #5375      +/-   ##
==================================================
+ Coverage           74.77%   74.82%   +0.04%     
==================================================
  Files                 439      440       +1     
  Lines               61378    61422      +44     
==================================================
+ Hits                45895    45957      +62     
+ Misses              12940    12928      -12     
+ Partials             2543     2537       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ktgowtham]

LGTM