Add test cases for RemoteConnector retry behavior

[peternied]

Description

Add test cases for RemoteConnector retry behavior

Refactored product code to make testing easier. Build up these tests to confirm behavior when getting a non-throttled response that was reported by a customer.

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dhrubo-os]

Could you please take updates from main to pass the integ tests. Thanks.

[peternied]

Thanks @dhrubo-os merged from main.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.30%. Comparing base (d7dec0f) to head (b5bf349).
Report is 5 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3504      +/-   ##
============================================
+ Coverage     80.25%   80.30%   +0.05%     
- Complexity     6906     6911       +5     
============================================
  Files           610      610              
  Lines         30077    30094      +17     
  Branches       3368     3368              
============================================
+ Hits          24137    24166      +29     
+ Misses         4487     4478       -9     
+ Partials       1453     1450       -3     

Flag	Coverage Δ
ml-commons	80.30% <100.00%> (+0.05%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pyek-bot]

Hi @peternied, just curious, can you please share more context on why this is being done? I see that you mention Build up these tests to confirm behavior when getting a **non-throttled response that was reported by a customer.** Can you share what this report was?

[peternied]

@pyek-bot Sure - I can provide a little more context. Below is a redacted version of their configuration, they setup the client_config policy to retry indefinitely with exponential backoff. However, they never saw any failed requests retried.

Probing in the code base to be sure why this was the case - this looks to be by design. I noticed a lack of test coverage and figured it would be good to area to augment. Especially if someone is interested in changing the behavior so retries occur on a broader set of failed responses.

Redacted configuration

{
  ...
  "parameters": {
    "embeddingTypes": """["float"]""",
    "client_config": """{"retry_timeout_seconds":60,"max_retry_times":-1,"retry_backoff_policy":"exponential_full_jitter"}""",
    ...
  },
  "actions": [
    {
      "action_type": "PREDICT",
      "method": "POST",
      "url": "https://...{REDACTED}...",
      "request_body": """{ "inputText": "${parameters.inputText}", "dimensions": ${parameters.dimensions}, "normalize": ${parameters.normalize}, "embeddingTypes": ${parameters.embeddingTypes} }""",
      ...
    }
  ]
}