[ML] Refactoring http settings and adding stats endpoint

[jonathan-buttner]

This PR adds some comments around the connection eviction and keep-alive strategy documentation for the apache http client implementation. I also increased the connection pool limit to exceed a single route's maximum. I've run into a situation during testing where a single service can lease all the connections in the pool and effectively blocking any other services from leasing a connection.

I also created a stats endpoint at /_inference_stats. I was hoping to use /_inference/_stats but theoretically a user could have that as their inference endpoint id 🤷‍♂️

Right now the endpoint returns stats for the internal apache connection pool. I think this could be useful because it could illuminate whether a cluster has reached the maximum connections in the pool.

Example response:

{
  "eTxRlWZzR7uarhC5x9KCtA": {
    "connection_pool_stats": {
      "leased_connections": 0,
      "pending_connections": 0,
      "available_connections": 1,
      "max_connections": 50
    }
  },
  "mTVhyQfxRCyKy83Yoc8Ttw": {
    "connection_pool_stats": {
      "leased_connections": 0,
      "pending_connections": 0,
      "available_connections": 0,
      "max_connections": 50
    }
  },
  "bAeiuk1jQAKkd3NMfqUDnw": {
    "connection_pool_stats": {
      "leased_connections": 0,
      "pending_connections": 0,
      "available_connections": 0,
      "max_connections": 50
    }
  }
}

[jonathan-buttner]

Increasing this to 50 by default to exceed the maximum for a single to avoid a single service exhausting the entire connection pool.

[jonathan-buttner]

Allowing the per route limit to be controlled separately. I don't expect users to modify these settings often if ever. I could see these being used if we run into a situation in serverless or an SDH where we can help increase throughput.

[jonathan-buttner]

The common pattern is to use volatile for settings, so moving to that.

[jonathan-buttner]

@elasticmachine merge upstream

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[maxhniebergall]

LGTM!