[ML] Refactoring http settings and adding stats endpoint

[jonathan-buttner]

This PR adds some comments around the connection eviction and keep-alive strategy documentation for the apache http client implementation. I also increased the connection pool limit to exceed a single route's maximum. I've run into a situation during testing where a single service can lease all the connections in the pool and effectively blocking any other services from leasing a connection.

I also created a stats endpoint at /_inference_stats. I was hoping to use /_inference/_stats but theoretically a user could have that as their inference endpoint id 🤷‍♂️

Right now the endpoint returns stats for the internal apache connection pool. I think this could be useful because it could illuminate whether a cluster has reached the maximum connections in the pool.

Example response:

{
  "eTxRlWZzR7uarhC5x9KCtA": {
    "connection_pool_stats": {
      "leased_connections": 0,
      "pending_connections": 0,
      "available_connections": 1,
      "max_connections": 50
    }
  },
  "mTVhyQfxRCyKy83Yoc8Ttw": {
    "connection_pool_stats": {
      "leased_connections": 0,
      "pending_connections": 0,
      "available_connections": 0,
      "max_connections": 50
    }
  },
  "bAeiuk1jQAKkd3NMfqUDnw": {
    "connection_pool_stats": {
      "leased_connections": 0,
      "pending_connections": 0,
      "available_connections": 0,
      "max_connections": 50
    }
  }
}

After discussing with the team, we decided on _inference/.diagnostics. If a user attempts to create an endpoint with a leading dot they'll get this error:

{
    "error": {
        "root_cause": [
            {
                "type": "action_request_validation_exception",
                "reason": "Validation Failed: 1: Invalid inference_id; '.diagnostic' can contain lowercase alphanumeric (a-z and 0-9), hyphens or underscores; must start and end with alphanumeric;"
            }
        ],
        "type": "action_request_validation_exception",
        "reason": "Validation Failed: 1: Invalid inference_id; '.diagnostic' can contain lowercase alphanumeric (a-z and 0-9), hyphens or underscores; must start and end with alphanumeric;"
    },
    "status": 400
}

If they attempt to create an endpoint called .diagnostics they'll get this error:

{
    "error": "Incorrect HTTP method for uri [/_inference/.diagnostics] and method [PUT], allowed: [GET]",
    "status": 405
}

[jonathan-buttner]

Increasing this to 50 by default to exceed the maximum for a single to avoid a single service exhausting the entire connection pool.

[jonathan-buttner]

Allowing the per route limit to be controlled separately. I don't expect users to modify these settings often if ever. I could see these being used if we run into a situation in serverless or an SDH where we can help increase throughput.

[jonathan-buttner]

The common pattern is to use volatile for settings, so moving to that.

[jonathan-buttner]

@elasticmachine merge upstream

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[maxhniebergall]

LGTM!

[maxhniebergall]

LGTM