Add evb violation counter for exchange client and http server IO threadpool #24545
Conversation
|
@kewang1024 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
kewang1024
force-pushed
the
add-evb-counter
branch
from
f010235 to
a36eeae
Compare
|
@kewang1024 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
kewang1024
force-pushed
the
add-evb-counter
branch
from
a36eeae to
a932ddc
Compare
|
@kewang1024 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
kewang1024
force-pushed
the
add-evb-counter
branch
from
a932ddc to
580e51c
Compare
|
@kewang1024 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
| for (auto evb : httpSrvIoExecutor_->getAllEventBases()) { | ||
| evb->setMaxLatency( | ||
| std::chrono::milliseconds(systemConfig->httpSrvIoEvbViolationThresholdMs()), | ||
| []() { RECORD_METRIC_VALUE(kCounterHttpServerIoEvbViolation, 1); }, | ||
| false); | ||
| } |
There was a problem hiding this comment.
What is this setMaxLatency() for?
For executing http callback?
So what we doing here is setting a latency for our callbacks and if it is breached the given functor will be executed?
Do we suspect some of our callbacks talking long time on http executor?
Will this setMaxLatency() fail the http call if limit breached?
kewang1024
changed the title
| @@ -409,6 +409,12 @@ void PrestoServer::run() { | |||
| << exchangeHttpIoExecutor_->getName() << "' has " | |||
| << exchangeHttpIoExecutor_->numThreads() | |||
| << " threads."; | |||
| for (auto evb : exchangeHttpIoExecutor_->getAllEventBases()) { | |||
| evb->setMaxLatency( | |||
There was a problem hiding this comment.
@kewang1024 : What is this for ? How do we turn this off ? Since we have a platform offering we would want to make this user (our client) configurable.
Are there any instructions for us on how to tune this for different clusters ?
There was a problem hiding this comment.
If this is user-configurable, documentation should be included in this PR.
When prestissimo worker is busy on CPU, we observed delays in exchange connection.
Add evb violation for exchange client IO thread pool and exchange server IO thread pool
to give us better visibility.