Description
Hello,
I would like to monitor the native memory that is consumed by a Quarkus application.
For Rest applications, I know, Netty and Vert.x are the core stacks of Quarkus.
However when monitoring the RSS size of a Quarkus application, I would like to have comprehensive reporting of the memory used in the Java application.
As an example, I have this kind of monitoring dashboard where I am able to trace the JVM memory used (bright yellow), commited (dark yellow) which are the sum of on-heap + off-heap memory metrics and compare it against the RSS container size. But there's blind spot (represented by the red arrow) that is not covered and I would like to monitor it as well.
If I am not mistaken at the heart of Netty's performance is the ability to allocate native memory using Unsafe and manages it on its own.
I believe a Quarkus application is using this native memory allocation through Unsafe, because allocation of direct memory using the JDK's ByteBuffer is tedious (and limited).
Also Since Quarkus 3.8, Netty allocator micrometer metrics are reported thanks to this PR, I can see now netty related metrics :
# TYPE netty_allocator_pooled_chunk_size gauge
# HELP netty_allocator_pooled_chunk_size
netty_allocator_pooled_chunk_size{allocator_type="PooledByteBufAllocator",id="528280723"} 65536.0
netty_allocator_pooled_chunk_size{allocator_type="PooledByteBufAllocator",id="513324936"} 65536.0
# TYPE netty_allocator_memory_pinned gauge
# HELP netty_allocator_memory_pinned
netty_allocator_memory_pinned{allocator_type="PooledByteBufAllocator",id="528280723",memory_type="heap"} 0.0
netty_allocator_memory_pinned{allocator_type="PooledByteBufAllocator",id="528280723",memory_type="direct"} 0.0
netty_allocator_memory_pinned{allocator_type="PooledByteBufAllocator",id="513324936",memory_type="direct"} 0.0
netty_allocator_memory_pinned{allocator_type="PooledByteBufAllocator",id="513324936",memory_type="heap"} 0.0
# TYPE netty_eventexecutor_tasks_pending gauge
# HELP netty_eventexecutor_tasks_pending
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-10"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-9"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-11"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-8"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-12"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-7"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-13"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-6"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-14"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-5"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-15"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-4"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-3"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-2"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-1"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-0"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-acceptor-thread-0"} 0.0
# TYPE netty_allocator_memory_used gauge
# HELP netty_allocator_memory_used
netty_allocator_memory_used{allocator_type="UnpooledByteBufAllocator",id="1520840423",memory_type="direct"} 286.0
netty_allocator_memory_used{allocator_type="UnpooledByteBufAllocator",id="384735911",memory_type="direct"} 0.0
netty_allocator_memory_used{allocator_type="PooledByteBufAllocator",id="528280723",memory_type="heap"} 0.0
netty_allocator_memory_used{allocator_type="UnpooledByteBufAllocator",id="1520840423",memory_type="heap"} 2690.0
netty_allocator_memory_used{allocator_type="UnpooledByteBufAllocator",id="1929576211",memory_type="direct"} 0.0
netty_allocator_memory_used{allocator_type="PooledByteBufAllocator",id="528280723",memory_type="direct"} 327680.0
netty_allocator_memory_used{allocator_type="UnpooledByteBufAllocator",id="1929576211",memory_type="heap"} 0.0
netty_allocator_memory_used{allocator_type="UnpooledByteBufAllocator",id="384735911",memory_type="heap"} 0.0
netty_allocator_memory_used{allocator_type="PooledByteBufAllocator",id="513324936",memory_type="direct"} 720896.0
netty_allocator_memory_used{allocator_type="PooledByteBufAllocator",id="513324936",memory_type="heap"} 0.0
# TYPE netty_allocator_pooled_threadlocal_caches gauge
# HELP netty_allocator_pooled_threadlocal_caches
netty_allocator_pooled_threadlocal_caches{allocator_type="PooledByteBufAllocator",id="528280723"} 1.0
netty_allocator_pooled_threadlocal_caches{allocator_type="PooledByteBufAllocator",id="513324936"} 4.0
# TYPE netty_allocator_pooled_arenas gauge
# HELP netty_allocator_pooled_arenas
netty_allocator_pooled_arenas{allocator_type="PooledByteBufAllocator",id="528280723",memory_type="heap"} 32.0
netty_allocator_pooled_arenas{allocator_type="PooledByteBufAllocator",id="528280723",memory_type="direct"} 32.0
netty_allocator_pooled_arenas{allocator_type="PooledByteBufAllocator",id="513324936",memory_type="direct"} 32.0
netty_allocator_pooled_arenas{allocator_type="PooledByteBufAllocator",id="513324936",memory_type="heap"} 32.0
# TYPE netty_allocator_pooled_cache_size gauge
# HELP netty_allocator_pooled_cache_size
netty_allocator_pooled_cache_size{allocator_type="PooledByteBufAllocator",cache_type="small",id="513324936"} 256.0
netty_allocator_pooled_cache_size{allocator_type="PooledByteBufAllocator",cache_type="normal",id="513324936"} 64.0
netty_allocator_pooled_cache_size{allocator_type="PooledByteBufAllocator",cache_type="normal",id="528280723"} 64.0
netty_allocator_pooled_cache_size{allocator_type="PooledByteBufAllocator",cache_type="small",id="528280723"} 256.0
However the metrics are a bit cryptic (even if some documentation can be found here) but I am not sure how to compute the total direct/native memory reported by Netty without any mistakes.
So my question is : is there any references dashboard, precise Netty documentation or some Netty experts that could help me building it ?
If some clarifications are done, I could provide a PR in Quarkus to add this information on a guide if you think this is useful for Quarkus developers.
Thanks a lot for your help
Metadata
Metadata
Assignees
Type
Projects
Status