Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds section to describe intended usage of pipeline.buffer.type #16083

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
25 changes: 25 additions & 0 deletions docs/static/config-details.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,31 @@ To summarize, we have 3 categories of memory usage, where 2 can be limited by th

Keep these memory requirements in mind as you calculate your ideal memory allocation.

[[reducing-off-heap-usage]]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should better highlight this somehow. The use of this setting completely changes the way Logstash works. Also, it would give us the ability to better understand Logstash when troubleshooting.

However, the direct result of this is that will completely impact/affect JVM Heap sizing. So I propose we:

  1. Bring some more visibility into this setting, maybe say this will be the default in future versions of Logstash.
  2. Tell users that this is a recommended setting to be set?
  3. And, in the memory size/performance tuning sections to have references to this setting being turned on??

Any thoughts?

@jsvd @flexitrev

===== Upcoming changes to Buffer Allocation and Troubleshooting Out of Memory errors

Plugins such as {agent}, {beats}, TCP, and HTTP inputs, currently default to using direct memory as it tends
to provide better performance, especially when interacting with the network stack.
Under heavy load, namely large number of connections and large messages, the direct memory space can be exhausted and lead to Out of Memory (OOM) errors in off-heap space.

An off-heap OOM is difficult to debug, so {ls} provides a `pipeline.buffer.type` setting in <<logstash-settings-file>> that lets you control where to allocate memory buffers for plugins that use them.
Currently it is set to `direct` by default, but you can change it to `heap` to use Java heap space instead, which will be become the default in the future.
When set to `heap`, buffer allocations used by plugins are configured to **prefer** the
Java Heap instead of direct memory, as direct memory allocations may still be necessary depending on the plugin.

When set to "heap", in the event of an out-of-memory, Logstash will produce a heap dump to facilitate debugging.

It is important to note that the Java heap sizing requirements will be impacted by this change since
allocations that previously resided on the direct memory will use heap instead.

Performance-wise there shouldn't be a noticeable impact, since while direct memory IO is faster, Logstash Event objects produced by these plugins end up being allocated on the Java Heap, incurring the cost of copying from direct memory to heap memory regardless of the setting.

[NOTE]
--
* When you set `pipeline.buffer.type` to `heap`, consider incrementing the Java heap by the
amount of memory that had been reserved for direct space.
--

andsel marked this conversation as resolved.
Show resolved Hide resolved
[[memory-size-calculation]]
===== Memory sizing

Expand Down
4 changes: 2 additions & 2 deletions docs/static/settings-file.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -365,6 +365,6 @@ Setting this flag to `warn` is deprecated and will be removed in a future releas

| `pipeline.buffer.type`
| Determine where to allocate memory buffers, for plugins that leverage them.
Default to `direct`, optionally can be switched to `heap` to select Java heap space.
| `direct`
Currently defaults to `direct` but can be switched to `heap` to select Java heap space, which will become the default in the future.
| `direct` Check out <<reducing-off-heap-usage>> for more info.
jsvd marked this conversation as resolved.
Show resolved Hide resolved
|=======================================================================