buffer space has too many data

[scalp42]

Hi folks,

Without having any infra change or increase in traffic, we started seeing the following issue:

2019-09-30 17:29:21 +0000 [error]: #0 syslog failed to emit error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" tag="syslog.docker.containers.daemon.info" 

We looked at increasing flush_thread_count but noticed the README mentioning:

This is currently fixed to 1 will cause fluentd to fail with a ConfigError if set to anything greater.

We believe the issue is due to the output not matching our input rate but we never ran into that issue before. What's the rationale behind limiting the number of threads for the output plugin?

Thanks!

cc @imron @czerwingithub

[scalp42]

Same issue trying to use multiple workers:

2019-09-30 20:33:11 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Plugin 'scalyr' does not support multi workers configuration (Scalyr::ScalyrOut)"

[imron]

Hi @scalp42 , thanks for getting in touch. The restriction on multi-workers was originally put in place due to an early limitation of the Scalyr servers, however those restrictions are no longer applicable so we'll be looking at providing an update that removes that restriction.

It also looks like your problem was if not caused, then at least exacerbated by a separate issue with the scalyr servers (I noticed your logs had a number of "error/server" messages), and we're currently looking at that also.

Fixes to one of both of these things should hopefully sort out this problem.