File monitoring input

[mbneimann]

The current file input does not support monitoring or tailing for changes, so I suggest to either add this feature or create a new input.

This feature have been suggested before, but never implemented. Probably because of the work it is to create something that works on multiple platforms and can keep track of changes when files are moved, renamed, truncated, etc.

A shortcut for a feature like this would be to look at FileBeat from Elastic. FileBeat is written in Go, is open source under Apache 2.0, and actively maintained.

FileBeat works similar to Bento with inputs --> transformations --> outputs and has delivery guaranties between inputs and outputs, so the basic concepts are not far from each other.

https://www.elastic.co/guide/en/beats/filebeat/current/how-filebeat-works.html
https://github.com/elastic/beats/tree/main/filebeat
https://github.com/elastic/beats

I have used FileBeat on both Windows and Linux and found it very reliable at tracking file changes. I don't like how the rest of FileBeat works and that the outputs are obviously very tailored for ending up in Elasticsearch.

I am not (yet) a Go developer, so can't really point to exactly how to leverage the inspiration from the FileBeat source code. And I am no expert in what is allowed with the software under the Apache 2.0 license.

My first idea however is that it should be possible to make a Bento input that imports and wraps the relevant parts of the filebeat input that reads from files and keeps the state.

[jem-davies]

#181 related issue