Skip to content

Commit 527823d

Browse files
authored
Merge pull request #1720 from fluent/lynettemiles/sc-136174/update-fluent-bit-docs-pipeline-inputs-elasticsearch
2 parents 0312ed4 + 7c8bd37 commit 527823d

File tree

1 file changed

+29
-24
lines changed

1 file changed

+29
-24
lines changed

pipeline/inputs/elasticsearch.md

Lines changed: 29 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,43 +1,44 @@
11
# Elasticsearch (Bulk API)
22

3-
The **elasticsearch** input plugin handles both Elasticsearch and OpenSearch Bulk API requests.
3+
The _Elasticsearch_ input plugin handles both Elasticsearch and OpenSearch Bulk API requests.
44

5-
## Configuration Parameters
5+
## Configuration parameters
66

77
The plugin supports the following configuration parameters:
88

99
| Key | Description | Default value |
1010
| :--- | :--- | :--- |
11-
| buffer\_max\_size | Set the maximum size of buffer. | 4M |
12-
| buffer\_chunk\_size | Set the buffer chunk size. | 512K |
13-
| tag\_key | Specify a key name for extracting as a tag. | `NULL` |
14-
| meta\_key | Specify a key name for meta information. | "@meta" |
15-
| hostname | Specify hostname or FQDN. This parameter can be used for "sniffing" (auto-discovery of) cluster node information. | "localhost" |
16-
| version | Specify Elasticsearch server version. This parameter is effective for checking a version of Elasticsearch/OpenSearch server version. | "8.0.0" |
17-
| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
18-
19-
**Note:** The Elasticsearch cluster uses "sniffing" to optimize the connections between its cluster and clients.
11+
| `buffer_max_size` | Set the maximum size of buffer. | `4M` |
12+
| `buffer_chunk_size` | Set the buffer chunk size. | `512K` |
13+
| `tag_key` | Specify a key name for extracting as a tag. | `NULL` |
14+
| `meta_key` | Specify a key name for meta information. | "@meta" |
15+
| `hostname` | Specify hostname or fully qualified domain name. This parameter can be used for "sniffing" (auto-discovery of) cluster node information. | "localhost" |
16+
| `version` | Specify Elasticsearch server version. This parameter is effective for checking a version of Elasticsearch/OpenSearch server version. | "8.0.0" |
17+
| `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
18+
19+
The Elasticsearch cluster uses "sniffing" to optimize the connections between its cluster and clients.
2020
Elasticsearch can build its cluster and dynamically generate a connection list which is called "sniffing".
2121
The `hostname` will be used for sniffing information and this is handled by the sniffing endpoint.
2222

23-
## Getting Started
23+
## Get started
2424

2525
In order to start performing the checks, you can run the plugin from the command line or through the configuration file:
2626

27-
### Command Line
27+
### Command line
2828

2929
From the command line you can configure Fluent Bit to handle Bulk API requests with the following options:
3030

3131
```bash
32-
$ fluent-bit -i elasticsearch -p port=9200 -o stdout
32+
fluent-bit -i elasticsearch -p port=9200 -o stdout
3333
```
3434

35-
### Configuration File
35+
### Configuration file
3636

37-
In your main configuration file append the following _Input_ & _Output_ sections:
37+
In your configuration file append the following `Input` and `Output` sections:
3838

3939
{% tabs %}
4040
{% tab title="fluent-bit.conf" %}
41+
4142
```python
4243
[INPUT]
4344
name elasticsearch
@@ -48,9 +49,11 @@ In your main configuration file append the following _Input_ & _Output_ sections
4849
name stdout
4950
match *
5051
```
52+
5153
{% endtab %}
5254

5355
{% tab title="fluent-bit.yaml" %}
56+
5457
```yaml
5558
pipeline:
5659
inputs:
@@ -62,14 +65,16 @@ pipeline:
6265
- name: stdout
6366
match: '*'
6467
```
68+
6569
{% endtab %}
6670
{% endtabs %}
6771
68-
As described above, the plugin will handle ingested Bulk API requests.
69-
For large bulk ingestions, you may have to increase buffer size with **buffer_max_size** and **buffer_chunk_size** parameters:
72+
As described previously, the plugin will handle ingested Bulk API requests.
73+
For large bulk ingestion, you might have to increase buffer size using the `buffer_max_size` and `buffer_chunk_size` parameters:
7074

7175
{% tabs %}
7276
{% tab title="fluent-bit.conf" %}
77+
7378
```python
7479
[INPUT]
7580
name elasticsearch
@@ -82,9 +87,11 @@ For large bulk ingestions, you may have to increase buffer size with **buffer_ma
8287
name stdout
8388
match *
8489
```
90+
8591
{% endtab %}
8692

8793
{% tab title="fluent-bit.yaml" %}
94+
8895
```yaml
8996
pipeline:
9097
inputs:
@@ -98,6 +105,7 @@ pipeline:
98105
- name: stdout
99106
match: '*'
100107
```
108+
101109
{% endtab %}
102110
{% endtabs %}
103111

@@ -106,20 +114,17 @@ pipeline:
106114
Ingesting from beats series agents is also supported.
107115
For example, [Filebeats](https://www.elastic.co/beats/filebeat), [Metricbeat](https://www.elastic.co/beats/metricbeat), and [Winlogbeat](https://www.elastic.co/beats/winlogbeat) are able to ingest their collected data through this plugin.
108116

109-
Note that Fluent Bit's node information is returning as Elasticsearch 8.0.0.
117+
The Fluent Bit node information is returning as Elasticsearch 8.0.0.
110118

111-
So, users have to specify the following configurations on their beats configurations:
119+
Users must specify the following configurations on their beats configurations:
112120

113121
```yaml
114122
output.elasticsearch:
115123
allow_older_versions: true
116124
ilm: false
117125
```
118126

119-
For large log ingestion on these beat plugins,
120-
users might have to configure rate limiting on those beats plugins
121-
when Fluent Bit indicates that the application is exceeding the size limit for HTTP requests:
122-
127+
For large log ingestion on these beat plugins, users might have to configure rate limiting on those beats plugins when Fluent Bit indicates that the application is exceeding the size limit for HTTP requests:
123128

124129
```yaml
125130
processors:

0 commit comments

Comments
 (0)