Doc: Add tutorial for filter-elastic_integration #15932
Conversation
Co-authored by: Sam Wickline <[email protected]>
karenzone
force-pushed
the
15871-tutorial
branch
from
b08d864
to
55d5045
Compare
|
run docs-build |
There was a problem hiding this comment.
I know this is in draft form, but I'm hoping some suggestions can steer this in a way that de-elevates the implementation details about Ingest Node Pipelines and elevates the plugin's true purpose: running Elastic Integrations inside Logstash.
I'm also hoping to trim out some security-bypassing configuration from the config examples.
Yes, still very much in draft form so that we can get a stake in the ground. But not too early to register directional comments, so thanks for this. |
karenzone
force-pushed
the
15871-tutorial
branch
from
7cec523
to
63962dc
Compare
| hosts => "{es-host}:9200" | ||
| password => "changeme" | ||
| user => "elastic" | ||
| cacert => "/usr/share/logstash/config/certs/ca-cert.pem" |
There was a problem hiding this comment.
Modern versions of the Elasticsearch Output plugin still support legacy settings like cacert, but they also support (and prefer) the normalized SSL settings that match the ones in the Elastic Integration Filter.
| cacert => "/usr/share/logstash/config/certs/ca-cert.pem" | |
| ssl_certificate_authorities => "/usr/share/logstash/config/certs/ca-cert.pem" |
| ----- | ||
| input { | ||
| elastic_agent { port => 5055 } | ||
| } | ||
|
|
||
| filter { | ||
| elastic_integration { | ||
| cloud_id => "your-cloud:id" | ||
| api_key => "api-key" | ||
| remove_field => ["_version"] | ||
| } | ||
| } | ||
|
|
||
| output { | ||
| stdout {} | ||
| elasticsearch { | ||
| cloud_auth => "elastic:<pwd>" | ||
| cloud_id => "your-cloud-id" | ||
| } | ||
| } | ||
| ----- |
There was a problem hiding this comment.
Nit: visually I find single-space indentation to be indistinguishable from arbitrary inconsistent indentation, while two-space indentation is enough to convey intent.
| ----- | |
| input { | |
| elastic_agent { port => 5055 } | |
| } | |
| filter { | |
| elastic_integration { | |
| cloud_id => "your-cloud:id" | |
| api_key => "api-key" | |
| remove_field => ["_version"] | |
| } | |
| } | |
| output { | |
| stdout {} | |
| elasticsearch { | |
| cloud_auth => "elastic:<pwd>" | |
| cloud_id => "your-cloud-id" | |
| } | |
| } | |
| ----- | |
| ----- | |
| input { | |
| elastic_agent { port => 5055 } | |
| } | |
| filter { | |
| elastic_integration { | |
| cloud_id => "your-cloud:id" | |
| api_key => "api-key" | |
| remove_field => ["_version"] | |
| } | |
| } | |
| output { | |
| stdout {} | |
| elasticsearch { | |
| cloud_auth => "elastic:<pwd>" | |
| cloud_id => "your-cloud-id" | |
| } | |
| } | |
| ----- |
| and uses them to apply the transformations from Elastic integrations to further process events before sending them to their | ||
| configured destinations. |
There was a problem hiding this comment.
| and uses them to apply the transformations from Elastic integrations to further process events before sending them to their | |
| configured destinations. | |
| and uses them to apply the transformations from Elastic integrations. This allows you to to further process events in the Logstash pipeline before sending them to their | |
| configured destinations. |
| This tutorial walks you through adding the {integrations-docs}/crowdstrike-intro[Crowdstrike integration], using {ls} to | ||
| remove the `_version` field, and then sending the data to {ess} or self-managed {es}. |
There was a problem hiding this comment.
I believe that removing the _version field was a workaround for the specific ingest pipeline setting the _version field on ingest document instead of on the ingest document's metadata. When it gets put on the metadata, our filter correctly propagates it to the right places on the resulting event so that downstream ES output can choose to use it or not depending on its configuration.
My worry here is that people will copy/paste this config and assume that it is necessary.
| * <<plugins-outputs-elasticsearch,`elasticsearch` output>> | ||
|
|
||
| Note that every event sent from the {agent} to {ls} contains specific meta-fields. | ||
| {ls} expects events to contain `data_stream.type`, `data_stream.dataset`, and `data_stream.namespace`. |
There was a problem hiding this comment.
I don't think we handle incoming being dot-notation, so we can be more clear.
| {ls} expects events to contain `data_stream.type`, `data_stream.dataset`, and `data_stream.namespace`. | |
| {ls} expects events to contain a top-level `data_stream` field with `type`, `dataset`, and `namespace` sub-fields. |
| ----- | ||
| input { | ||
| elastic_agent { port => 5055 } | ||
| } | ||
|
|
||
| filter { | ||
| elastic_integration { | ||
| hosts => "{es-host}:9200" | ||
| ssl_enabled => true | ||
| ssl_certificate_authorities => ["/usr/share/logstash/config/certs/ca-cert.pem"] | ||
| auth_basic_username => "elastic" <1> | ||
| auth_basic_password => "changeme" <2> | ||
| remove_field => ["_version"] | ||
| } | ||
| } | ||
|
|
||
| output { | ||
| stdout { | ||
| codec => rubydebug # to debug datastream inputs | ||
| } | ||
| ## add elasticsearch | ||
| elasticsearch { | ||
| hosts => "{es-host}:9200" | ||
| password => "changeme" | ||
| user => "elastic" | ||
| cacert => "/usr/share/logstash/config/certs/ca-cert.pem" | ||
| } | ||
| } | ||
| ----- |
There was a problem hiding this comment.
Similar nitpick with indentation, plus the elastic_integration filter switched from the descriptive auth_basic_{username,password} to the standard-ish {username,password} in a very early internal release:
| ----- | |
| input { | |
| elastic_agent { port => 5055 } | |
| } | |
| filter { | |
| elastic_integration { | |
| hosts => "{es-host}:9200" | |
| ssl_enabled => true | |
| ssl_certificate_authorities => ["/usr/share/logstash/config/certs/ca-cert.pem"] | |
| auth_basic_username => "elastic" <1> | |
| auth_basic_password => "changeme" <2> | |
| remove_field => ["_version"] | |
| } | |
| } | |
| output { | |
| stdout { | |
| codec => rubydebug # to debug datastream inputs | |
| } | |
| ## add elasticsearch | |
| elasticsearch { | |
| hosts => "{es-host}:9200" | |
| password => "changeme" | |
| user => "elastic" | |
| cacert => "/usr/share/logstash/config/certs/ca-cert.pem" | |
| } | |
| } | |
| ----- | |
| ----- | |
| input { | |
| elastic_agent { port => 5055 } | |
| } | |
| filter { | |
| elastic_integration { | |
| hosts => "{es-host}:9200" | |
| ssl_enabled => true | |
| ssl_certificate_authorities => "/usr/share/logstash/config/certs/ca-cert.pem" | |
| username => "elastic" <1> | |
| password => "changeme" <2> | |
| remove_field => ["_version"] | |
| } | |
| } | |
| output { | |
| stdout { | |
| codec => rubydebug # to debug datastream inputs | |
| } | |
| ## add elasticsearch | |
| elasticsearch { | |
| hosts => "{es-host}:9200" | |
| password => "changeme" | |
| user => "elastic" | |
| ssl_certificate_authorities => "/usr/share/logstash/config/certs/ca-cert.pem" | |
| } | |
| } | |
| ----- |
Starts with tutorial content from gdoc and converts it to asciidoctor format.
Co-authored by: Sam Wickline [email protected]
PREVIEW: https://logstash_bk_15932.docs-preview.app.elstc.co/guide/en/logstash/master/ea-integrations-tutorial.html
Closes: #15871