Adds general S3 use-case documentation. #3833 #3842

dlvenable · 2023-04-24T20:58:35Z

Description

This adds a new use-case to Data Prepper for S3 logs.

Issues Resolved

Contributes toward #3833

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: David Venable <dlv@amazon.com>

hdhalter

A few suggestions... thanks!

_data-prepper/common-use-cases/s3-logs.md

hdhalter · 2023-04-27T20:31:53Z

_data-prepper/common-use-cases/s3-logs.md

+
+## Architecture
+
+Data Prepper supports reading objects from S3 buckets using an [Amazon Simple Queue Service (SQS)](https://aws.amazon.com/sqs/) (Amazon SQS) queue and [Amazon S3 Event Notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html).


Same formulation as above

Switched to "can read objects".

_data-prepper/common-use-cases/s3-logs.md

Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

cwillum

Looks good.

cwillum · 2023-04-27T20:49:14Z

_data-prepper/common-use-cases/s3-logs.md

+
+<img src="{{site.url}}{{site.baseurl}}/images/data-prepper/s3-source/s3-pipeline.jpg" alt="S3 source architecture">{: .img-fluid}
+
+## Prerequisites


Some of the H2 headings have two spaces above, others a single space. Settle on one or the other.

_data-prepper/common-use-cases/s3-logs.md

Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Adds general S3 use-case documentation. #3833 Signed-off-by: David Venable <dlv@amazon.com> * Add edits to S3 logs doc * Apply suggestions from code review Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update s3-logs.md * Update s3-logs.md * Apply suggestions from code review Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: David Venable <dlv@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> (cherry picked from commit 196c750) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Adds general S3 use-case documentation. #3833 * Add edits to S3 logs doc * Apply suggestions from code review * Update s3-logs.md * Update s3-logs.md * Apply suggestions from code review --------- (cherry picked from commit 196c750) Signed-off-by: David Venable <dlv@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>

natebower

@Naarcha-AWS Please see my comments/changes and let me know if you have any questions. Thanks!

natebower · 2023-05-02T11:30:00Z

_data-prepper/common-use-cases/s3-logs.md

+
+## Architecture
+
+Data Prepper can read objects from S3 buckets using an [Amazon Simple Queue Service (SQS)](https://aws.amazon.com/sqs/) (Amazon SQS) queue and [Amazon S3 Event Notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html).


Remove "(SQS)".

natebower · 2023-05-02T11:30:34Z

_data-prepper/common-use-cases/s3-logs.md

+
+<img src="{{site.url}}{{site.baseurl}}/images/data-prepper/s3-source/s3-architecture.jpg" alt="S3 source architecture">{: .img-fluid}
+
+The flow of data is as follows.


Replace the period with a colon.

natebower · 2023-05-02T11:33:29Z

_data-prepper/common-use-cases/s3-logs.md

+
+The flow of data is as follows.
+
+1. A system produces logs into the S3 bucket.


I'm not sure one can "produce into". Do we mean "A system sends logs to an S3 bucket"?

natebower · 2023-05-02T11:34:33Z

_data-prepper/common-use-cases/s3-logs.md

+The flow of data is as follows.
+
+1. A system produces logs into the S3 bucket.
+2. S3 creates an S3 event notification in the SQS queue.


Replace "the" with "an".

natebower · 2023-05-02T11:35:27Z

_data-prepper/common-use-cases/s3-logs.md

+2. S3 creates an S3 event notification in the SQS queue.
+3. Data Prepper polls Amazon SQS for messages and then receives a message.
+4. Data Prepper downloads the content from the S3 object.
+5. Data Prepper sends a document to OpenSearch for the content in the S3 object.


This sentence is a bit unclear. Instead of "for", do we mean something like "containing"?

natebower · 2023-05-02T11:48:30Z

_data-prepper/common-use-cases/s3-logs.md

+
+## Pipeline design
+
+Create a pipeline to read logs from S3, starting with an [`s3`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/s3/) source plugin. Use the following example for guidance. 


"Create a pipeline for reading logs from S3"? Replace the terminal period with a colon.

natebower · 2023-05-02T11:50:58Z

_data-prepper/common-use-cases/s3-logs.md

+
+Configure the following options according to your use case:
+
+* `queue_url`: This the SQS queue URL and is always unique to your pipeline.


natebower · 2023-05-02T11:52:24Z

_data-prepper/common-use-cases/s3-logs.md

+* `codec`: The codec determines how to parse the incoming data.
+* `visibility_timeout`: Configure this value to be large enough for Data Prepper to process 10 S3 objects. However, if you make this value too large, messages that fail to process will take at least as long as the specified value before Data Prepper retries.
+
+The default values for each option work for the majority of use cases. For all available options for the S3 source, see [`s3`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/s3/).


"For all available S3 source options"

natebower · 2023-05-02T11:53:16Z

_data-prepper/common-use-cases/s3-logs.md

+
+The default values for each option work for the majority of use cases. For all available options for the S3 source, see [`s3`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/s3/).
+
+```yaml


This needs to be introduced by a brief sentence ending in a colon.

natebower · 2023-05-02T11:55:16Z

_data-prepper/common-use-cases/s3-logs.md

+
+## Multiple Data Prepper pipelines
+
+We recommend that you have one SQS queue per Data Prepper pipeline. In addition, you can have multiple nodes in the same cluster reading from the same SQS queue, which doesn't require additional configuration with Data Prepper.


"We recommend that you create one SQS queue per Data Prepper pipeline. Additionally, multiple nodes in the same cluster can read from the same SQS queue, which doesn't require additional Data Prepper configuration."

* Adds general S3 use-case documentation. #3833 Signed-off-by: David Venable <dlv@amazon.com> * Add edits to S3 logs doc * Apply suggestions from code review Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update s3-logs.md * Update s3-logs.md * Apply suggestions from code review Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: David Venable <dlv@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>

This reverts commit fec44e8.

…search-project#3842) * Adds general S3 use-case documentation. opensearch-project#3833 Signed-off-by: David Venable <dlv@amazon.com> * Add edits to S3 logs doc * Apply suggestions from code review Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update s3-logs.md * Update s3-logs.md * Apply suggestions from code review Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: David Venable <dlv@amazon.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com>

Adds general S3 use-case documentation. opensearch-project#3833

13f83a1

Signed-off-by: David Venable <dlv@amazon.com>

dlvenable requested review from ariamarble, carolxob, cwillum, hdhalter, JeffHuss, kolchfa-aws, Naarcha-AWS, vagimeli, ananzh, seanneumann, AMoo-Miki and natebower as code owners April 24, 2023 20:58

Add edits to S3 logs doc

2155c91

Naarcha-AWS approved these changes Apr 27, 2023

View reviewed changes

hdhalter reviewed Apr 27, 2023

View reviewed changes

Naarcha-AWS and others added 3 commits April 27, 2023 15:50

Apply suggestions from code review

c0c66d4

Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

Update s3-logs.md

623d109

Update s3-logs.md

343a840

cwillum approved these changes Apr 27, 2023

View reviewed changes

Naarcha-AWS reviewed Apr 27, 2023

View reviewed changes

_data-prepper/common-use-cases/s3-logs.md Outdated Show resolved Hide resolved

Apply suggestions from code review

c48800c

Co-authored-by: Chris Moore <107723039+cwillum@users.noreply.github.com> Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

Naarcha-AWS merged commit 196c750 into opensearch-project:main Apr 27, 2023

Naarcha-AWS added backport 2.5 backport 2.6 labels Apr 27, 2023

opensearch-trigger-bot bot mentioned this pull request Apr 27, 2023

[Backport 2.5] Adds general S3 use-case documentation. #3833 #3904

Merged

opensearch-trigger-bot bot mentioned this pull request Apr 27, 2023

[Backport 2.6] Adds general S3 use-case documentation. #3833 #3905

Merged

natebower reviewed May 2, 2023

View reviewed changes

vagimeli added a commit that referenced this pull request May 4, 2023

Revert "Adds general S3 use-case documentation. #3833 (#3842)"

6c91683

This reverts commit fec44e8.

dlvenable deleted the 3833-s3-use-case branch August 21, 2023 16:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds general S3 use-case documentation. #3833 #3842

Adds general S3 use-case documentation. #3833 #3842

dlvenable commented Apr 24, 2023

hdhalter left a comment

hdhalter Apr 27, 2023

Naarcha-AWS Apr 27, 2023

cwillum left a comment

cwillum Apr 27, 2023

natebower left a comment

natebower May 2, 2023

natebower May 2, 2023

natebower May 2, 2023

natebower May 2, 2023

natebower May 2, 2023

natebower May 2, 2023

natebower May 2, 2023

natebower May 2, 2023

natebower May 2, 2023

natebower May 2, 2023


		## Architecture

		Data Prepper supports reading objects from S3 buckets using an [Amazon Simple Queue Service (SQS)](https://aws.amazon.com/sqs/) (Amazon SQS) queue and [Amazon S3 Event Notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html).


		<img src="{{site.url}}{{site.baseurl}}/images/data-prepper/s3-source/s3-pipeline.jpg" alt="S3 source architecture">{: .img-fluid}

		## Prerequisites


		## Architecture

		Data Prepper can read objects from S3 buckets using an [Amazon Simple Queue Service (SQS)](https://aws.amazon.com/sqs/) (Amazon SQS) queue and [Amazon S3 Event Notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html).


		<img src="{{site.url}}{{site.baseurl}}/images/data-prepper/s3-source/s3-architecture.jpg" alt="S3 source architecture">{: .img-fluid}

		The flow of data is as follows.


		The flow of data is as follows.

		1. A system produces logs into the S3 bucket.


		## Pipeline design

		Create a pipeline to read logs from S3, starting with an [`s3`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/s3/) source plugin. Use the following example for guidance.


		Configure the following options according to your use case:

		* `queue_url`: This the SQS queue URL and is always unique to your pipeline.


		The default values for each option work for the majority of use cases. For all available options for the S3 source, see [`s3`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/sources/s3/).

		```yaml


		## Multiple Data Prepper pipelines

		We recommend that you have one SQS queue per Data Prepper pipeline. In addition, you can have multiple nodes in the same cluster reading from the same SQS queue, which doesn't require additional configuration with Data Prepper.

Adds general S3 use-case documentation. #3833 #3842

Adds general S3 use-case documentation. #3833 #3842

Conversation

dlvenable commented Apr 24, 2023

Description

Issues Resolved

Checklist

hdhalter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cwillum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment