Skip to content

No retry backoff for sqs in aws_s3 source #22947

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
medzin opened this issue Apr 25, 2025 · 0 comments · May be fixed by #22999
Open

No retry backoff for sqs in aws_s3 source #22947

medzin opened this issue Apr 25, 2025 · 0 comments · May be fixed by #22999
Labels
source: aws_s3 Anything `aws_s3` source related type: bug A code related bug.

Comments

@medzin
Copy link

medzin commented Apr 25, 2025

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

aws_s3 source doesn't have any handling or backoff for SQS errors like IAM or 5xx. It will just retry the whole logic without any backoff in a loop, generating a flood of traffic. As this will be a CPU-intensive task, it will also trigger autoscaling if Vector is deployed with it, scaling up the problem 😄 Please see the code below:

vector/src/sources/aws_s3/sqs.rs

Lines 401 to 424 in 1e7da76

async fn run(mut self) {
let shutdown = self.shutdown.clone().fuse();
pin!(shutdown);
loop {
select! {
_ = &mut shutdown => break,
_ = self.run_once() => {},
}
}
}
async fn run_once(&mut self) {
let messages = self.receive_messages().await;
let messages = messages
.inspect(|messages| {
emit!(SqsMessageReceiveSucceeded {
count: messages.len(),
});
})
.inspect_err(|err| {
emit!(SqsMessageReceiveError { error: err });
})
.unwrap_or_default();

In case of error, it will just return an empty Vec, go quickly through the rest of the code, and retry the whole process in a loop (no backoff).

Configuration

Version

0.46.1

Debug Output

Example Data

No response

Additional Context

No response

References

No response
@medzin medzin added the type: bug A code related bug. label Apr 25, 2025
@pront pront added the source: aws_s3 Anything `aws_s3` source related label Apr 29, 2025
@medzin medzin linked a pull request May 6, 2025 that will close this issue
Open
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
source: aws_s3 Anything `aws_s3` source related type: bug A code related bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(aws_s3 source): add retry delay in sqs::Ingestor medzin/vector
2 participants
@medzin @pront