Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix pending state being used incorrectly, ignore ThrottlingException error, accept more options #137

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

itsmepetrov
Copy link

@itsmepetrov itsmepetrov commented Nov 4, 2024

Hi!

This PR fixes two issues:

  1. we noticed that sometimes we could see some log records repeated, like so:
    Processed: 500/407210 (took 0.84s)
    Processed: 26000/407210 (took 0.56s)
    Processed: 53500/407210 (took 0.51s)
    Processed: 53500/407210 (took 0.51s) <- double
    Processed: 98500/407210 (took 0.67s)
    Processed: 123500/407210 (took 0.53s)
    Processed: 123500/407210 (took 0.53s)
    Processed: 123500/407210 (took 0.53s) <- triple
    Processed: 152000/407210 (took 0.47s)
    
    it was caused by the pending key not being set correctly, so there was some sort of race condition;
  2. because of the race condition we could get ThrottlingException, get logs calls have a hard limit, and they don't qualify for a limit increase in AWS, so having two, three... tasks running simultaneously could cause the error quickly, the easiest way just to ignore it as it do not affect anything.

I've also added configuration for durationBetweenPolls and timeoutBeforeFirstLogs

@itsmepetrov
Copy link
Author

@ohookins could you please review it

@ohookins
Copy link
Contributor

@itsmepetrov thanks for the contribution! It's been a bit busy recently but I will review it soon :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants