handleEvent to async to deal with increased firehose volume #81
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Moving error handling to src/subscription, this will allow handleEvent to occur in parallel
Verifying getOpsByType within the function now that handleEvent is managed async
|
This PR should really be merged, it has improved throughput on every project I've ported it to (3, at last count). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With the recent increase in firehose activity, the firehose subscription struggles to keep up in real time. This gets much worse if you are doing more than a simple string match on posts.
To boost throughput significantly, this PR no longer awaits handleEvent in the subscription. Possible side effect is that if the generator is intended to handle every post and can't do that in real time, this just increases load until failure; but I think risk is small (this is the approach I take in https://github.com/Bossett/bsky-feeds without ill effect).
Exception handling has been moved to handleEvent - leaving it in the subscription with this approach makes logging harder to parse.