Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to "close" streams #146

Open
albe opened this issue Feb 10, 2021 · 1 comment
Open

Allow to "close" streams #146

albe opened this issue Feb 10, 2021 · 1 comment
Labels
enhancement P: Index Affects the indexing layer P: Storage Affects the storage layer

Comments

@albe
Copy link
Owner

albe commented Feb 10, 2021

Right now, any index/stream defined will live indefinitely and have every future event/document be checked against it's matcher, even if the stream is known to longer match any future events. The more streams get created, the slower the whole system will become. Some examples are time boxed streams, like by fiscal year/calendar week/etc. or by correlationId (processes).

Therefore it would be useful if streams could be marked "closed"/"finished", or maybe even "deleted" (to prevent them from being recreated).

One approach could be to define a specific combination of index entry values (sequence number, file position, data size, partition id), e.g. (0, 0, 0, 0) for this marker and check for that as last entry in the index when opening. Any such index should no longer be considered for checking new documents.

@albe albe added enhancement P: Storage Affects the storage layer P: Index Affects the indexing layer labels Feb 10, 2021
@albe
Copy link
Owner Author

albe commented May 29, 2021

Another idea is to maybe put the "closed" (and/or other) marker information into the filename, so that a scan of all streams can filter out closed streams early on without having to do additional IO on every file.

stream-X.closed.index, stream-X.deleted.index, etc.

The interesting question is, how those streams should behave on typical operations. Reading from a closed stream should work as normal likely, but it should not be registered for indexing new events and trying to append to it should throw.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement P: Index Affects the indexing layer P: Storage Affects the storage layer
Projects
None yet
Development

No branches or pull requests

1 participant