Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Database Plugin #104

Open
11 of 16 tasks
dominiklohmann opened this issue Sep 15, 2023 · 0 comments
Open
11 of 16 tasks

Database Plugin #104

dominiklohmann opened this issue Sep 15, 2023 · 0 comments
Labels
feature New functionality operator Source, transformation, and sink

Comments

@dominiklohmann
Copy link
Member

dominiklohmann commented Sep 15, 2023 •
edited
Loading

We want to bring back VAST as a plugin for Tenzir—the database plugin.

Notably, this rewrite of the Tenzir's underlying storage engine aims to achieve the following:

  1. import and export should be generalized as publish and subscribe, respectively, that support sharing data between pipelines under a topic. This allows for decoupling of backpressure by splitting one pipeline into multiple, which in turn enables Load on Scroll.
  2. Data transformations should be atomic replacements on the partition level, superseding the partition transformer underlying the rebuild, compaction, aging, and disk-monitor mechanisms on top of a single transform <expr> <pipeline> operator.
  3. Schema explosion should be manageable by merging schemas with matching names and mergeable layouts.

Stories
Beta Give feedback
@dominiklohmann dominiklohmann mentioned this issue Sep 15, 2023
Open
@mavam mavam added operator Source, transformation, and sink feature New functionality labels Sep 25, 2023
@dominiklohmann dominiklohmann changed the title Publish and Subscribe Operators Database as a Plugin Jan 28, 2024
@dominiklohmann dominiklohmann changed the title Database as a Plugin Database Plugin Jan 28, 2024
@dominiklohmann dominiklohmann mentioned this issue Jan 29, 2024
Closed
@mavam mavam mentioned this issue Apr 8, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature New functionality operator Source, transformation, and sink
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mavam @dominiklohmann