Best architecture implementation for Stream and IncrementalStream for same Soure #38114
icychocolate98
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Good afternoon people !
I'm creating a new connector, is a basic Source that will support full-refresh and incremental one. My doubt is mainly about the architecture or how to invoke both Streams.
I'm creating the module
streams.py
where I define 2 Streams. My first Stream was created to process full-refresh mainly, asFirstStream(Stream)
as a common Stream and another one that inherits from FirstStream so it's something like this:IncrementalStream(FirstStream, IncrementalMixin)
.At the final my first Stream does nothing, so all logic to distinguish between full-refresh or incremental is implemented on IncrementalStream.
Then, in my source module, I only instantiate Incremental one, and inside this, I create some logic to perform incremental read or not in read_records
Slices have a field 'event_date' that should be my incremental field.
I do not know if I should return both streams in source.py module, only incremental ( as it inherits from common one), or what's best for a source of this type.
All help is welcome, thank you !
Beta Was this translation helpful? Give feedback.
All reactions