Skip to content

Fan-out support for the same source #288

Open
@yl-lisen

Description

@yl-lisen

The multiplexer fans out one streaming store reader to different streaming queries.
This has efficiency of disk read / TFF deserialization, memory allocation etc.
But we may introduce new problems like one slow query processing pipeline will slow down other pipelines.
So it maintains some metrics for channel and detach the channel and its the pipeline when slowness is detected.

                                                            -> StreamingStoreSourceChannel -> QueryProcessing Pipeline
StreamingStore Partition -> StreamingStoreSourceMultiplexer -> StreamingStoreSourceChannel -> QueryProcessing Pipeline
                                                            -> StreamingStoreSourceChannel -> QueryProcessing Pipeline

Multiplexer is multiple-thread safe, and a multiplexer is bound to a shard

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions