Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Different message offset (kafka) per schema(topic). #233

Open
5 tasks
YooSunYoung opened this issue Sep 26, 2024 · 2 comments
Open
5 tasks

Different message offset (kafka) per schema(topic). #233

YooSunYoung opened this issue Sep 26, 2024 · 2 comments

Comments

@YooSunYoung
Copy link
Member

Beamlime should start consuming from different offset for different schema (type of data)

  • PL72: Latest run start message, and match with the run_number or proposal_id.
  • EV44: From the start_time of the run_start_message.
  • F144: From the latest message regardless of the start time.
  • TDCT: From the start_time of the run_start_message.
  • DA00: From the start_time of the run_start_message.

It is because logs like detector position might be set in the past and never moved.

@SimonHeybrock
Copy link
Member

Is this actually true? I had assumed that we would get a message in all cases when we start streaming, just like the filewriter will always get values? Only the timestamp may be in the past. Should check with ECDC before doing any work here!

@YooSunYoung
Copy link
Member Author

Is this actually true? I had assumed that we would get a message in all cases when we start streaming, just like the filewriter will always get values? Only the timestamp may be in the past. Should check with ECDC before doing any work here!

It doesn't change where we set the offset of the messages though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants