DRF Expansion #24

rneswold · 2025-07-18T14:55:49Z

This pull request is offered up for discussion -- it's not ready to be merged, if at all.

Add arm, sample, and stop events to the specification.

Cut-n-paste bug. :/

beauremus · 2025-07-18T15:46:38Z

proto/controls/common/v1/drf.proto

    DataChannel data_channel = 1;
-    optional common.event.Event event = 2;
-    optional common.sources.Source source = 3;
+    common.event.ArmEvent event = 2;


I think this is the "right" way to do this, but it's a bit smelly that these two blocks need to stay in "sync".

We could combine the similar fields in another message and use that message in both. It adds another "layer" of fields, though.

Are we proposing to combine them in another message?

I think that's best. It adds another layer, but it'll be less error-prone.

beauremus · 2025-07-18T16:09:13Z

proto/controls/common/v1/event.proto

-        google.protobuf.Duration period = 2;
-        bool immediate = 3;
+    int32 event_number = 1;
+    EventType event_type = 2;


We should deprecate this. Can we mark it as deprecated?

I think the default is good, which is good news!

If the default value is not specified for an optional element, a type-specific default value is used instead: for strings, the default value is the empty string. For bools, the default value is false. For numeric types, the default value is zero. For enums, the default value is the first value listed in the enum's type definition. This means care must be taken when adding a value to the beginning of an enum value list.

What do you feel is deprecated?

The event_type. We should hardware-trigger everything.

proto/controls/common/v1/event.proto

beauremus · 2025-07-18T18:23:24Z

proto/controls/common/v1/event.proto

+message SampleEvent {
+    message Periodic {
+        google.protobuf.Duration period = 1;
+        bool immediate = 2;


I think immediate is a confusing feature. We should do some documentation to explain to users what to expect.

Maybe we could come up with a better name.

no_delay?

Yeah. I like no_delay.

Or add_immediate?

I don't think add_immediate is clear.

proto/controls/common/v1/event.proto

beauremus · 2025-07-18T18:27:08Z

@KitFieldhouse, I'm interested in whether these structures match your arm and trigger functionality.

beauremus · 2025-07-18T19:49:06Z

I got the help of an LLM to write some pseudo-code user examples to see how this "feels" from a user perspective.

# Example 1: Requesting data periodically
# This example shows how to request a specific data channel ('device.name')
# with its primary field, sampled every 5 seconds, starting immediately.
request = DRF(
    data_channel=DataChannel(
        name="device.name",
        range=Range(full_range=google.protobuf.Empty()), # Is this required? Can it be left undefined?
        field=Field(reading=Field.Flavor.PRIMARY) # I think because `RAW` is first in the ENUM, it's the default
    ),
    sample=SampleEvent(
        periodic=SampleEvent.Periodic(
            period=google.protobuf.Duration(seconds=5),
            immediate=True
        )
    )
)

# Example 2: Requesting data on a TCLK accelerator event
# This example demonstrates requesting the 'device.name' raw data
# on TCLK event $1D.
request = DRF(
    data_channel=DataChannel(
        name="device.name",
        range=Range(full_range=google.protobuf.Empty()),
        field=Field(reading=Field.Flavor.RAW) # Is this required? Can it be left undefined?
    ),
    sample=SampleEvent(
        clock=common.event.Clock(
            event_number=123, # It's not a huge burden to have users convert from hex to decimal, but it's not great
            # Python would look something like this, I think
            event_number=int("1D", 16),
            event_type=common.event.Clock.EventType.HARDWARE # Should we consider deprecation?
        )
    )
)

# Example 3: Requesting data on a state event (value change)
# This example shows how to request data when 'state.monitor' changes its value.
request = DRF(
    data_channel=DataChannel(
        name="device.name",
        range=Range(full_range=google.protobuf.Empty()),
        field=Field(reading=Field.Flavor.COMMON)
    ),
    sample=SampleEvent(
        state=common.event.State(
            state_name="state.monitor",
            any_value=google.protobuf.Empty() # Trigger on any value change
        )
    )
)

# Example 4: Requesting data with a delay after a clock event
# This example requests 'device.name' data on TCLK event $FA,
# but with a 100 millisecond delay after the event fires.
request = DRF(
    data_channel=DataChannel(
        name="device.name",
        range=Range(full_range=google.protobuf.Empty()),
        # Note that this test shows an error state because `setting` doesn't have a `NOMINAL` field
        # It would be a lot of duplication but should we consider an ENUM per field?
        field=Field(setting=Field.Flavor.NOMINAL)
    ),
    sample=SampleEvent(
        clock=common.event.Clock(
            event_number=int("FA", 16),
            event_type=common.event.Clock.EventType.HARDWARE,
            # second and nanos are the available arguments
            delay=google.protobuf.Duration(nanos=100_000_000) # 100 milliseconds
        )
    )
)

# Example 5: Requesting a specific array range periodically
# This example shows requesting elements from index 10 to 20 of an array device,
# sampled every 2 seconds.
request = DRF(
    data_channel=DataChannel(
        name="array.device",
        range=Range(
            array_range=ArrayRange(
                start_index=10,
                end_index=20
            )
        ),
        field=Field(reading=Field.Flavor.RAW)
    ),
    sample=SampleEvent(
        periodic=SampleEvent.Periodic(
            period=google.protobuf.Duration(seconds=2),
            immediate=True
        )
    )
)

# Example 6: Requesting control field to send a command (one-time)
# This example illustrates sending a control command to 'device.command'
# immediately, without continuous sampling.

# This API is reading-oriented and not setting-oriented
# This request doesn't make sense, but is valid
request = DRF(
    data_channel=DataChannel(
        name="device.command",
        range=Range(full_range=google.protobuf.Empty()),
        field=Field(control=google.protobuf.Empty())
    ),
    sample=SampleEvent(
        immediate=google.protobuf.Empty()
    )
)

# Does it make sense if we remove the sample?
request = DRF(
    data_channel=DataChannel(
        name="device.command",
        range=Range(full_range=google.protobuf.Empty()),
        field=Field(control=google.protobuf.Empty())
    )
)

# Modifying the DAQ service to accept DRF instead of a string
DAQ.Set([
    device=request, # See above
    value=Value(boolean=True) # We do not currently support `Control` actions in `common.device.Value`
])

# Example 7: Arming a data acquisition to stop after a certain count
# This example shows how to arm a data acquisition for 'device.status'
# to start on a hardware clock event $0C, and stop after 5 samples.
request = DRF(
    data_channel=DataChannel(
        name="device.status",
        range=Range(full_range=google.protobuf.Empty()),
        field=Field(status=Field.Flavor.ON)
    ),
    arm=ArmEvent(
        clock=common.event.Clock(
            event_number=int("0C", 16),
            event_type=common.event.Clock.EventType.HARDWARE
        )
    ),
    stop=StopEvent(
        count=5
    )
)

# Example 8: Requesting multiple data channels periodically
# This example shows how to request primary field data from 'device.one'
# and raw data from 'device.two' simultaneously, both sampled every 10 seconds.
request = DRFs(
    data_channel=[
        DataChannel(
            name="device.one",
            range=Range(full_range=google.protobuf.Empty()),
            field=Field(reading=Field.Flavor.PRIMARY)
        ),
        DataChannel(
            name="device.two",
            range=Range(full_range=google.protobuf.Empty()),
            field=Field(reading=Field.Flavor.RAW)
        )
    ],
    sample=SampleEvent(
        periodic=SampleEvent.Periodic(
            period=google.protobuf.Duration(seconds=10),
            immediate=True
        )
    )
)

KitFieldhouse · 2025-07-18T19:41:14Z

proto/controls/common/v1/drf.proto

+    common.event.SampleEvent sample = 3;
+    common.event.StopEvent stop = 4;
+    optional common.sources.Source source = 5;
 }


What I have implemented has slightly different syntax here. Instead of the fields: arm, sample, and stop, I essentially use four fields; sample, arm, trigger, and stop (though I call them other names and I didn't implement them as a proto).

The semantics I had in mind is that sample is what is continuously generating the data stream and the other fields act as a "valve" that controls if that data stream is emitted to the client.

I think this allows for slightly more flexible acquisition triggering. For example, we could be sampling 1440Hz data and trigger on the first 1D after a 12 up until the extraction event by setting the fields like so:

sample: Periodic 1440Hz
arm: 0x12
trigger: 0x1D
stop: 0X1F

Or sampling 1440Hz ftp only on 1Ds that are accompanied by the HEP enable event 52 like so:

sample: 1440Hz
arm: 0x1D
trigger: 0x52
stop: 0x1F

Essentially, compared to this change, separating sample and trigger allows for an additional clock condition (or other condition, like a state change) that needs to be cleared before sending data. However, one could imagine allowing for arbitrarily long sequences of clock events (or other conditions) that need to happen in a sequence before allowing data to be emitted, so having this additional condition might not really buy us much.

Does this request "feel" good to you?

sample: Periodic 1440Hz
arm: 0x12
trigger: 0x1D
stop: 0X1F

request = DRF( data_channel=DataChannel( name="device.status", range=Range(full_range=google.protobuf.Empty()), field=Field(status=Field.Flavor.ON) ), sample=SampleEvent( periodic=SampleEvent.Periodic( # This stinks and could be an issue with how this is "rounded" period=google.protobuf.Duration(nanos=math.trunc(1/1440*1_000_000_000)), immediate=True ) ), arm=ArmEvent( clock=common.event.Clock( event_number=int("12", 16), event_type=common.event.Clock.EventType.HARDWARE ) ), trigger=TriggerEvent( clock=common.event.Clock( event_number=int("1D", 16), event_type=common.event.Clock.EventType.HARDWARE ) ), stop=StopEvent( clock=common.event.Clock( event_number=int("1F", 16), event_type=common.event.Clock.EventType.HARDWARE ) ) )

sample: 1440Hz
arm: 0x1D
trigger: 0x52
stop: 0x1F

request = DRF( data_channel=DataChannel( name="device.status", range=Range(full_range=google.protobuf.Empty()), field=Field(status=Field.Flavor.ON) ), sample=SampleEvent( periodic=SampleEvent.Periodic( # This stinks and could be an issue with how this is "rounded" period=google.protobuf.Duration(nanos=math.trunc(1/1440*1_000_000_000)), immediate=True ) ), arm=ArmEvent( clock=common.event.Clock( event_number=int("1D", 16), event_type=common.event.Clock.EventType.HARDWARE ) ), trigger=TriggerEvent( clock=common.event.Clock( event_number=int("52", 16), event_type=common.event.Clock.EventType.HARDWARE ) ), stop=StopEvent( clock=common.event.Clock( event_number=int("1F", 16), event_type=common.event.Clock.EventType.HARDWARE ) ) )

I think this would require something like:

message TriggerEvent { oneof event { google.protobuf.Empty immediate = 1; # This is the default, so a no-op Clock clock = 2; State state = 3; google.protobuf.Duration delay = 4; } }

I like it!

We may be able to complicate/enhance it further in order to support more complex trigger conditions. It doesn't guarantee all front-ends will be ale to support it. But DPM should be able to simulate to conditions that the FE can't. For instance, it could ask for 1440Hz data and do all the filtering itself based on monitoring the clock system.

My understanding from @KitFieldhouse is that is what KitDPM does.
I think we should pull that into a service, but that's nitpicking.

Speaking to "complex trigger conditions" I was considering posting this and decided not to. Does this align with what you were thinking?

// Represents a single condition within a compound event (either a Clock or a State). message EventCondition { oneof condition_type { Clock clock = 1; State state = 2; } } // Defines a compound event by combining multiple EventConditions with logical operators. message CompoundEvent { enum Operator { AND = 0; // All conditions must be met (e.g., all clocks fire near-simultaneously) OR = 1; // Any one condition must be met SEQUENCE = 2; // Conditions must be met in the specified order } Operator operator = 1; repeated EventCondition conditions = 2; // List of individual event conditions }

I guess I need to know how a "user trigger" works. Is it a hardware signal that a user can activate? Or a gRPC request? Setting a state device would be similar complexity as the latter.

RE: your LLM examples. Those are fairly messy for a developer to use. But I would imagine that our libraries would provide a cleaner interface and would generate those messages.

I guess I need to know how a "user trigger" works. Is it a hardware signal that a user can activate? Or a gRPC request? Setting a state device would be similar complexity as the latter.

I had assumed the latter. @KitFieldhouse did you mean something different?

For this architecture yeah it would be a gRPC request.

The idea was that, when submitting a request with a user trigger, you would get some unique ID for that trigger that you could then use with another API endpoint to satisfy that trigger.

The implementation I had originally had in mind might not fit very well with the current and planned daq architecture, but the idea was for the API route that fires the user trigger to be a simple HTTP POST with an empty body and a query parameter equal to the "User Trigger ID". By sending that request the trigger is then satisfied. The idea is that this trigger then essentially has a unique URL that can then be used for firing that trigger, either through a browser or with a command line utility like curl.

For the gRPC architecture, maybe this would be a new API endpoint that takes a "User Trigger ID" and then fires that trigger on the DPMs or returns some sort of error if no user trigger matches that ID?

I feel like this is slightly more formal than state events but does, essentially, the same thing. Except the DPMs already listen to state multicasts.

This reorganization is a bit simpler than the previous and still allows post triggers. Rather than having an arm event indicating the arming goes at the end of collection, we just use the stop event. But that means we can't use the `count` field, so that field was moved to the `SampleEvent` message as a separate field. If 0, it means "infinite" samples, otherwise it indicates an upper limit of how many points will be sent. Any acquisition service should treat a `StopEvent` that isn't `never` and a `SampleEvent` with a `count` not zero as a post-trigger request; that is, it keeps a circular buffer of the most recent `count` readings and sends them when the `StopEvent` occurs.

rneswold · 2025-07-21T19:39:27Z

With 5cf99ab, if you want to see the data (1,000 points) before an event ($21) happened, @1440Hz, you'd do:

device : R:IBEAM
arm: immediate
sample: periodic @ 1440Hz, count: 1,000
stop: clock $21

This is a "post-trigger" request.

rneswold · 2025-07-21T20:04:54Z

Once the stop event occurs, presumably the arming event becomes "active" again and can start another acquisition cycle when the event occurs. Should we add a count field to ArmEvent to limit how many times we want to cycle?

If the arm event is an absolute time, a service can ignore the count field since it'll never occur again, rather than keep the stream open forever.

rneswold added 2 commits July 18, 2025 09:52

📝 document some messages

407ce27

🚧 enhance DRF messages

1af5c05

Add arm, sample, and stop events to the specification.

rneswold requested review from beauremus, charlieking65 and cnlklink July 18, 2025 14:55

rneswold added 2 commits July 18, 2025 11:16

🐛 field names need to be different

96ba14d

Cut-n-paste bug. :/

♻️ add time to arm and stop events

a9bac15

beauremus reviewed Jul 18, 2025

View reviewed changes

beauremus requested a review from KitFieldhouse July 18, 2025 18:26

KitFieldhouse reviewed Jul 18, 2025

View reviewed changes

rneswold added 2 commits July 21, 2025 14:09

🐛 move never to ArmEvent

d73ab7b

rneswold requested review from KitFieldhouse and beauremus July 23, 2025 12:39

DRF Expansion #24

Are you sure you want to change the base?

DRF Expansion #24

Uh oh!

Conversation

rneswold commented Jul 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

beauremus commented Jul 18, 2025

Uh oh!

beauremus commented Jul 18, 2025

Uh oh!

KitFieldhouse Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KitFieldhouse Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rneswold commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rneswold commented Jul 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

KitFieldhouse Jul 18, 2025 •

edited

Loading

KitFieldhouse Jul 22, 2025 •

edited

Loading

rneswold commented Jul 21, 2025 •

edited

Loading