[Core] [Doc] Add OSS Document for Task Events #56203

MengjinYan · 2025-09-03T19:42:23Z

Why are these changes needed?

This PR adds a page under observability user guide for Ray event export and task events.

It contains basic information about how to enable/configure the figure and the current format of the task events with an example.

As the feature is becoming more mature, the page should be updated with a better way to configure the feature and more explanation about the event format & examples.

Related issue number

N/A

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Mengjin Yan <[email protected]>

nikitavemuri · 2025-09-03T20:04:01Z

doc/source/ray-observability/user-guides/ray-event-export.rst

+    Ray event export feature is still in alpha version. The way to configure event 
+    reporting and the format of the events are subject to change.
+
+Enable event reporting


What do you think about having a section briefly describing the architecture of the aggregator agent and listing some env vars to configure the buffer?

I think it is a good idea to describe the architecture. I'll add one section for that.

At the same time, I'm a little bit hesitate to expose the buffer configuration now.

The reason are that:
(1) env vars might not be a good way for the users to configure Ray and I'm planning to revamp the section after we check in the code to add the command line parameter & ray.init() parameter to configure the event export destination;
(2) we've already exposed a lot of API to the users today and we should only expose new ones if they are absolutely necessary. So without user feedback, I think we should default to expose as less APIs as possible.

jjyao · 2025-09-03T20:04:44Z

doc/source/ray-observability/user-guides/ray-event-export.rst

+
+Enable event reporting
+----------------------
+To enable event reporting, you need to set the ``RAY_enable_core_worker_ray_event_to_aggregator`` environment 


Suggested change

To enable event reporting, you need to set the ``RAY_enable_core_worker_ray_event_to_aggregator`` environment

To enable event reporting, you need to set the ``RAY_enable_core_worker_ray_event_to_aggregator`` environment

Should the env var be something like RAY_enable_ray_event_export given events can be from raylet, gcs etc as well?

From my perspective, this env var will only be temporarily used before we add the public API to ray start scrip & ray.init() and before we migrate the GCS path. With the 2 changes done, we can remove the environment variable and only use the values configured from the public API to determine whether to send the event to GCS. We will revamp the doc then as well.

doc/source/ray-observability/user-guides/ray-event-export.rst

jjyao · 2025-09-03T20:05:38Z

doc/source/ray-observability/user-guides/ray-event-export.rst

+variable to ``1`` when starting each Ray worker node.
+
+To set the target HTTP end point, you need to set the ``RAY_events_export_addr`` 
+environment variable. The value should be a valid HTTP URL with the scheme of ``http``.


can it be https?

Currently it only supports http. I think we can add https support later.

jjyao · 2025-09-03T20:06:21Z

doc/source/ray-observability/user-guides/ray-event-export.rst

+Events are JSON objects in the POST request body.
+
+All events contains the same base fields and different event specific fields. 
+See ``src/ray/protobuf/events_base_event.proto`` for the base fields.


can you create a link to the file?

jjyao · 2025-09-03T20:09:58Z

doc/source/ray-observability/user-guides/ray-event-export.rst

+                "2":"2025-09-03T18:52:14.467402Z",
+                "1":"2025-09-03T18:52:14.467290Z",
+                "5":"2025-09-03T18:52:14.469074Z"


oh, we show 1,2,3 here instead of the state string?

can-anyscale · 2025-09-03T20:08:12Z

doc/source/ray-observability/user-guides/ray-event-export.rst

+
+For each task, Ray exports two types of events: Task Definition Event and Task Execution Event.
+
+* Task Definition Event generated once per task attempt. It contains the metadata of the task. See ``src/ray/protobuf/events_task_definition_event.proto`` and ``src/ray/protobuf/actor_task_definition_event.proto`` for the event format for normal tasks and actor tasks respectively.


nit: src/ray/protobuf/public/events_task_definition_event.proto and others (src/ray/protobuf/events_task_definition_event.proto)

can-anyscale · 2025-09-03T20:09:42Z

doc/source/ray-observability/user-guides/ray-event-export.rst

+        "taskExecutionEvent":{
+            "taskId":"yO9FzNARJXH///////////////8BAAAA",
+            "taskState":{
+                "2":"2025-09-03T18:52:14.467402Z",


qq: what states do these number 2, 1, etc. map to?

They map to the TaskStatus in common.proto

maybe we can point to that file next to one of these numbers in a comment

can-anyscale · 2025-09-03T20:18:19Z

doc/source/ray-observability/user-guides/ray-event-export.rst

+variable to ``1`` when starting each Ray worker node.
+
+To set the target HTTP end point, you need to set the ``RAY_events_export_addr`` 
+environment variable. The value should be a valid HTTP URL with the scheme of ``http``.


i wonder if we can add some simple example on how to setup a local http endpoint for testing, etc.

Good point! I think later we can add an end-to-end example for the events once the configs are finalized. Added an action item in the followup GitHub issue: #54515

Co-authored-by: Jiajun Yao <[email protected]> Signed-off-by: Mengjin Yan <[email protected]>

Signed-off-by: Mengjin Yan <[email protected]>

can-anyscale · 2025-09-03T22:16:25Z

doc/source/ray-observability/user-guides/ray-event-export.rst

+        "taskExecutionEvent":{
+            "taskId":"yO9FzNARJXH///////////////8BAAAA",
+            "taskState":{
+                "2":"2025-09-03T18:52:14.467402Z",


maybe we can point to that file next to one of these numbers in a comment

MengjinYan · 2025-09-04T16:52:16Z

@dayshah Wondering if you can help to review from doc's perspective? Thanks!

Signed-off-by: Mengjin Yan <[email protected]>

doc/source/ray-observability/user-guides/ray-event-export.rst

jjyao · 2025-09-04T17:25:17Z

doc/source/ray-observability/user-guides/ray-event-export.rst

+
+Enable event reporting
+----------------------
+To enable event reporting, you need to set the ``RAY_enable_core_worker_ray_event_to_aggregator`` environment 


jjyao · 2025-09-04T17:30:40Z

doc/source/ray-observability/user-guides/ray-event-export.rst

+Events are JSON objects in the POST request body.
+
+All events contains the same base fields and different event specific fields. 
+See `src/ray/protobuf/events_base_event.proto <https://github.com/ray-project/ray/blob/master/src/ray/protobuf/events_base_event.proto>`_ for the base fields.


@angelinalg @dstrodtman do you know how can I have the link point to the right version. Like if the doc is for 2.49 then I want to point to ray/blob/releases-2.49/xxx and if the doc is for master, then I want to point to ray/blob/master/xxx

this is merged now so you can point this to the public directory as well

Move events_base_event_proto to the public proto directory. Will merge and update doc after #56203. Test: - CI Signed-off-by: Cuong Nguyen <[email protected]>

Co-authored-by: Jiajun Yao <[email protected]> Signed-off-by: Mengjin Yan <[email protected]>

Signed-off-by: Mengjin Yan <[email protected]>

dayshah · 2025-09-04T23:25:14Z

doc/source/ray-observability/user-guides/ray-event-export.rst

+Ray Event Export
+================
+
+Ray supports exporting structured events to a configured HTTP endpoint. Each worker node


specify in the past or from what version? Sounds a little weird because the next sentence that says from 2.49.

Also link the old feature.

Good point. Will do.

doc/source/ray-observability/user-guides/ray-event-export.rst

dayshah · 2025-09-05T00:02:30Z

doc/source/ray-observability/user-guides/ray-event-export.rst

+  generated during the task execution. 
+  See `src/ray/protobuf/public/events_task_execution_event.proto <https://github.com/ray-project/ray/blob/master/src/ray/protobuf/public/events_task_execution_event.proto>`_ for the event format.
+
+An example of the task events is as follows:


Suggested change

An example of the task events is as follows:

An example of a Task Definition Event followed by a Task Execution Event:

? I don't know if this is right

yeah I mainly want to provide a Task Definition Event and a Task Execution Event. Probably I'll update the doc to that.

doc/source/ray-observability/user-guides/ray-event-export.rst

Co-authored-by: Dhyey Shah <[email protected]> Signed-off-by: Mengjin Yan <[email protected]>

Signed-off-by: Mengjin Yan <[email protected]>

doc/source/ray-observability/user-guides/ray-event-export.rst

Move events_base_event_proto to the public proto directory. Will merge and update doc after ray-project#56203. Test: - CI Signed-off-by: Cuong Nguyen <[email protected]> Signed-off-by: sampan <[email protected]>

Co-authored-by: Dhyey Shah <[email protected]> Signed-off-by: Mengjin Yan <[email protected]>

add doc for task event

c82f594

Signed-off-by: Mengjin Yan <[email protected]>

MengjinYan marked this pull request as ready for review September 3, 2025 19:43

MengjinYan requested a review from a team as a code owner September 3, 2025 19:43

can-anyscale self-requested a review September 3, 2025 19:48

jjyao assigned can-anyscale Sep 3, 2025

nikitavemuri reviewed Sep 3, 2025

View reviewed changes

jjyao reviewed Sep 3, 2025

View reviewed changes

can-anyscale reviewed Sep 3, 2025

View reviewed changes

MengjinYan and others added 2 commits September 3, 2025 14:02

Update doc/source/ray-observability/user-guides/ray-event-export.rst

706207a

Co-authored-by: Jiajun Yao <[email protected]> Signed-off-by: Mengjin Yan <[email protected]>

fix comments

ffa2c3d

Signed-off-by: Mengjin Yan <[email protected]>

can-anyscale mentioned this pull request Sep 3, 2025

[core][proto] move events_base_event_proto to public #56212

Merged

can-anyscale approved these changes Sep 3, 2025

View reviewed changes

ray-gardener bot added docs An issue or change related to documentation core Issues that should be addressed in Ray Core labels Sep 4, 2025

add comment for the task state

ac0f66d

Signed-off-by: Mengjin Yan <[email protected]>

jjyao reviewed Sep 4, 2025

View reviewed changes

MengjinYan and others added 2 commits September 4, 2025 15:14

Update doc/source/ray-observability/user-guides/ray-event-export.rst

b5de6b4

Co-authored-by: Jiajun Yao <[email protected]> Signed-off-by: Mengjin Yan <[email protected]>

fix comment

5e7e7a6

Signed-off-by: Mengjin Yan <[email protected]>

dayshah reviewed Sep 5, 2025

View reviewed changes

MengjinYan and others added 8 commits September 5, 2025 14:08

Update doc/source/ray-observability/user-guides/ray-event-export.rst

5fd7507

Co-authored-by: Dhyey Shah <[email protected]> Signed-off-by: Mengjin Yan <[email protected]>

Update doc/source/ray-observability/user-guides/ray-event-export.rst

5e98167

Co-authored-by: Dhyey Shah <[email protected]> Signed-off-by: Mengjin Yan <[email protected]>

Update doc/source/ray-observability/user-guides/ray-event-export.rst

dae3376

Co-authored-by: Dhyey Shah <[email protected]> Signed-off-by: Mengjin Yan <[email protected]>

Update doc/source/ray-observability/user-guides/ray-event-export.rst

55800aa

Co-authored-by: Dhyey Shah <[email protected]> Signed-off-by: Mengjin Yan <[email protected]>

Update doc/source/ray-observability/user-guides/ray-event-export.rst

f3c1fef

Co-authored-by: Dhyey Shah <[email protected]> Signed-off-by: Mengjin Yan <[email protected]>

Update doc/source/ray-observability/user-guides/ray-event-export.rst

e8f3ab7

Co-authored-by: Dhyey Shah <[email protected]> Signed-off-by: Mengjin Yan <[email protected]>

Update doc/source/ray-observability/user-guides/ray-event-export.rst

9578c19

Co-authored-by: Dhyey Shah <[email protected]> Signed-off-by: Mengjin Yan <[email protected]>

fix comment

b5fd763

Signed-off-by: Mengjin Yan <[email protected]>

MengjinYan requested a review from a team as a code owner September 5, 2025 21:38

MengjinYan assigned dayshah Sep 5, 2025

jjyao approved these changes Sep 5, 2025

View reviewed changes

dayshah approved these changes Sep 8, 2025

View reviewed changes

doc/source/ray-observability/user-guides/ray-event-export.rst Outdated Show resolved Hide resolved

doc/source/ray-observability/user-guides/ray-event-export.rst Outdated Show resolved Hide resolved

jjyao added the go add ONLY when ready to merge, run all tests label Sep 8, 2025

MengjinYan and others added 3 commits September 8, 2025 12:44

Update doc/source/ray-observability/user-guides/ray-event-export.rst

3821b66

Co-authored-by: Dhyey Shah <[email protected]> Signed-off-by: Mengjin Yan <[email protected]>

Merge branch 'master' into core-703-9

b48454e

Update doc/source/ray-observability/user-guides/ray-event-export.rst

5313830

Co-authored-by: Dhyey Shah <[email protected]> Signed-off-by: Mengjin Yan <[email protected]>

jjyao enabled auto-merge (squash) September 8, 2025 20:37

jjyao merged commit 010791e into ray-project:master Sep 8, 2025
5 of 6 checks passed

	To enable event reporting, you need to set the ``RAY_enable_core_worker_ray_event_to_aggregator`` environment
	To enable event reporting, you need to set the ``RAY_enable_core_worker_ray_event_to_aggregator`` environment


		For each task, Ray exports two types of events: Task Definition Event and Task Execution Event.

		* Task Definition Event generated once per task attempt. It contains the metadata of the task. See ``src/ray/protobuf/events_task_definition_event.proto`` and ``src/ray/protobuf/actor_task_definition_event.proto`` for the event format for normal tasks and actor tasks respectively.

	An example of the task events is as follows:
	An example of a Task Definition Event followed by a Task Execution Event:

[Core] [Doc] Add OSS Document for Task Events #56203

[Core] [Doc] Add OSS Document for Task Events #56203

Conversation

MengjinYan commented Sep 3, 2025

Why are these changes needed?

Related issue number

Checks

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MengjinYan commented Sep 4, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

can-anyscale Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

can-anyscale Sep 4, 2025 •

edited

Loading