chore: changes for sdf-beta9 (#382)

Co-authored-by: Luis Moreno <[email protected]>

Author: morenol

sdf/SDF_VERSION

@@ -1 +1 @@
-sdf-beta8
+sdf-beta9

sdf/_embeds/install-sdf.bash

@@ -1 +1 @@
-fvm install sdf-beta8
+fvm install sdf-beta9

sdf/cli/worker/connect.mdx

@@ -0,0 +1,10 @@
+---
+title: connect
+sidebar_position: 20
+---
+
+# `sdf worker connect`
+
+The `sdf worker connect`  enters an [interactive shell] that allows you to interact with the worker and its dataflows.
+
+[interactive shell]: ../deploy.mdx#deploy-interactive-shell

sdf/cli/worker/create.mdx

@@ -1,5 +1,6 @@
 ---
 title: create
+sidebar_position: 10
 ---
 
 # `sdf worker create`

sdf/cli/worker/index.mdx

@@ -11,6 +11,7 @@ The `sdf worker` contains all the subcommands needed for [Deployment].
 ### `sdf worker` commands
 
 * [sdf worker create] - to create host worker
+* [sdf worker connect] - to connect to current worker
 * [sdf worker list] - to list cluster workers
 * [sdf worker register] - to register remote worker
 * [sdf worker unregister] - to unregister remote worker
@@ -19,6 +20,7 @@ The `sdf worker` contains all the subcommands needed for [Deployment].
 
 [Deployment]: deployment.mdx
 [sdf worker create]: create.mdx
+[sdf worker connect]: connect
 [sdf worker list]: list.mdx
 [sdf worker register]: register.mdx
 [sdf worker unregister]: unregister.mdx

sdf/concepts/smartmodule-sdf.mdx

@@ -0,0 +1,85 @@
+---
+title: Scheduling events
+description: Stateful Dataflow schedule configuration.
+sidebar_position: 1100
+---
+
+:::info
+**schedule** requires using `apiVersion: 0.6.0` or higher in your dataflow configuration.
+:::
+
+The `schedule` configuration defines recurring events that can be used as input sources for your services, enabling time-based automation.
+
+
+### Structure:
+
+```yaml
+apiVersion: 0.6.0 # Requires 0.6.0 or higher
+...
+schedule:
+  <event-name>:
+    cron: "<cron-expression>"
+```
+
+### Using Scheduled Events as Service Inputs:
+
+```yaml
+services:
+  <service-name>:
+    sources:
+      - type: schedule
+        id: <event-name>
+```
+
+
+## Schedule configuration fields
+
+- `<event-name>`: (String, required) A unique identifier for the scheduled event. This name is used to reference the event in service configurations.
+- `cron`: (String, required) A cron expression that defines the schedule. The expression follows the standard Unix cron format:
+```
+* * * * *
+- - - - -
+| | | | |
+| | | | +----- day of week (0 - 7) (Sunday=0 or 7)
+| | | +------- month (1 - 12)
+| | +--------- day of month (1 - 31)
+| +----------- hour (0 - 23)
++------------- minute (0 - 59)
+```
+For more details on cron expressions, refer to online cron expression generators and documentation.
+
+## Example:
+
+This dataflow will produce a record to the `output` topic once per day at 8:00 AM
+
+```yaml
+apiVersion: 0.6.0
+meta:
+  name: schedule
+  version: 0.1.0
+  namespace: example
+# default config
+config:
+  converter: json
+  consumer:
+    default_starting_offset:
+      value: 0
+      position: End
+topics:
+  output:
+    name: time-events
+    schema:
+      value:
+        type: i64
+schedule:
+  daily-report:
+    cron: "0 8 * * *" # Trigger at 8:00 AM daily
+services:
+  produce-daily:
+    sources:
+      - type: schedule
+        id: daily-report
+    sinks:
+      - type: topic
+        id: output
+```

sdf/how-to/scheduling.mdx

@@ -1,6 +1,6 @@
 ---
 title: "Tumbling Window"
-description: "Tumbling Window"
+description: "Learn how to use tumbling windows in SDF to aggregate data over fixed time intervals."
 sidebar_position: 1100
 ---
 
@@ -61,14 +61,16 @@ tumbling:
 ```
 The above configuration sets the duration of the window to discrete 15 second blocks.
 
-#### 2. Assigning an idleness time (optional)
+#### 2. Assigning an `idleness` time and `grace-period` (optional)
 
 ```YAML
 watermark: 
   idleness: 60s
+  grace-period: 3s
 ```
 
-With the above configuration, the system will wait for 60 seconds of inactivity (no new data) before closing a window. This allows a 60-second grace period for late-arriving events.
+- `watermark.idleness: 60s`: This specifies that if there's no new data for 60 seconds, the window will be closed. This handles late-arriving events
+- `watermark.grace-period: 3s`: This allows a 3 second grace period for out-of-order events. If an event arrives within this grace period after the window's end, it will still be included in the window's calculation. This prevents the dropping of slightly late events due to network delays or other factors.
 
 #### 3. Assigning a timestamp
 

sdf/how-to/tumbling_window.mdx

@@ -7,7 +7,7 @@ sidebar_position: 20
 import CodeBlock from '@theme/CodeBlock';
 import InstallFvm from '!!raw-loader!./_embeds/install-sdf.bash';
 
-# Beta 8
+# Beta 9
 
 ## Upgrading 
 
@@ -26,19 +26,18 @@ For upgrading Infinyon Cloud (remote) workers, please contact [InfinyOn support]
 
 ### CLI changes
 
-- Show df command now also returns the last restart time.
-- Do not ignore deployment when the dataflow is the same. 
+- SDF interactive shell not longer exits on ctrl-c
+- Introduced the `--clear-all` and `--clear-state`  options to the dataflow restart command, allowing users to clear the dataflow state before restarting.
 
-### Improvements
+### New features
 
-- Updated fluvio version which includes better reconnection support for consumers.
+- Added capability to schedule events using CRON format.
+- Added grace period configuration to window watermark, in order to support allowing out of order events.
 
-### Features
+### Logging and Debugging Enhancements
 
-- Added support for idleness on window based services.
+- Enhanced the sdf log -v command to include record offset and partition information.
 
 ## InfinyOn Support
 
 For any questions or issues, please contact InfinyOn support at [email protected]  or https://discordapp.com/invite/bBG2dTz
-
-[idleness]: concepts/window-processing.mdx#idleness