Skip to content

Commit 320b747

Browse files
authored
chore: comment about idleness on window tumbling how-to (#374)
Co-authored-by: Luis Moreno <[email protected]>
1 parent 46edba8 commit 320b747

File tree

1 file changed

+17
-7
lines changed

1 file changed

+17
-7
lines changed

sdf/how-to/tumbling_window.mdx

Lines changed: 17 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,8 @@ We will define a window with the following syntax
3939
window:
4040
tumbling:
4141
(...)
42+
watermark:
43+
(...)
4244
assign-timestamp:
4345
(...)
4446
partition:
@@ -50,16 +52,25 @@ We will define a window with the following syntax
5052
(...)
5153
```
5254
53-
### Five components of a window
55+
### Six components of a window
5456
5557
#### 1. Add Window duration
5658
```YAML
5759
tumbling:
5860
duration: 15s
5961
```
60-
The following sets the duration of the window to discrete 15 second blocks.
62+
The above configuration sets the duration of the window to discrete 15 second blocks.
6163
62-
#### 2. Assigning a timestamp
64+
#### 2. Assigning an idleness time (optional)
65+
66+
```YAML
67+
watermark:
68+
idleness: 60s
69+
```
70+
71+
With the above configuration, the system will wait for 60 seconds of inactivity (no new data) before closing a window. This allows a 60-second grace period for late-arriving events.
72+
73+
#### 3. Assigning a timestamp
6374
6475
```YAML
6576
assign-timestamp:
@@ -70,7 +81,7 @@ assign-timestamp:
7081
```
7182
The above code assigns a timestamp to each of the entries from the source. In other cases, if a timestamp is encoded in the data, the code could also instead use the encoded timestamp.
7283
73-
#### 3. Assigning a partition key
84+
#### 4. Assigning a partition key
7485
7586
```YAML
7687
assign-key:
@@ -84,7 +95,7 @@ assign-key:
8495
```
8596
We need to make sure we have a mapping function that takes the string and maps it to a key. In this case, we are writing a dataflow that counts the number of occurance the first character appears in the window. Thus, the code just extracts the first digit as a key.
8697
87-
#### 4. Update the state
98+
#### 5. Update the state
8899
89100
```YAML
90101
update-state:
@@ -96,7 +107,7 @@ update-state:
96107
```
97108
To update the state, we have to call the object created by the state listed above. The `update-state` will automatically apply the state instance of the parameter. For our case, we call the increment function. Each time a string appears, the function will update the mapped key's value. In this case, the first digit.
98109

99-
#### 5. Flush the window
110+
#### 6. Flush the window
100111

101112
The final step is to flush the window. All the contents of the window gets popped and outputted into source.
102113
```YAML
@@ -183,7 +194,6 @@ Exit `sdf` terminal and clean-up. The `--force` flag removes the topics:
183194
$ sdf clean --force
184195
```
185196

186-
187197
## Conclusion
188198

189199
We just implemented tumbling windows. Windows can be applied to applications needing aggregated data over a specified time window.

0 commit comments

Comments
 (0)