You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: sdf/how-to/tumbling_window.mdx
+17-7Lines changed: 17 additions & 7 deletions
Original file line number
Diff line number
Diff line change
@@ -39,6 +39,8 @@ We will define a window with the following syntax
39
39
window:
40
40
tumbling:
41
41
(...)
42
+
watermark:
43
+
(...)
42
44
assign-timestamp:
43
45
(...)
44
46
partition:
@@ -50,16 +52,25 @@ We will define a window with the following syntax
50
52
(...)
51
53
```
52
54
53
-
### Five components of a window
55
+
### Six components of a window
54
56
55
57
#### 1. Add Window duration
56
58
```YAML
57
59
tumbling:
58
60
duration: 15s
59
61
```
60
-
The following sets the duration of the window to discrete 15 second blocks.
62
+
The above configuration sets the duration of the window to discrete 15 second blocks.
61
63
62
-
#### 2. Assigning a timestamp
64
+
#### 2. Assigning an idleness time (optional)
65
+
66
+
```YAML
67
+
watermark:
68
+
idleness: 60s
69
+
```
70
+
71
+
With the above configuration, the system will wait for 60 seconds of inactivity (no new data) before closing a window. This allows a 60-second grace period for late-arriving events.
72
+
73
+
#### 3. Assigning a timestamp
63
74
64
75
```YAML
65
76
assign-timestamp:
@@ -70,7 +81,7 @@ assign-timestamp:
70
81
```
71
82
The above code assigns a timestamp to each of the entries from the source. In other cases, if a timestamp is encoded in the data, the code could also instead use the encoded timestamp.
72
83
73
-
#### 3. Assigning a partition key
84
+
#### 4. Assigning a partition key
74
85
75
86
```YAML
76
87
assign-key:
@@ -84,7 +95,7 @@ assign-key:
84
95
```
85
96
We need to make sure we have a mapping function that takes the string and maps it to a key. In this case, we are writing a dataflow that counts the number of occurance the first character appears in the window. Thus, the code just extracts the first digit as a key.
86
97
87
-
#### 4. Update the state
98
+
#### 5. Update the state
88
99
89
100
```YAML
90
101
update-state:
@@ -96,7 +107,7 @@ update-state:
96
107
```
97
108
To update the state, we have to call the object created by the state listed above. The `update-state` will automatically apply the state instance of the parameter. For our case, we call the increment function. Each time a string appears, the function will update the mapped key's value. In this case, the first digit.
98
109
99
-
#### 5. Flush the window
110
+
#### 6. Flush the window
100
111
101
112
The final step is to flush the window. All the contents of the window gets popped and outputted into source.
102
113
```YAML
@@ -183,7 +194,6 @@ Exit `sdf` terminal and clean-up. The `--force` flag removes the topics:
183
194
$ sdf clean --force
184
195
```
185
196
186
-
187
197
## Conclusion
188
198
189
199
We just implemented tumbling windows. Windows can be applied to applications needing aggregated data over a specified time window.
0 commit comments