Global Windows don't emit data on time

[philipsdoctor]

I have a GlobalWindow with a custom trigger (I leave windows open for a few seconds after I have enough data to close the window).

When I emit data into my data stream, the flink execution environment appears to halt after the test data is exhausted but before my GlobalWidow is triggered.

I tried changing my trigger to wait zero seconds on window full, but that just appears to have made my test racy where sometimes the global window triggers and calls apply (so the test passes) and sometimes the environment appears to halt first.

Is there a way for me to leave the execution environment running for a few seconds after all of my data is emitted? Or is there a good way for me to test this? So far my only solution has been to stop using flink-spector, swap to using env.fromCollection() in flink, and then pass a custom iterator where the iterator itself hangs before delivering the last value Thread.sleep(10_000) and then the last value is also untested. That gives the window a chance to trigger and I always get the correct results (huzzah) but it's both hacky and stops me from leveraging flink-spector.

Any advice here is greatly appreciated. Thanks.

[lofifnc]

Hi,

You've probably have some form of processing-time logic in your customer trigger. Which means you've encountered this issue #29. There's a pretty lengthy statement in there why I don't wan't to work with sleeps. But there has been some development on Flink which should make it possible to manipulate processing time in the future.

In the meantime I would suggest looking at TestHarnesses provided by Flink.
https://github.com/apache/flink/blob/8dfb9d00653271ea4adbeb752da8f62d7647b6d8/flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/operators/windowing/ContinuousEventTimeTriggerTest.java
This is a test case using a TriggerTestHarness, providing a neat way to thoroughly test your trigger.

You can find test harnesses for almost everything in flink. Flinkspector can then be used to make lightweight integration tests to see if your pipeline works in a distributed fashion.

Hope this is useful.

[philipsdoctor]

@lofifnc understood, thank you!