[FLINK-40131][docs] Document LATERAL SNAPSHOT join#28737
Open
fhueske wants to merge 1 commit into
Open
Conversation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collaborator
davidradl
reviewed
Jul 15, 2026
|
|
||
| {{< label Streaming >}} | ||
|
|
||
| A `LATERAL SNAPSHOT` join is a *stream enrichment* join that augments an append-only table (the *probe* side) with the current state of an updating table (the *build* side). |
Contributor
There was a problem hiding this comment.
have we defined what the probe side is - if so we should include a link to the definition). If not we should define the names for each side before referencing this.
davidradl
reviewed
Jul 15, 2026
|
|
||
| A `LATERAL SNAPSHOT` join is a *stream enrichment* join that augments an append-only table (the *probe* side) with the current state of an updating table (the *build* side). | ||
| Unlike an [event-time temporal join](#event-time-temporal-join), it does not correlate each probe-side row with a specific historical version of the build side. | ||
| Instead, every probe-side row is joined with the build-side state that is current at the time the row is processed, similar to a [processing-time temporal join](#processing-time-temporal-join), but with well-defined behavior at query start-up. |
Contributor
There was a problem hiding this comment.
I am curious what the other joins not having well-defined behavior means. It would be worth mentioning this here to help show the value of this join.
davidradl
reviewed
Jul 15, 2026
| [ load_completed_time => <timestamp_ltz>, ] | ||
| [ load_completed_idle_timeout => <interval>, ] | ||
| [ state_ttl => <interval> ])) AS s | ||
| ON probe_table.key = s.key |
Contributor
There was a problem hiding this comment.
it would be worth expanding the example so we can see where the .keys are defined for each side.
davidradl
reviewed
Jul 15, 2026
|
|
||
| The `LATERAL SNAPSHOT` join is designed for scenarios where the other temporal joins are a poor fit: | ||
|
|
||
| - The build side updates **infrequently**. An event-time temporal join relies on continuous build-side watermarks to emit results, so a build side that rarely advances its watermark stalls the join and lets probe-side state accumulate. A `LATERAL SNAPSHOT` join does not stall when the build side goes idle. |
Contributor
There was a problem hiding this comment.
when you say infrequently. It would be worth defining how the frequency is defined
davidradl
reviewed
Jul 15, 2026
|
|
||
| - The build side updates **infrequently**. An event-time temporal join relies on continuous build-side watermarks to emit results, so a build side that rarely advances its watermark stalls the join and lets probe-side state accumulate. A `LATERAL SNAPSHOT` join does not stall when the build side goes idle. | ||
| - The application has **low-latency** requirements that are incompatible with the watermark-induced delay of an event-time temporal join. | ||
| - The build side has **no primary key**. Event-time and processing-time temporal joins require the build-side primary key to appear in the equi-join condition; a `LATERAL SNAPSHOT` join does not. |
Contributor
There was a problem hiding this comment.
does this mean the probe side has a primary key?
davidradl
reviewed
Jul 15, 2026
| ON orders.currency = rates.currency; | ||
| ``` | ||
|
|
||
| The probe side must be an append-only table, and its time attributes are preserved in the output. The build side may be an updating table and must declare a watermark. The result is append-only. A build-side rowtime attribute that is projected into the output is materialized as a regular `TIMESTAMP` and is no longer a time attribute. Probe-side watermarks are forwarded downstream in join phase; build-side watermarks are consumed internally and are not propagated. |
Contributor
There was a problem hiding this comment.
I suggest a picture (sequence diagram or the like) showing the time line including
- the build side and probe side
- watermarks (the internal build one and the probe one)
- how the select is triggered including the load phase
- maybe an example input and output
davidradl
reviewed
Jul 15, 2026
| ON probe_table.key = s.key | ||
| ``` | ||
|
|
||
| The `SNAPSHOT` function accepts the following arguments: |
Contributor
There was a problem hiding this comment.
shouldn't the SNAPSHOT function be documented in the SQL functions section
davidradl
reviewed
Jul 15, 2026
| | Argument | Type | Required | Description | | ||
| | --- | --- | --- | --- | | ||
| | `input` | TABLE | yes | The build-side table. It may use any [changelog mode]({{< ref "docs/sql/reference/queries/changelog" >}}) (inserts, updates, and deletes) and must declare a [watermark]({{< ref "docs/concepts/sql-table-concepts/time_attributes" >}}#event-time). | | ||
| | `load_completed_condition` | STRING | no | Determines when the initial load phase completes. One of `'compile_time'` (default) or `'user_time'`. With `'compile_time'`, the load phase completes once the build-side watermark reaches the wall-clock time at which the query was compiled. With `'user_time'`, it completes once the build-side watermark reaches the explicit `load_completed_time`. | |
Contributor
There was a problem hiding this comment.
I think a table here would be good describing each option and why you would use them . I think more description as to what exactly compile time and user time means would help.
does the load phase happen more than once?What other phases are there?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Add documentation for FLIP-579: LATERAL SNAPSHOT join
Brief change log
Add docs for FLIP-579: LATERAL SNAPSHOT join
Verifying this change
docs only change
Does this pull request potentially affect one of the following parts:
@Public(Evolving): noDocumentation
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code 2.1.200 (Opus 4.8)