Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support time travel using temporal version in Delta Lake #24801

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Support time travel using temporal version in Delta Lake #24801

wants to merge 1 commit into from

Conversation

chenjian2664
Copy link
Contributor

@chenjian2664 chenjian2664 commented Jan 27, 2025 •
edited
Loading

Description

Fix #21024

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

## Delta Lake
* Support time travel using temporal version. ({issue}`21024`)

@cla-bot cla-bot bot added the cla-signed label Jan 27, 2025
@github-actions github-actions bot added docs delta-lake Delta Lake connector labels Jan 27, 2025
@chenjian2664 chenjian2664 changed the title Support time travel Support time travel using temporal version in DeltaLake Jan 27, 2025
@chenjian2664 chenjian2664 force-pushed the delta_time_travel branch 4 times, most recently from 991211c to 6cc05ca Compare January 27, 2025 09:38
@mosabua
Copy link
Member

mosabua commented Jan 27, 2025

You are my hero @chenjian2664 .. awesome job picking this up.

@chenjian2664 chenjian2664 force-pushed the delta_time_travel branch 2 times, most recently from 2c9389f to e9d28b7 Compare January 28, 2025 02:54
wendigo
wendigo reviewed Jan 29, 2025
View reviewed changes
docs/src/main/sphinx/connector/delta-lake.md
FROM example.testdb.customer_orders FOR TIMESTAMP AS OF TIMESTAMP '2022-03-23 09:59:29.803 Europe/Vienna';
```

You can use a date to specify a point a time in the past for using a snapshot of a table in a query.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mosabua please review docs part

findinpath
findinpath reviewed Jan 29, 2025
View reviewed changes
docs/src/main/sphinx/connector/delta-lake.md

```sql
SELECT *
FROM example.testdb.customer_orders FOR TIMESTAMP AS OF DATE '2022-03-23';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you pls add coverage for this use case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It requires the commits in different date, do you have idea how to create it?

ebyhr
ebyhr reviewed Jan 30, 2025
View reviewed changes
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java Outdated Show resolved Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMetadata.java Outdated Show resolved Hide resolved
...sts/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeTimeTravelCompatibility.java Outdated Show resolved Hide resolved
...elta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TemporalTimeTravelUtil.java Outdated Show resolved Hide resolved
...elta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TemporalTimeTravelUtil.java Outdated Show resolved Hide resolved
plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeConnectorTest.java Show resolved Hide resolved
...elta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TemporalTimeTravelUtil.java Show resolved Hide resolved
...sts/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeTimeTravelCompatibility.java Outdated Show resolved Hide resolved
@chenjian2664 chenjian2664 force-pushed the delta_time_travel branch 4 times, most recently from 83c69f0 to 303f498 Compare February 2, 2025 10:25
@chenjian2664 chenjian2664 force-pushed the delta_time_travel branch 2 times, most recently from b9d11e4 to 612a6b6 Compare February 3, 2025 03:38
@chenjian2664 chenjian2664 changed the title Support time travel using temporal version in DeltaLake Support time travel using temporal version in Delta Lake Feb 3, 2025
chenjian2664
chenjian2664 commented Feb 3, 2025
View reviewed changes
...elta-lake/src/main/java/io/trino/plugin/deltalake/transactionlog/TemporalTimeTravelUtil.java
private TemporalTimeTravelUtil() {}

/**
* Finds the latest version where the commit timestamp is less than or equal to the given `epochMillis`:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main changes are here, refactor the helper method.
Would like to discuss about the maxLinearSearchSize, shall we add it as a session property?

@chenjian2664 chenjian2664 force-pushed the delta_time_travel branch 3 times, most recently from 36a55ce to 9c2973f Compare February 3, 2025 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pajaks pajaks Awaiting requested review from pajaks

@wendigo wendigo Awaiting requested review from wendigo

@ebyhr ebyhr Awaiting requested review from ebyhr

@findinpath findinpath Awaiting requested review from findinpath

Assignees
No one assigned
Labels
cla-signed delta-lake Delta Lake connector docs
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add support for reading tables with temporal versions in Delta Lake connector
5 participants
@chenjian2664 @mosabua @wendigo @ebyhr @findinpath