Skip to content

Preserve timezone=False for naive Arrow timestamps#4016

Open
dbascoules wants to merge 1 commit into
dlt-hub:develfrom
dbascoules:fix/4015-preserve-naive-timestamps
Open

Preserve timezone=False for naive Arrow timestamps#4016
dbascoules wants to merge 1 commit into
dlt-hub:develfrom
dbascoules:fix/4015-preserve-naive-timestamps

Conversation

@dbascoules

Copy link
Copy Markdown

Description

This PR fixes a regression where DLT dropped the explicit timezone=False hint inferred from naive Arrow timestamps during capabilities adjustment.

Previously, Arrow input with naive timestamps was normalized as if the timezone hint were absent, which caused downstream loading to convert values to UTC-aware timestamps instead of keeping them naive. This affected pass-through behavior for sources that already expose naive Arrow timestamps, including ConnectorX-based extraction.

The fix preserves timezone=False when it is explicitly present on a column, while continuing to strip other timezone values so the default behavior remains unchanged where no explicit hint is set.

Related Issues

Additional Context

  • Added or updated tests covering naive Arrow timestamps and timezone-aware timestamps.
  • Verified the fix with ConnectorX in both arrow and arrow_stream modes.
  • Confirmed naive timestamps stay naive in Parquet output and timestamptz values remain UTC-aware.

@rudolfix rudolfix added ci from fork Use to trigger CI on a PR (even from a fork) ci full Use to trigger CI on a PR for full load tests labels Jun 2, 2026
@rudolfix rudolfix self-assigned this Jun 2, 2026
@rudolfix

rudolfix commented Jun 3, 2026

Copy link
Copy Markdown
Collaborator

what you say looks right! this behavior is thoroughly tested - I need to see why some of the tests had failed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci from fork Use to trigger CI on a PR (even from a fork) ci full Use to trigger CI on a PR for full load tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Naive Arrow timestamps lose timezone=False and are loaded as UTC-aware instead of naive

2 participants