feat(python): add capability to read unity catalog (uc://) uris #3113
Conversation
github-actions
bot
added
binding/python
|
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
omkar-foss
force-pushed
the
feat-uc-python
branch
2 times, most recently
from
8fcd7f4 to
0824abe
Compare
omkar-foss
changed the title
| /// Allow http url (e.g. http://localhost:8080/api/2.1/...) | ||
| /// Supported keys: | ||
| /// - `unity_allow_http_url` | ||
| AllowHttpUrl, |
There was a problem hiding this comment.
This allows users to work with a local (non-https) Unity Catalog REST API with delta-rs.
omkar-foss
force-pushed
the
feat-uc-python
branch
from
0da4600 to
faf9ba9
Compare
omkar-foss
force-pushed
the
feat-uc-python
branch
from
faf9ba9 to
1a810d9
Compare
|
@ion-elgreco I've updated this PR to include the temp credentials functionality from PR #3078. Cheers. |
omkar-foss
requested review from
wjones127,
fvaleye,
roeap,
rtyler and
hntd187
as code owners
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3113 +/- ##
==========================================
- Coverage 72.18% 72.08% -0.10%
==========================================
Files 138 138
Lines 45292 45380 +88
Branches 45292 45380 +88
==========================================
+ Hits 32694 32714 +20
- Misses 10535 10600 +65
- Partials 2063 2066 +3 ☔ View full report in Codecov by Sentry. |
This adds capability to read directly from uc:// uris using the local catalog-unity crate. This also exposes the UC temporary credentials in storage_options of the `DeltaTable` instance so polars or similar readers can use it. Signed-off-by: Omkar P <[email protected]>
omkar-foss
force-pushed
the
feat-uc-python
branch
from
1a810d9 to
2fdb6e2
Compare
|
@omkar-foss looks good from my side! @hntd187 can you please also take a look. Specifically on whether the credentials need to be refreshed as part of the objectstore? Does UC also support writing temp credentials? |
|
I'll need to add a few python tests for this flow, will add tomorrow and then if all good, I suppose we can merge it. Also please note, the current UC REST integration that we have is more fine-tuned for Databricks Unity Catalog. The Unity Catalog OSS has slightly different response payloads for some APIs. e.g. Generate Temp Credentials API returns I can look into supporting OSS UC smoothly in a separate PR after this one. Cheers. |
Description
This adds capability to read directly from uc:// uris using the local catalog-unity crate. This also exposes the UC temporary credentials in storage_options of the
DeltaTableinstance so polars or similar readers can use it.Related Issue(s)
Documentation
N/A