Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate legacy Azure storage support #24871

Merged
merged 3 commits into from
Feb 4, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions docs/src/main/sphinx/object-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,6 @@ object storage system as *legacy* support. Additional documentation is available
with the Hive connector and relevant dedicated pages:

- [](/connector/hive)
- [](/object-storage/legacy-azure)
- [](/object-storage/legacy-s3)

(object-storage-other)=
Expand All @@ -100,7 +99,6 @@ storage:
/object-storage/file-system-gcs
/object-storage/file-system-s3
/object-storage/file-system-hdfs
/object-storage/legacy-azure
/object-storage/legacy-s3
/object-storage/file-system-cache
/object-storage/file-system-alluxio
Expand Down
49 changes: 49 additions & 0 deletions docs/src/main/sphinx/object-storage/file-system-azure.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,3 +117,52 @@ storage accounts:
use the **Client ID**, **Secret** and **Tenant ID** values from the
application registration, to configure the catalog using properties from
[](azure-oauth-authentication).


(fs-legacy-azure-migration)=
## Migration from legacy Azure Storage file system

Trino includes legacy Azure Storage support to use with a catalog using the
Delta Lake, Hive, Hudi, or Iceberg connectors. Upgrading existing deployments to
the current native implementation is recommended. Legacy support is deprecated
and will be removed.

To migrate a catalog to use the native file system implementation for Azure,
make the following edits to your catalog configuration:

1. Add the `fs.native-azure.enabled=true` catalog configuration property.
2. Configure the `azure.auth-type` catalog configuration property.
3. Refer to the following table to rename your existing legacy catalog
configuration properties to the corresponding native configuration
properties. Supported configuration values are identical unless otherwise
noted.

:::{list-table}
:widths: 35, 35, 65
:header-rows: 1
* - Legacy property
- Native property
- Notes
* - `hive.azure.abfs-access-key`
- `azure.access-key`
-
* - `hive.azure.abfs.oauth.endpoint`
- `azure.oauth.endpoint`
- Also see `azure.oauth.tenant-id` in [](azure-oauth-authentication).
* - `hive.azure.abfs.oauth.client-id`
- `azure.oauth.client-id`
-
* - `hive.azure.abfs.oauth.secret`
- `azure.oauth.secret`
-
* - `hive.azure.abfs.oauth2.passthrough`
- `azure.use-oauth-passthrough-token`
-
:::

4. Remove the following legacy configuration properties if they exist in your
catalog configuration:

* `hive.azure.abfs-storage-account`
* `hive.azure.wasb-access-key`
* `hive.azure.wasb-storage-account`
31 changes: 31 additions & 0 deletions docs/src/main/sphinx/object-storage/file-system-gcs.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,3 +78,34 @@ Cloud Storage:
- Path to the JSON file on each node that contains your Google Cloud Platform
service account key. Not to be set together with `gcs.json-key`.
:::

(fs-legacy-gcs-migration)=
## Migration from legacy Google Cloud Storage file system

Trino includes legacy Google Cloud Storage support to use with a catalog using
the Delta Lake, Hive, Hudi, or Iceberg connectors. Upgrading existing
deployments to the current native implementation is recommended. Legacy support
is deprecated and will be removed.

To migrate a catalog to use the native file system implementation for Google
Cloud Storage, make the following edits to your catalog configuration:

1. Add the `fs.native-gcs.enabled=true` catalog configuration property.
2. Refer to the following table to rename your existing legacy catalog
configuration properties to the corresponding native configuration
properties. Supported configuration values are identical unless otherwise
noted.

:::{list-table}
:widths: 35, 35, 65
:header-rows: 1
* - Legacy property
- Native property
- Notes
* - `hive.gcs.use-access-token`
- `gcs.use-access-token`
-
* - `hive.gcs.json-key-file-path`
- `gcs.json-key-file-path`
- Also see `gcs.json-key` in preceding sections
:::
127 changes: 127 additions & 0 deletions docs/src/main/sphinx/object-storage/file-system-s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -277,3 +277,130 @@ Example JSON configuration:
are converted to a colon.
Choose a value not used in any of your IAM ARNs.
:::


(fs-legacy-s3-migration)=
## Migration from legacy S3 file system

Trino includes legacy Amazon S3 support to use with a catalog using the Delta
Lake, Hive, Hudi, or Iceberg connectors. Upgrading existing deployments to the
current native implementation is recommended. Legacy support is deprecated and
will be removed.

To migrate a catalog to use the native file system implementation for S3, make
the following edits to your catalog configuration:

1. Add the `fs.native-s3.enabled=true` catalog configuration property.
2. Refer to the following table to rename your existing legacy catalog
configuration properties to the corresponding native configuration
properties. Supported configuration values are identical unless otherwise
noted.

:::{list-table}
:widths: 35, 35, 65
:header-rows: 1
* - Legacy property
- Native property
- Notes
* - `hive.s3.aws-access-key`
- `s3.aws-access-key`
-
* - `hive.s3.aws-secret-key`
- `s3.aws-secret-key`
-
* - `hive.s3.iam-role`
- `s3.iam-role`
- Also see `s3.role-session-name` in preceding sections
for more role configuration options.
* - `hive.s3.external-id`
- `s3.external-id`
-
* - `hive.s3.endpoint`
- `s3.endpoint`
- Add the `https://` prefix to make the value a correct URL.
* - `hive.s3.region`
- `s3.region`
-
* - `hive.s3.sse.enabled`
- None
- `s3.sse.type` set to the default value of `NONE` is equivalent to
`hive.s3.sse.enabled=false`.
* - `hive.s3.sse.type`
- `s3.sse.type`
-
* - `hive.s3.sse.kms-key-id`
- `s3.sse.kms-key-id`
-
* - `hive.s3.upload-acl-type`
- `s3.canned-acl`
- See preceding sections for supported values.
* - `hive.s3.streaming.part-size`
- `s3.streaming.part-size`
-
* - `hive.s3.proxy.host`, `hive.s3.proxy.port`
- `s3.http-proxy`
- Specify the host and port in one URL, for example `localhost:8888`.
* - `hive.s3.proxy.protocol`
- `s3.http-proxy.secure`
- Set to `TRUE` to enable HTTPS.
* - `hive.s3.proxy.non-proxy-hosts`
- `s3.http-proxy.non-proxy-hosts`
-
* - `hive.s3.proxy.username`
- `s3.http-proxy.username`
-
* - `hive.s3.proxy.password`
- `s3.http-proxy.password`
-
* - `hive.s3.proxy.preemptive-basic-auth`
- `s3.http-proxy.preemptive-basic-auth`
-
* - `hive.s3.sts.endpoint`
- `s3.sts.endpoint`
-
* - `hive.s3.sts.region`
- `s3.sts.region`
-
* - `hive.s3.max-error-retries`
- `s3.max-error-retries`
- Also see `s3.retry-mode` in preceding sections for more retry behavior
configuration options.
* - `hive.s3.connect-timeout`
- `s3.connect-timeout`
-
* - `hive.s3.connect-ttl`
- `s3.connection-ttl`
- Also see `s3.connection-max-idle-time` in preceding section for more
connection keep-alive options.
* - `hive.s3.socket-timeout`
- `s3.socket-read-timeout`
- Also see `s3.tcp-keep-alive` in preceding sections for more socket
connection keep-alive options.
* - `hive.s3.max-connections`
- `s3.max-connections`
-
* - `hive.s3.path-style-access`
- `s3.path-style-access`
-
:::

1. Remove the following legacy configuration properties if they exist in your
catalog configuration:

* `hive.s3.storage-class`
* `hive.s3.signer-type`
* `hive.s3.signer-class`
* `hive.s3.staging-directory`
* `hive.s3.pin-client-to-current-region`
* `hive.s3.ssl.enabled`
* `hive.s3.sse.enabled`
* `hive.s3.kms-key-id`
* `hive.s3.encryption-materials-provider`
* `hive.s3.streaming.enabled`
* `hive.s3.max-client-retries`
* `hive.s3.max-backoff-time`
* `hive.s3.max-retry-time`
* `hive.s3.multipart.min-file-size`
* `hive.s3.multipart.min-part-size`
* `hive.s3-file-system-type`
* `hive.s3.user-agent-prefix`
Loading
Loading