feat: Add support for Azure Blob Storage and ADLS Gen2 in Hive connector #25107

mehradpk · 2025-05-14T08:57:45Z

Description

Introduce support for Azure storage backends including Azure Blob Storage (using the wasb:// scheme) and Azure Data Lake Storage Gen2 (using the abfs:// scheme) in the Hive connector.

Key changes:

Added HiveAzureConfigurationInitializer to inject relevant Azure configurations into Hadoop Configuration
Introduced HiveAzureConfig to allow catalog-level configuration of Azure properties
Updated HdfsConfigurationInitializer and HiveConnectorFactory to delegate Azure-specific config setup
Registered configuration initializer in Hive module

Supports shared key and OAuth2-based authentication.

##Motivation and Context
Several enterprise data lake workloads are hosted on Azure storage platforms. This change allows Presto to directly query data from Azure Blob Storage and ADLS Gen2, bringing Azure compatibility in line with other cloud storage systems like Amazon S3 and Google Cloud Storage.

Impact

Adds support for Azure cloud storage within the Hive connector
Introduces new configuration properties under HiveAzureConfig

No breaking changes to existing Hive catalogs or connectors

Test Plan

Test done via Hive Connector.

Contributor checklist

Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.

== RELEASE NOTES ==

Hive Connector Changes
* Add support for Azure Blob Storage and Azure Data Lake Storage Gen2 in the Hive connector
* Add support for wasb[s]:// and abfs[s]:// URI schemes
* Add support for shared key and OAuth2 authentication for Azure storage

prestodb-ci · 2025-05-14T10:53:57Z

@imjalpreet imported this issue as lakehouse/presto #25107

imjalpreet

@mehradpk Thank you for the PR, I have taken an initial pass on the PR.

Have you carried out some actual tests with Azure?

imjalpreet · 2025-05-22T13:18:10Z

presto-hive/src/main/java/com/facebook/presto/hive/azure/HiveAzureConfigurationInitializer.java

+    {
+        return optional.filter(value -> !value.isEmpty());
+    }
+    private static Proxy proxyForHost(HostAndPort address)


nit: this is unused

Thank you for reviewing the PR. I have carried out real test with Azure storage account.

The method proxyForHost was unused, so I’ve removed it. It likely came in earlier as part of the ADLS Gen1 support, which has since been removed in this implementation.

imjalpreet · 2025-05-22T13:23:05Z

presto-delta/src/main/java/com/facebook/presto/delta/DeltaModule.java

+        configBinder(binder).bindConfig(HiveAzureConfig.class);
+        binder.bind(AzureConfigurationInitializer.class).to(HiveAzureConfigurationInitializer.class).in(Scopes.SINGLETON);


As we have already created HiveAzureModule, please re-use that. You can bind it in DeltaConnectorFactory

Thanks for the suggestion. I’ve now removed the separate binding and reused the existing HiveAzureModule by binding it in DeltaConnectorFactory as recommended.

imjalpreet · 2025-05-22T13:25:08Z

presto-hive/src/main/java/com/facebook/presto/hive/azure/HiveAzureConfigurationInitializer.java

+    public HiveAzureConfigurationInitializer(HiveAzureConfig config)
+    {
+        this.wasbAccessKey = dropEmpty(config.getWasbAccessKey());
+        this.wasbStorageAccount = dropEmpty(config.getWasbStorageAccount());
+        if (wasbAccessKey.isPresent() || wasbStorageAccount.isPresent()) {
+            checkArgument(
+                    wasbAccessKey.isPresent() && wasbStorageAccount.isPresent(),
+                    "If WASB storage account or access key is set, both must be set");
+        }
+
+        this.abfsAccessKey = dropEmpty(config.getAbfsAccessKey());
+        this.abfsStorageAccount = dropEmpty(config.getAbfsStorageAccount());
+        this.abfsOAuthClientEndpoint = dropEmpty(config.getAbfsOAuthClientEndpoint());
+        this.abfsOAuthClientId = dropEmpty(config.getAbfsOAuthClientId());
+        this.abfsOAuthClientSecret = dropEmpty(config.getAbfsOAuthClientSecret());
+        if (abfsOAuthClientEndpoint.isPresent() || abfsOAuthClientSecret.isPresent() || abfsOAuthClientId.isPresent()) {
+            checkArgument(
+                    abfsOAuthClientEndpoint.isPresent() && abfsOAuthClientId.isPresent() && abfsOAuthClientSecret.isPresent(),
+                    "If any of ABFS OAuth2 Client endpoint, ID, and secret are set, all must be set.");
+        }
+    }


Do we allow the same catalog to have both WASB and ABFS configs? Is it supported?

Currently, the implementation accepts both WASB and ABFS configs if all their required fields are present. However, there's no specific logic that handles what happens if both are configured — it depends on the Hadoop file system behavior. If we want to restrict using both in a single catalog, I can add a validation check to ensure only one type is configured.

Would you be able to validate what happens if we configure both of them simultaneously? Based on that, we can decide whether or not to restrict this behavior.

imjalpreet · 2025-05-22T13:26:44Z

presto-hive/src/main/java/com/facebook/presto/hive/azure/HiveAzureConfig.java

Please add a UT for this class

imjalpreet · 2025-05-22T13:29:25Z

presto-hudi/src/main/java/com/facebook/presto/hudi/HudiModule.java

+        configBinder(binder).bindConfig(HiveAzureConfig.class);
        binder.bind(GcsConfigurationInitializer.class).to(HiveGcsConfigurationInitializer.class).in(Scopes.SINGLETON);
+        binder.bind(AzureConfigurationInitializer.class).to(HiveAzureConfigurationInitializer.class).in(Scopes.SINGLETON);


Same here, as we have already created HiveAzureModule, please re-use that. You can bind it in HudiConnectorFactory

imjalpreet · 2025-05-22T13:31:09Z

presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergCommonModule.java

+        configBinder(binder).bindConfig(HiveAzureConfig.class);
        binder.bind(GcsConfigurationInitializer.class).to(HiveGcsConfigurationInitializer.class).in(Scopes.SINGLETON);
+        binder.bind(AzureConfigurationInitializer.class).to(HiveAzureConfigurationInitializer.class).in(Scopes.SINGLETON);


As we have already created HiveAzureModule, please re-use that. You can bind it in InternalIcebergConnectorFactory

Code updated.

imjalpreet · 2025-09-18T19:55:16Z

@mehradpk, can you please help rebase this PR on master? Hadoop Upgrade is merged to master, so we can mark this PR as ready for review. Thanks!

sourcery-ai · 2025-09-22T04:07:41Z

Reviewer's Guide

Adds Azure Blob Storage (wasb://) and ADLS Gen2 (abfs://) support to the Hive connector by introducing new Azure-specific configuration classes and modules, and integrating them into existing HDFS configuration initializers, connector factories, and tests.

Class diagram for updated HdfsConfigurationInitializer and connector factories

classDiagram
    class HdfsConfigurationInitializer {
        -AzureConfigurationInitializer azureConfigurationInitialize
        +HdfsConfigurationInitializer(..., AzureConfigurationInitializer)
        +updateConfiguration(Configuration)
    }
    HdfsConfigurationInitializer o-- AzureConfigurationInitializer

    class HiveConnectorFactory {
        +create(...)
    }
    HiveConnectorFactory --> HiveAzureModule

    class HiveAzureModule {
        +setup(Binder)
    }
    HiveAzureModule --> HiveAzureConfig
    HiveAzureModule --> HiveAzureConfigurationInitializer

    class IcebergNativeCatalogFactory {
        -AzureConfigurationInitializer azureConfigurationInitialize
        +IcebergNativeCatalogFactory(..., AzureConfigurationInitializer)
        +getHadoopConfiguration()
    }
    IcebergNativeCatalogFactory o-- AzureConfigurationInitializer

    class IcebergNessieCatalogFactory {
        +IcebergNessieCatalogFactory(..., AzureConfigurationInitializer)
    }
    IcebergNessieCatalogFactory o-- AzureConfigurationInitializer

    class IcebergRestCatalogFactory {
        +IcebergRestCatalogFactory(..., AzureConfigurationInitializer)
    }
    IcebergRestCatalogFactory o-- AzureConfigurationInitializer

File-Level Changes

Change	Details	Files
Introduce Azure configuration and initialization classes	Define HiveAzureConfig for wasb and abfs account, key, and OAuth2 properties Create AzureConfigurationInitializer interface Implement HiveAzureConfigurationInitializer to apply Azure settings to Hadoop Configuration Add HiveAzureModule for Guice binding of Azure config and initializer	`HiveAzureConfig.java` `AzureConfigurationInitializer.java` `HiveAzureConfigurationInitializer.java` `HiveAzureModule.java`
Extend HDFS and Iceberg initializers to include Azure setup	Add azureConfigurationInitialize parameter to HdfsConfigurationInitializer and Iceberg configuration constructors Invoke azureConfigurationInitialize.updateConfiguration in updateConfiguration methods Propagate Azure initializer through IcebergNativeCatalogFactory, IcebergNessieCatalogFactory, IcebergRestCatalogFactory	`HdfsConfigurationInitializer.java` `IcebergNativeCatalogFactory.java` `IcebergNessieCatalogFactory.java` `IcebergRestCatalogFactory.java`
Register Azure module in connector factories	Include HiveAzureModule in DeltaConnectorFactory, HiveConnectorFactory, HudiConnectorFactory, and InternalIcebergConnectorFactory chains Bind AzureConfigurationInitializer through HiveConnectorFactory setup	`DeltaConnectorFactory.java` `HiveConnectorFactory.java` `HudiConnectorFactory.java` `InternalIcebergConnectorFactory.java`
Update tests to inject Azure initializer	Modify HiveTestUtils to instantiate HiveAzureConfigurationInitializer Update Iceberg and Hive file system tests to pass azure initializer where HdfsConfigurationInitializer is constructed	`HiveTestUtils.java` `TestIcebergSmokeHadoop.java` `TestIcebergSmokeOnS3Hadoop.java` `TestIcebergSmokeNessie.java` `TestIcebergSmokeRest.java` `TestIcebergSmokeRestNestedNamespace.java` `AbstractTestHiveFileSystemS3.java` `S3SelectTestHelper.java` `IcebergDistributedTestBase.java`

Tips and commands

Interacting with Sourcery

Trigger a new review: Comment @sourcery-ai review on the pull request.
Continue discussions: Reply directly to Sourcery's review comments.
Generate a GitHub issue from a review comment: Ask Sourcery to create an
issue from a review comment by replying to it. You can also reply to a
review comment with @sourcery-ai issue to create an issue from it.
Generate a pull request title: Write @sourcery-ai anywhere in the pull
request title to generate a title at any time. You can also comment
@sourcery-ai title on the pull request to (re-)generate the title at any time.
Generate a pull request summary: Write @sourcery-ai summary anywhere in
the pull request body to generate a PR summary at any time exactly where you
want it. You can also comment @sourcery-ai summary on the pull request to
(re-)generate the summary at any time.
Generate reviewer's guide: Comment @sourcery-ai guide on the pull
request to (re-)generate the reviewer's guide at any time.
Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
pull request to resolve all Sourcery comments. Useful if you've already
addressed all the comments and don't want to see them anymore.
Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
request to dismiss all existing Sourcery reviews. Especially useful if you
want to start fresh with a new review - don't forget to comment
@sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

Enable or disable review features such as the Sourcery-generated pull request
summary, the reviewer's guide, and others.
Change the review language.
Add, remove or edit custom review instructions.
Adjust other review settings.

Getting Help

Contact our support team for questions or feedback.
Visit our documentation for detailed guides and information.
Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

sourcery-ai

Hey there - I've reviewed your changes and they look great!

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location> `presto-hive/src/main/java/com/facebook/presto/hive/azure/HiveAzureConfigurationInitializer.java:72-77` </location>
<code_context>
+            config.set("fs.abfs.impl", AzureBlobFileSystem.class.getName());
+        }
+
+        if (abfsOAuthClientEndpoint.isPresent() && abfsOAuthClientId.isPresent() && abfsOAuthClientSecret.isPresent()) {
+            config.set(format("fs.azure.account.auth.type.%s.dfs.core.windows.net", abfsStorageAccount.get()), "OAuth");
+            config.set("fs.azure.account.oauth.provider.type", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider");
+            config.set(format("fs.azure.account.oauth2.client.endpoint.%s.dfs.core.windows.net", abfsStorageAccount.get()), abfsOAuthClientEndpoint.get());
+            config.set(format("fs.azure.account.oauth2.client.id.%s.dfs.core.windows.net", abfsStorageAccount.get()), abfsOAuthClientId.get());
+            config.set(format("fs.azure.account.oauth2.client.secret.%s.dfs.core.windows.net", abfsStorageAccount.get()), abfsOAuthClientSecret.get());
+        }
+
</code_context>

<issue_to_address>
**issue (bug_risk):** Potential NPE if abfsStorageAccount is not present when setting OAuth config.

Please add an explicit check to ensure abfsStorageAccount is present before calling get(), or include it in the earlier checkArgument for OAuth parameters to prevent a NullPointerException.
</issue_to_address>

### Comment 2
<location> `presto-hive/src/main/java/com/facebook/presto/hive/azure/HiveAzureConfigurationInitializer.java:67-70` </location>
<code_context>
+
+        if (abfsAccessKey.isPresent() && abfsStorageAccount.isPresent()) {
+            config.set(format("fs.azure.account.key.%s.dfs.core.windows.net", abfsStorageAccount.get()), abfsAccessKey.get());
+            config.set("fs.abfs.impl", AzureBlobFileSystem.class.getName());
+        }
+
</code_context>

<issue_to_address>
**suggestion:** Setting 'fs.abfs.impl' may override user configuration.

Check if 'fs.abfs.impl' is already set in the configuration before assigning it, or clearly document that this will override any existing value.

```suggestion
        if (abfsAccessKey.isPresent() && abfsStorageAccount.isPresent()) {
            config.set(format("fs.azure.account.key.%s.dfs.core.windows.net", abfsStorageAccount.get()), abfsAccessKey.get());
            if (config.get("fs.abfs.impl") == null) {
                config.set("fs.abfs.impl", AzureBlobFileSystem.class.getName());
            }
        }
```
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

presto-hive/src/main/java/com/facebook/presto/hive/azure/HiveAzureConfigurationInitializer.java

steveburnett · 2025-09-22T14:12:45Z

Should any documentation be added? Perhaps some examples? https://github.com/prestodb/presto/blob/master/presto-docs/src/main/sphinx/connector/hive.rst

mehradpk · 2025-09-24T17:33:45Z

Should any documentation be added? Perhaps some examples? https://github.com/prestodb/presto/blob/master/presto-docs/src/main/sphinx/connector/hive.rst

Yes, documentation should be updated. Since ADLS support is being added, we should document the required configuration properties (e.g., fs.azure.account.key, fs.azure.account.auth.type, etc.), how users can enable it, and any limitations.

I can add a new subsection under the Hive connector.

sourcery-ai

New security issues found

presto-docs/src/main/sphinx/connector/hive.rst

mehradpk · 2025-10-30T07:24:11Z

@imjalpreet All changes addressed. Re-requested review but it may not have triggered properly - could you please review the changes?

@steveburnett Documentation has been added, Could you please review?

steveburnett

Thanks for the doc! The content looks good, but there are several formatting errors.

You might find https://github.com/prestodb/presto/tree/master/presto-docs#building-the-documentation and https://github.com/prestodb/presto/tree/master/presto-docs#viewing-the-documentation helpful to run local doc builds and test your documentation format locally.

steveburnett · 2025-10-30T14:20:30Z

presto-docs/src/main/sphinx/connector/hive.rst

+-------------------
+
+The Hive connector supports Azure Blob Storage (``wasb://``) and Azure Data Lake Storage Gen2 (``abfs://``).
+Configure Azure storage access in your catalog properties file (e.g., ``etc/catalog/hive.properties``).


Suggested change

Configure Azure storage access in your catalog properties file (e.g., ``etc/catalog/hive.properties``).

Configure Azure storage access in your catalog properties file, such as in ``etc/catalog/hive.properties``.

Avoid Latin abbreviations. See the GitLab documentation style guide recommended word list entry for e.g. for alternatives. I've suggested one alternative here, please revise it if a different alternative fits better.

presto-docs/src/main/sphinx/connector/hive.rst

steveburnett · 2025-10-30T14:26:51Z

presto-docs/src/main/sphinx/connector/hive.rst

+Configuration Properties
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+================================================= =====================================================================


The table in lines 737-757 isn't displaying in the local doc build. See screenshot.

steveburnett · 2025-10-30T14:27:34Z

presto-docs/src/main/sphinx/connector/hive.rst

+Property Name                                       Description
+================================================= =====================================================================
+``hive.azure.wasb.storage-account``                 Azure Blob Storage account name
+                                                    (without ``.blob.core.windows.net suffix``)


suffix should not be within the formatting ticks

steveburnett · 2025-10-30T14:27:40Z

presto-docs/src/main/sphinx/connector/hive.rst

+``hive.azure.wasb.access-key``                      Azure Blob Storage access key
+
+``hive.azure.abfs.storage-account``                 ADLS Gen2 storage account name
+                                                    (without ``.blob.core.windows.net suffix``)


suffix should not be within the formatting ticks

steveburnett · 2025-10-30T14:29:35Z

presto-docs/src/main/sphinx/connector/hive.rst

+* WASB requires both ``wasb.storage-account`` and ``wasb.access-key`` to be set
+* ABFS with shared key requires both ``abfs.storage-account`` and ``abfs.access-key`` to be set
+* ABFS with OAuth requires ``abfs.storage-account``, ``abfs.oauth.client-endpoint``, ``abfs.oauth.client-id``, and
+``abfs.oauth.client-secret`` (all four properties) to be set


line 765 should be part of the list item above it. See screenshot for how it currently displays in a local doc build.

presto-docs/src/main/sphinx/connector/hive.rst

steveburnett · 2025-10-30T14:32:27Z

presto-docs/src/main/sphinx/connector/hive.rst

+    abfs://<container>@<storage-account>.dfs.core.windows.net/<path>
+
+Usage Examples
+^^^^^^^^^^


Suggested change

^^^^^^^^^^

^^^^^^^^^^^^^^

presto-docs/src/main/sphinx/connector/hive.rst

steveburnett · 2025-10-30T14:35:48Z

presto-docs/src/main/sphinx/connector/hive.rst

+#####################
+
+If you see ``403 Forbidden`` or ``401 Unauthorized``:
+* Verify storage account names are correct (without .blob.core.windows.net or .dfs.core.windows.net suffix)


The unordered lists in Authentication Errors, Connection Errors, and Path Errors have formatting errors.

fixed all doc related issues.

imjalpreet

Thank you, @mehradpk. I finished another round of review. It mostly looks good now, with some minor suggestions.

imjalpreet · 2025-11-03T21:10:32Z

presto-delta/src/main/java/com/facebook/presto/delta/DeltaModule.java


        configBinder(binder).bindConfig(HiveGcsConfig.class);
        binder.bind(GcsConfigurationInitializer.class).to(HiveGcsConfigurationInitializer.class).in(Scopes.SINGLETON);
-


nit: unrelated, please revert

imjalpreet · 2025-11-03T21:27:20Z

presto-hive/src/main/java/com/facebook/presto/hive/azure/HiveAzureConfigurationInitializer.java

+        if (wasbAccessKey.isPresent() || wasbStorageAccount.isPresent()) {
+            checkArgument(
+                    wasbAccessKey.isPresent() && wasbStorageAccount.isPresent(),
+                    "If WASB storage account or access key is set, both must be set");
+        }


Suggested change

if (wasbAccessKey.isPresent() || wasbStorageAccount.isPresent()) {

checkArgument(

wasbAccessKey.isPresent() && wasbStorageAccount.isPresent(),

"If WASB storage account or access key is set, both must be set");

}

checkArgument(

wasbAccessKey.isPresent() == wasbStorageAccount.isPresent(),

"If WASB storage account or access key is set, both must be set");

imjalpreet · 2025-11-03T21:30:26Z

presto-hive/src/main/java/com/facebook/presto/hive/azure/HiveAzureConfigurationInitializer.java

+        this.abfsOAuthClientEndpoint = dropEmpty(config.getAbfsOAuthClientEndpoint());
+        this.abfsOAuthClientId = dropEmpty(config.getAbfsOAuthClientId());
+        this.abfsOAuthClientSecret = dropEmpty(config.getAbfsOAuthClientSecret());
+        if (abfsOAuthClientEndpoint.isPresent() || abfsOAuthClientSecret.isPresent() || abfsOAuthClientId.isPresent()) {


abfsStorageAccount.isPresent() is missing in the if condition

Refactored this piece of code to handle all scenarios. Previously, including abfsStorageAccount in the OAuth check can result in error if a user intended to use only the access key. Now the code separately validates - access key and OAuth.

imjalpreet · 2025-11-03T21:31:49Z

presto-hive/src/main/java/com/facebook/presto/hive/azure/HiveAzureConfigurationInitializer.java

+    public HiveAzureConfigurationInitializer(HiveAzureConfig config)
+    {
+        this.wasbAccessKey = dropEmpty(config.getWasbAccessKey());
+        this.wasbStorageAccount = dropEmpty(config.getWasbStorageAccount());
+        if (wasbAccessKey.isPresent() || wasbStorageAccount.isPresent()) {
+            checkArgument(
+                    wasbAccessKey.isPresent() && wasbStorageAccount.isPresent(),
+                    "If WASB storage account or access key is set, both must be set");
+        }
+
+        this.abfsAccessKey = dropEmpty(config.getAbfsAccessKey());
+        this.abfsStorageAccount = dropEmpty(config.getAbfsStorageAccount());
+        this.abfsOAuthClientEndpoint = dropEmpty(config.getAbfsOAuthClientEndpoint());
+        this.abfsOAuthClientId = dropEmpty(config.getAbfsOAuthClientId());
+        this.abfsOAuthClientSecret = dropEmpty(config.getAbfsOAuthClientSecret());
+        if (abfsOAuthClientEndpoint.isPresent() || abfsOAuthClientSecret.isPresent() || abfsOAuthClientId.isPresent()) {
+            checkArgument(
+                    abfsOAuthClientEndpoint.isPresent() && abfsOAuthClientId.isPresent() && abfsOAuthClientSecret.isPresent(),
+                    "If any of ABFS OAuth2 Client endpoint, ID, and secret are set, all must be set.");
+        }
+    }


Would you be able to validate what happens if we configure both of them simultaneously? Based on that, we can decide whether or not to restrict this behavior.

imjalpreet · 2025-11-03T21:33:33Z

presto-hive/src/main/java/com/facebook/presto/hive/azure/HiveAzureConfigurationInitializer.java

+            }
+        }
+
+        if (abfsOAuthClientEndpoint.isPresent() && abfsOAuthClientId.isPresent() && abfsOAuthClientSecret.isPresent()) {


abfsStorageAccount.isPresent() check is missing here as well

imjalpreet · 2025-11-03T21:40:43Z

presto-hive/src/main/java/com/facebook/presto/hive/azure/HiveAzureModule.java

+import static com.facebook.airlift.configuration.ConfigBinder.configBinder;
+
+public class HiveAzureModule
+        extends AbstractConfigurationAwareModule


We don't need to use AbstractConfigurationAwareModule here, as we are not binding based on the value of any configuration.

You can implement com.google.inject.Module

imjalpreet · 2025-11-03T21:43:14Z

presto-hudi/src/main/java/com/facebook/presto/hudi/HudiModule.java

        configBinder(binder).bindConfig(MetastoreClientConfig.class);
        configBinder(binder).bindConfig(ThriftHiveMetastoreConfig.class);
        configBinder(binder).bindConfig(HiveGcsConfig.class);
+//        configBinder(binder).bindConfig(HiveAzureConfig.class);


nit: remove commented code

imjalpreet · 2025-11-03T21:43:18Z

presto-hudi/src/main/java/com/facebook/presto/hudi/HudiModule.java

        configBinder(binder).bindConfig(HiveGcsConfig.class);
+//        configBinder(binder).bindConfig(HiveAzureConfig.class);
        binder.bind(GcsConfigurationInitializer.class).to(HiveGcsConfigurationInitializer.class).in(Scopes.SINGLETON);
+//        binder.bind(AzureConfigurationInitializer.class).to(HiveAzureConfigurationInitializer.class).in(Scopes.SINGLETON);


nit: remove commented code

imjalpreet · 2025-11-03T21:43:58Z

presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergCommonModule.java

        configBinder(binder).bindConfig(FileMergeCacheConfig.class);
        binder.bind(CacheStats.class).in(Scopes.SINGLETON);
        configBinder(binder).bindConfig(HiveGcsConfig.class);
+//        configBinder(binder).bindConfig(HiveAzureConfig.class);


nit: remove commented code

imjalpreet · 2025-11-03T21:44:02Z

presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergCommonModule.java

        configBinder(binder).bindConfig(HiveGcsConfig.class);
+//        configBinder(binder).bindConfig(HiveAzureConfig.class);
        binder.bind(GcsConfigurationInitializer.class).to(HiveGcsConfigurationInitializer.class).in(Scopes.SINGLETON);
+//        binder.bind(AzureConfigurationInitializer.class).to(HiveAzureConfigurationInitializer.class).in(Scopes.SINGLETON);


nit: remove commented code

Introduce support for Azure storage backends including Azure Blob Storage (using the wasbs:// scheme) and Azure Data Lake Storage Gen2 (using the abfss:// scheme) in the Hive connector. Key changes: - Added HiveAzureConfigurationInitializer to inject relevant Azure configurations into Hadoop Configuration - Introduced HiveAzureConfig to allow catalog-level configuration of Azure properties - Updated HdfsConfigurationInitializer and HiveConnectorFactory to delegate Azure-specific config setup - Registered configuration initializer in Hive module Supports shared key and OAuth2-based authentication.

mehradpk · 2025-11-10T21:46:13Z

@imjalpreet #25107 (comment) --> Yes.

After testing the behavior of the HiveAzureConfigurationInitializer with different configurations:

ABFS + WASB together: This works perfectly fine. The two are independent, as their configuration properties and endpoints are separate, so having both configured does not cause any issues.
ABFS - OAuth + AccessKey together: If both are configured, OAuth takes precedence over AccessKey at runtime. But, there’s no fallback to AccessKey if OAuth authentication credentials are not valid, the operation will fail.

steveburnett

LGTM! (docs)

Pull updated branch, new local doc build. Looks good, thanks!

prestodb-ci added the from:IBM PR from IBM label May 14, 2025

imjalpreet requested changes May 22, 2025

View reviewed changes

mehradpk force-pushed the adls_support branch from 87ac275 to 64eff1c Compare May 29, 2025 14:21

mehradpk force-pushed the adls_support branch from 64eff1c to b221385 Compare September 22, 2025 04:05

mehradpk marked this pull request as ready for review September 22, 2025 04:06

mehradpk requested review from a team, 7c00, ZacBlanco, hantangwangd and vinothchandar as code owners September 22, 2025 04:06

prestodb-ci requested review from a team, auden-woolfson and jp-sivaprasad and removed request for a team September 22, 2025 04:06

mehradpk requested a review from imjalpreet September 22, 2025 04:07

sourcery-ai bot reviewed Sep 22, 2025

View reviewed changes

presto-hive/src/main/java/com/facebook/presto/hive/azure/HiveAzureConfigurationInitializer.java Outdated Show resolved Hide resolved

presto-hive/src/main/java/com/facebook/presto/hive/azure/HiveAzureConfigurationInitializer.java Show resolved Hide resolved

mehradpk changed the title ~~[DO NOT REVIEW] Add support for Azure Blob Storage and ADLS Gen2 in Hive connector~~ Add support for Azure Blob Storage and ADLS Gen2 in Hive connector Sep 23, 2025

mehradpk force-pushed the adls_support branch from b221385 to 46c87ca Compare October 6, 2025 06:37

mehradpk changed the title ~~Add support for Azure Blob Storage and ADLS Gen2 in Hive connector~~ feat: Add support for Azure Blob Storage and ADLS Gen2 in Hive connector Oct 30, 2025

mehradpk force-pushed the adls_support branch from 46c87ca to f98eb88 Compare October 30, 2025 07:07

mehradpk requested review from elharo and steveburnett as code owners October 30, 2025 07:07

sourcery-ai bot requested changes Oct 30, 2025

View reviewed changes

presto-docs/src/main/sphinx/connector/hive.rst Outdated Show resolved Hide resolved

presto-docs/src/main/sphinx/connector/hive.rst Outdated Show resolved Hide resolved

presto-docs/src/main/sphinx/connector/hive.rst Outdated Show resolved Hide resolved

mehradpk force-pushed the adls_support branch from f98eb88 to 9025dd1 Compare October 30, 2025 07:14

steveburnett requested changes Oct 30, 2025

View reviewed changes

imjalpreet requested changes Nov 3, 2025

View reviewed changes

mehradpk force-pushed the adls_support branch from 9025dd1 to a65a447 Compare November 10, 2025 20:21

mehradpk requested review from imjalpreet and steveburnett November 10, 2025 21:46

steveburnett approved these changes Nov 10, 2025

View reviewed changes

		configBinder(binder).bindConfig(HiveAzureConfig.class);
		binder.bind(AzureConfigurationInitializer.class).to(HiveAzureConfigurationInitializer.class).in(Scopes.SINGLETON);

	Configure Azure storage access in your catalog properties file (e.g., ``etc/catalog/hive.properties``).
	Configure Azure storage access in your catalog properties file, such as in ``etc/catalog/hive.properties``.


		configBinder(binder).bindConfig(HiveGcsConfig.class);
		binder.bind(GcsConfigurationInitializer.class).to(HiveGcsConfigurationInitializer.class).in(Scopes.SINGLETON);

feat: Add support for Azure Blob Storage and ADLS Gen2 in Hive connector #25107

Are you sure you want to change the base?

feat: Add support for Azure Blob Storage and ADLS Gen2 in Hive connector #25107

Uh oh!

Conversation

mehradpk commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Impact

Test Plan

Contributor checklist

Uh oh!

prestodb-ci commented May 14, 2025

Uh oh!

imjalpreet left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

imjalpreet commented Sep 18, 2025

Uh oh!

sourcery-ai bot commented Sep 22, 2025

Reviewer's Guide

Class diagram for updated HdfsConfigurationInitializer and connector factories

File-Level Changes

Interacting with Sourcery

Customizing Your Experience

Getting Help

Uh oh!

sourcery-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

steveburnett commented Sep 22, 2025

Uh oh!

mehradpk commented Sep 24, 2025

Uh oh!

sourcery-ai bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mehradpk commented Oct 30, 2025

Uh oh!

steveburnett left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mehradpk commented May 14, 2025 •

edited

Loading