Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support WASB scheme in ADLSFileIO #11830

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

mrcnc
Copy link
Contributor

@mrcnc mrcnc commented Dec 19, 2024

This is my third attempt to resolve #10127 (...third time's the charm 😉)

We needed to revert #11504 since it introduced a breaking change for ADLSFileIO clients that were using SAS tokens. This PR fixes the issue by introducing a new host() method to ADLSLocation and uses it in ADLSFileIO when calling applyClientConfiguration.

@mrcnc mrcnc marked this pull request as ready for review December 19, 2024 23:27
Comment on lines +66 to +67
"wasb", ADLS_FILE_IO_IMPL,
"wasbs", ADLS_FILE_IO_IMPL);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to verify that updating this default behavior of ResolvingFileIO is not considered a breaking change, since it could potentially force clients using the WASB scheme with HadoopFileIO to update their configuration. For example, if clients are currently using SAS tokens in an core-site.xml file, then it seems they need to either configure these in the AzureProperties for ADLSFileIO or explicitly configure HadoopFileIO as the implementation to get the same behavior.

@jbonofre jbonofre self-requested a review December 20, 2024 15:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enable reading WASB and WASBS file paths with ABFS and ABFSS
1 participant