Added force virtual addressing configuration for S3, Alibaba OSS protocol to use PyArrowFileIO #1392

helmiazizm · 2024-12-02T03:31:32Z

Added the option to enforce virtual hosted style request for S3 PyArrow IO as it already has the support by PyArrow. Also added oss:// protocol into the list of S3 PyArrow scheme. This was addressed in #21 before.

…nd r2 protocol.

kevinjqliu · 2024-12-02T18:38:42Z

pyiceberg/io/pyarrow.py

@@ -373,6 +374,9 @@ def _initialize_fs(self, scheme: str, netloc: Optional[str] = None) -> FileSyste
            if session_name := get_first_property_value(self.properties, S3_ROLE_SESSION_NAME, AWS_ROLE_SESSION_NAME):
                client_kwargs["session_name"] = session_name

+            if force_virtual_addressing := self.properties.get(S3_FORCE_VIRTUAL_ADDRESSING):
+                client_kwargs["force_virtual_addressing"] = property_as_bool(self.properties, force_virtual_addressing, False)


force_virtual_addressing docs
https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html

kevinjqliu · 2024-12-02T18:39:32Z

pyiceberg/io/pyarrow.py

@@ -350,7 +351,7 @@ def parse_location(location: str) -> Tuple[str, str, str]:
            return uri.scheme, uri.netloc, f"{uri.netloc}{uri.path}"

    def _initialize_fs(self, scheme: str, netloc: Optional[str] = None) -> FileSystem:
-        if scheme in {"s3", "s3a", "s3n"}:
+        if scheme in {"s3", "s3a", "s3n", "oss", "r2"}:


what is oss? I've never heard of it. And does S3FileSystem support both oss and r2?

oss is protocol for Alibaba Cloud Object Storage Service, and it's compatible with S3 API as long as the URL is in virtual address style.

From my quick test it looks like the class does support OSS, but not sure yet about R2.

Plugging out R2. It uses account ID to create the URL, which is quite different from how S3 natively set the virtual hosted style.

thanks for testing it out! do you mind editing the PR description and mention this PR only supports oss

pyiceberg/io/__init__.py

kevinjqliu · 2024-12-02T18:41:12Z

mkdocs/docs/configuration.md

@@ -115,6 +115,7 @@ For the FileIO there are several configuration options available:
 | s3.region            | us-west-2                  | Sets the region of the bucket                                                                                                                                                                                                                             |
 | s3.proxy-uri         | <http://my.proxy.com:8080> | Configure the proxy server to be used by the FileIO.                                                                                                                                                                                                      |
 | s3.connect-timeout   | 60.0                       | Configure socket connection timeout, in seconds.                                                                                                                                                                                                          |
+| s3.force-virtual-addressing   | False                       | Configure the style of requests. Set `False` to use path-style request and `True` for virtual-hosted-style request.                                                                                                                                                                                                       |


nit: perhaps just copy/paste S3FileSystem docs
https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html

force_virtual_addressing bool, default False Whether to use virtual addressing of buckets. If true, then virtual addressing is always enabled. If false, then virtual addressing is only enabled if endpoint_override is empty. This can be used for non-AWS backends that only support virtual hosted-style access.

kevinjqliu · 2024-12-02T18:42:03Z

reopened #21, and added a few comments on this PR. Thanks for the contribution!

kevinjqliu · 2024-12-04T01:56:05Z

pyiceberg/io/pyarrow.py

@@ -350,7 +351,7 @@ def parse_location(location: str) -> Tuple[str, str, str]:
            return uri.scheme, uri.netloc, f"{uri.netloc}{uri.path}"

    def _initialize_fs(self, scheme: str, netloc: Optional[str] = None) -> FileSystem:
-        if scheme in {"s3", "s3a", "s3n"}:
+        if scheme in {"s3", "s3a", "s3n", "oss"}:


lets also update the docs to mention oss
https://py.iceberg.apache.org/configuration/#fileio

should we also add oss to fsspec?

iceberg-python/pyiceberg/io/fsspec.py

Lines 245 to 255 in 68e17af

SCHEME_TO_FS = {

"": _file,

"file": _file,

"s3": _s3,

"s3a": _s3,

"s3n": _s3,

"abfs": _adls,

"abfss": _adls,

"gs": _gs,

"gcs": _gs,

}

looks like theres https://github.com/fsspec/ossfs

looks like theres https://github.com/fsspec/ossfs

The project hasn't been maintained for almost a year now, and when I tried it on DuckDB it returns an unknown dependency error (?). I'd say it's probably safer to use s3fs than ossfs (not that I've tried it, though). I'll try making a new pull request for that later.

makes sense, thanks for looking into it! lets keep the scope just pyarrow for now

kevinjqliu · 2024-12-04T02:02:17Z

looks like theres a linter issue, you can run make lint locally to resolve it

kevinjqliu

LGTM! thanks for the contribution!

helmiazizm · 2024-12-04T09:59:14Z

I've tried to run make lint on python3.9 and it succeed, yet the test still failed. Is there any other thing I need to change?

kevinjqliu · 2024-12-04T18:55:47Z

@helmiazizm that's a intermittent issue with the CI, I've retrigger the run

Fokko

Looks good, thanks for updating the docs as well @helmiazizm 🙌

* Added force virtual addressing configuration for S3. Also added oss and r2 protocol. * Rewrote force virtual addressing as written in PyArrow documentation * Added the missing r2 key value in schema_to_file_io * Removed R2 protocol for now * Linter fix * Updated documentation for OSS support * Another linter fix

Added force virtual addressing configuration for S3. Also added oss a…

5b24172

…nd r2 protocol.

kevinjqliu reviewed Dec 2, 2024

View reviewed changes

helmiazizm added 2 commits December 3, 2024 09:50

Rewrote force virtual addressing as written in PyArrow documentation

360e305

Added the missing r2 key value in schema_to_file_io

31291f4

helmiazizm changed the title ~~Added force virtual addressing configuration for S3~~ Added force virtual addressing configuration for S3, oss and r2 protocol to use PyArrowFileIO Dec 3, 2024

Removed R2 protocol for now

6575552

helmiazizm changed the title ~~Added force virtual addressing configuration for S3, oss and r2 protocol to use PyArrowFileIO~~ Added force virtual addressing configuration for S3, Alibaba OSS protocol to use PyArrowFileIO Dec 3, 2024

kevinjqliu reviewed Dec 4, 2024

View reviewed changes

helmiazizm added 2 commits December 4, 2024 09:44

Linter fix

282bd47

Updated documentation for OSS support

7315525

kevinjqliu approved these changes Dec 4, 2024

View reviewed changes

Another linter fix

727666d

kevinjqliu requested a review from Fokko December 5, 2024 18:29

Fokko approved these changes Dec 9, 2024

View reviewed changes

Fokko merged commit d82f8f7 into apache:main Dec 9, 2024
8 checks passed

helmiazizm deleted the add-s3-virtual-addressing branch December 9, 2024 14:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added force virtual addressing configuration for S3, Alibaba OSS protocol to use PyArrowFileIO #1392

Added force virtual addressing configuration for S3, Alibaba OSS protocol to use PyArrowFileIO #1392

helmiazizm commented Dec 2, 2024 •

edited

Loading

kevinjqliu Dec 2, 2024

kevinjqliu Dec 2, 2024

helmiazizm Dec 3, 2024 •

edited

Loading

helmiazizm Dec 3, 2024

kevinjqliu Dec 4, 2024

kevinjqliu Dec 2, 2024

kevinjqliu commented Dec 2, 2024 •

edited

Loading

kevinjqliu Dec 4, 2024

kevinjqliu Dec 4, 2024

kevinjqliu Dec 4, 2024

helmiazizm Dec 4, 2024 •

edited

Loading

kevinjqliu Dec 4, 2024

kevinjqliu commented Dec 4, 2024

kevinjqliu left a comment

helmiazizm commented Dec 4, 2024

kevinjqliu commented Dec 4, 2024

Fokko left a comment

	SCHEME_TO_FS = {
	"": _file,
	"file": _file,
	"s3": _s3,
	"s3a": _s3,
	"s3n": _s3,
	"abfs": _adls,
	"abfss": _adls,
	"gs": _gs,
	"gcs": _gs,
	}

Added force virtual addressing configuration for S3, Alibaba OSS protocol to use PyArrowFileIO #1392

Added force virtual addressing configuration for S3, Alibaba OSS protocol to use PyArrowFileIO #1392

Conversation

helmiazizm commented Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helmiazizm Dec 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinjqliu commented Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

helmiazizm Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinjqliu commented Dec 4, 2024

kevinjqliu left a comment

Choose a reason for hiding this comment

helmiazizm commented Dec 4, 2024

kevinjqliu commented Dec 4, 2024

Fokko left a comment

Choose a reason for hiding this comment

helmiazizm commented Dec 2, 2024 •

edited

Loading

helmiazizm Dec 3, 2024 •

edited

Loading

kevinjqliu commented Dec 2, 2024 •

edited

Loading

helmiazizm Dec 4, 2024 •

edited

Loading