HADOOP-19343. Add support for mkdir() and getFileStatus() #7721

arunkumarchacko · 2025-05-30T15:31:24Z

Description of PR

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…sts for getFileStatus(), delete() and mkdir() Some of the contract tests are disabled and will be enabled in a subsequent change

hadoop-yetus · 2025-06-05T09:53:43Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 17s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 1s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ HADOOP-19343 Compile Tests _
+0 🆗	mvndep	7m 19s		Maven dependency ordering for branch
+1 💚	mvninstall	47m 33s		HADOOP-19343 passed
+1 💚	compile	18m 0s		HADOOP-19343 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	15m 30s		HADOOP-19343 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	4m 40s		HADOOP-19343 passed
+1 💚	mvnsite	1m 22s		HADOOP-19343 passed
+1 💚	javadoc	1m 16s		HADOOP-19343 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 11s		HADOOP-19343 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+0 🆗	spotbugs	0m 43s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	39m 51s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	1m 32s		Maven dependency ordering for patch
+1 💚	mvninstall	10m 54s		the patch passed
+1 💚	compile	17m 11s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	17m 11s		the patch passed
+1 💚	compile	15m 34s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	15m 34s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 41s		the patch passed
+1 💚	mvnsite	1m 20s		the patch passed
+1 💚	javadoc	1m 19s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 16s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+0 🆗	spotbugs	0m 38s		hadoop-project has no data from spotbugs
+1 💚	shadedclient	39m 47s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 36s		hadoop-project in the patch passed.
+1 💚	unit	0m 45s		hadoop-gcp in the patch passed.
+1 💚	asflicense	1m 4s		The patch does not generate ASF License warnings.
		242m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.50 ServerAPI=1.50 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7721/2/artifact/out/Dockerfile
GITHUB PR	#7721
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux bf32186103d0 5.15.0-139-generic #149-Ubuntu SMP Fri Apr 11 22:06:13 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HADOOP-19343 / `11a75b5`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7721/2/testReport/
Max. process+thread count	600 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-tools/hadoop-gcp U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7721/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

cnauroth

Hello @arunkumarchacko . This mostly looks good. I entered a few minor comments. Thank you.

hadoop-tools/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/ApiErrorExtractor.java

cnauroth · 2025-06-05T16:54:07Z

hadoop-tools/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/CreateBucketOptions.java

+import java.time.Duration;
+
+final class CreateBucketOptions {
+  static final CreateBucketOptions DEFAULT = new Builder().build(); // TODO: Make sure the defaults


I'm unclear on what this TODO means.

Added more details. The TODO comments will be addressed very soon.

hadoop-tools/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/CreateBucketOptions.java

hadoop-tools/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorageFileSystem.java

cnauroth · 2025-06-05T17:45:00Z

hadoop-tools/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleHadoopFileSystem.java

+      getGcsFs().mkdirs(gcsPath);
+    } catch (java.nio.file.FileAlreadyExistsException faee) {
+      // Need to convert to the Hadoop flavor of FileAlreadyExistsException.
+      throw (FileAlreadyExistsException)


Is the downcast redundant, as we are directly instantiating FileAlreadyExistsException?

without the typecast, the code wont compile and fails with error. "unreported exception java.lang.Throwable; must be caught or declared to be thrown"

hadoop-tools/hadoop-gcp/src/test/java/org/apache/hadoop/fs/gs/contract/GoogleContract.java

cnauroth

Thanks for the updates, @arunkumarchacko ! +1 pending pre-submit.

hadoop-yetus · 2025-06-06T07:29:17Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 9 new or modified test files.
			_ HADOOP-19343 Compile Tests _
+0 🆗	mvndep	8m 28s		Maven dependency ordering for branch
+1 💚	mvninstall	44m 58s		HADOOP-19343 passed
+1 💚	compile	17m 42s		HADOOP-19343 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	compile	15m 13s		HADOOP-19343 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	checkstyle	4m 44s		HADOOP-19343 passed
+1 💚	mvnsite	1m 23s		HADOOP-19343 passed
+1 💚	javadoc	1m 23s		HADOOP-19343 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 15s		HADOOP-19343 passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+0 🆗	spotbugs	0m 43s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	39m 44s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	1m 44s		Maven dependency ordering for patch
+1 💚	mvninstall	10m 56s		the patch passed
+1 💚	compile	17m 56s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javac	17m 56s		the patch passed
+1 💚	compile	18m 21s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+1 💚	javac	18m 21s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	4m 57s		the patch passed
+1 💚	mvnsite	1m 30s		the patch passed
+1 💚	javadoc	1m 12s		the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	1m 8s		the patch passed with JDK Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
+0 🆗	spotbugs	0m 33s		hadoop-project has no data from spotbugs
+1 💚	shadedclient	41m 39s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 31s		hadoop-project in the patch passed.
+1 💚	unit	0m 40s		hadoop-gcp in the patch passed.
+1 💚	asflicense	1m 3s		The patch does not generate ASF License warnings.
		246m 34s

Subsystem	Report/Notes
Docker	ClientAPI=1.50 ServerAPI=1.50 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7721/3/artifact/out/Dockerfile
GITHUB PR	#7721
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
uname	Linux 713c5509d04f 5.15.0-139-generic #149-Ubuntu SMP Fri Apr 11 22:06:13 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	HADOOP-19343 / `35c30c3`
Default Java	Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~~us1-0ubuntu1~~20.04-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7721/3/testReport/
Max. process+thread count	530 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-tools/hadoop-gcp U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7721/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Add support for mkdir() and getFileStatus()

eb70f03

github-actions bot added build trunk TOOLS labels May 30, 2025

Add support for listStatus() and delete() operations. Add contract te…

11a75b5

…sts for getFileStatus(), delete() and mkdir() Some of the contract tests are disabled and will be enabled in a subsequent change

cnauroth reviewed Jun 5, 2025

View reviewed changes

Take CR comments

35c30c3

cnauroth approved these changes Jun 6, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-19343. Add support for mkdir() and getFileStatus() #7721

HADOOP-19343. Add support for mkdir() and getFileStatus() #7721

arunkumarchacko commented May 30, 2025

Uh oh!

hadoop-yetus commented Jun 5, 2025

Uh oh!

cnauroth left a comment

Uh oh!

Uh oh!

cnauroth Jun 5, 2025

Uh oh!

arunkumarchacko Jun 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cnauroth Jun 5, 2025

Uh oh!

arunkumarchacko Jun 6, 2025

Uh oh!

Uh oh!

cnauroth left a comment

Uh oh!

hadoop-yetus commented Jun 6, 2025

Uh oh!

Uh oh!

HADOOP-19343. Add support for mkdir() and getFileStatus() #7721

Are you sure you want to change the base?

HADOOP-19343. Add support for mkdir() and getFileStatus() #7721

Conversation

arunkumarchacko commented May 30, 2025

Description of PR

How was this patch tested?

For code changes:

Uh oh!

hadoop-yetus commented Jun 5, 2025

Uh oh!

cnauroth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cnauroth Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

arunkumarchacko Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cnauroth Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

arunkumarchacko Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cnauroth left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Jun 6, 2025

Uh oh!

Uh oh!