Skip to content

Commit 622bbf2

Browse files
zekai-liMridul Muralidharan
authored andcommitted
[SPARK-44845][YARN][DEPLOY] Fix file system uri comparison function
### What changes were proposed in this pull request? What changes were proposed in this pull request? ### Why are the changes needed? In the org.apache.spark.deploy.yarn.Client#compareUri method, hdfs://hadoop81:8020 and hdfs://192.168.0.81:8020 are regarded as different file systems (hadoop81 corresponds to 192.168.0.81). The specific reason is that in the last pr, different URIs of user information are also regarded as different file systems. Uri.getauthority is used to determine the user information, but authority contains the host so the URI above must be different from authority. To determine whether the user authentication information is different, you only need to determine URI.getUserInfo. the last pr and issue link: https://issues.apache.org/jira/browse/SPARK-22587 apache#19885 ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Closes apache#42529 from zekai-li/master. Authored-by: zekai-li <[email protected]> Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
1 parent 56393b9 commit 622bbf2

File tree

2 files changed

+7
-5
lines changed

2 files changed

+7
-5
lines changed

resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1618,9 +1618,10 @@ private[spark] object Client extends Logging {
16181618
return false
16191619
}
16201620

1621-
val srcAuthority = srcUri.getAuthority()
1622-
val dstAuthority = dstUri.getAuthority()
1623-
if (srcAuthority != null && !srcAuthority.equalsIgnoreCase(dstAuthority)) {
1621+
val srcUserInfo = Option(srcUri.getUserInfo).getOrElse("")
1622+
val dstUserInfo = Option(dstUri.getUserInfo).getOrElse("")
1623+
1624+
if (!srcUserInfo.equals(dstUserInfo)) {
16241625
return false
16251626
}
16261627

resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -675,7 +675,8 @@ class ClientSuite extends SparkFunSuite
675675
("files URI match test2", "file:///c:file1", "file://c:file2"),
676676
("files URI match test3", "file://host/file1", "file://host/file2"),
677677
("wasb URI match test", "wasb://bucket1@user", "wasb://bucket1@user/"),
678-
("hdfs URI match test", "hdfs:/path1", "hdfs:/path1")
678+
("hdfs URI match test1", "hdfs:/path1", "hdfs:/path1"),
679+
("hdfs URI match test2", "hdfs://localhost:8080", "hdfs://127.0.0.1:8080")
679680
)
680681

681682
matching.foreach { t =>
@@ -691,7 +692,7 @@ class ClientSuite extends SparkFunSuite
691692
("files URI unmatch test3", "file://host/file1", "file://host2/file2"),
692693
("wasb URI unmatch test1", "wasb://bucket1@user", "wasb://bucket2@user/"),
693694
("wasb URI unmatch test2", "wasb://bucket1@user", "wasb://bucket1@user2/"),
694-
("s3 URI unmatch test", "s3a://user@pass:bucket1/", "s3a://user2@pass2:bucket1/"),
695+
("s3 URI unmatch test", "s3a://user:pass@bucket1/", "s3a://user2:pass2@bucket1/"),
695696
("hdfs URI unmatch test1", "hdfs://namenode1/path1", "hdfs://namenode1:8080/path2"),
696697
("hdfs URI unmatch test2", "hdfs://namenode1:8020/path1", "hdfs://namenode1:8080/path2")
697698
)

0 commit comments

Comments
 (0)