Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HUDI-9040] Set the correct table path when renaming tables #12848

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
+38 −37
Open

[HUDI-9040] Set the correct table path when renaming tables #12848

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5dbee0c
[HUDI-9040] Set the correct table path when renaming tables
CTTY Feb 12, 2025
5199fac
address comments
CTTY Feb 23, 2025
00e6147
restructure test, minor
CTTY Feb 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions ...mmon/src/main/scala/org/apache/spark/sql/hudi/command/AlterHoodieTableRenameCommand.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,7 @@ case class AlterHoodieTableRenameCommand(
// update table properties path in every op
if (hoodieCatalogTable.table.properties.contains("path")) {
val catalogTable = sparkSession.sessionState.catalog.getTableMetadata(newName)
val path = catalogTable.storage.locationUri.get.getPath
AlterTableSetPropertiesCommand(newName, Map("path" -> path), isView).run(sparkSession)
AlterTableSetPropertiesCommand(newName, Map("path" -> catalogTable.location.toString), isView).run(sparkSession)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the base path of Hudi table change under the hood?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i.e., all files are moved to the new path?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, data files will not be relocated. The issue happened because how we set the path in AlterHoodieTableRenameCommand may strip the scheme and authority of path in some cases.

}

}
Expand Down
72 changes: 37 additions & 35 deletions ...k-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/ddl/TestAlterTable.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -310,40 +310,44 @@ class TestAlterTable extends HoodieSparkSqlTestBase {
}

test("Test Alter Rename Table") {
Seq("cow", "mor").foreach { tableType =>
val tableName = generateTableName
// Create table
spark.sql(
s"""
|create table $tableName (
| id int,
| name string,
| price double,
| ts long
|) using hudi
| tblproperties (
| type = '$tableType',
| primaryKey = 'id',
| preCombineField = 'ts'
| )
""".stripMargin)

// alter table name.
val newTableName = s"${tableName}_1"
val oldLocation = spark.sessionState.catalog.getTableMetadata(new TableIdentifier(tableName)).properties.get("path")
spark.sql(s"alter table $tableName rename to $newTableName")
val newLocation = spark.sessionState.catalog.getTableMetadata(new TableIdentifier(newTableName)).properties.get("path")
// only hoodieCatalog will set path to tblp
if (oldLocation.nonEmpty) {
assertResult(false)(
newLocation.equals(oldLocation)
)
} else {
assertResult(None)(newLocation)
}
}
}

test("Test Alter Rename Table With Location") {
withTempDir { tmp =>
Seq("cow", "mor").foreach { tableType =>
val tableName = generateTableName
// Create table
spark.sql(
s"""
|create table $tableName (
| id int,
| name string,
| price double,
| ts long
|) using hudi
| tblproperties (
| type = '$tableType',
| primaryKey = 'id',
| preCombineField = 'ts'
| )
""".stripMargin)

// alter table name.
val newTableName = s"${tableName}_1"
val oldLocation = spark.sessionState.catalog.getTableMetadata(new TableIdentifier(tableName)).properties.get("path")
spark.sql(s"alter table $tableName rename to $newTableName")
val newLocation = spark.sessionState.catalog.getTableMetadata(new TableIdentifier(newTableName)).properties.get("path")
// only hoodieCatalog will set path to tblp
if (oldLocation.nonEmpty) {
assertResult(false)(
newLocation.equals(oldLocation)
)
} else {
assertResult(None) (newLocation)
}


// Create table with location
val locTableName = s"${tableName}_loc"
val tablePath = s"${tmp.getCanonicalPath}/$locTableName"
Expand Down Expand Up @@ -371,11 +375,9 @@ class TestAlterTable extends HoodieSparkSqlTestBase {
val newLocation2 = spark.sessionState.catalog.getTableMetadata(new TableIdentifier(newLocTableName))
.properties.get("path")
if (oldLocation2.nonEmpty) {
// Remove the impact of the schema.
val oldLocation2Path = new Path(oldLocation2.get.stripPrefix("file:"))
val newLocation2Path = new Path(newLocation2.get.stripPrefix("file:"))
// the scheme and authority need to match as well
assertResult(true)(
newLocation2Path.equals(oldLocation2Path)
oldLocation2.get.equals(newLocation2.get)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the key change in test, we want to ensure the path won't change even with the scheme

)
} else {
assertResult(None) (newLocation2)
Expand Down
Loading