Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix table operations failing for ROW type column name containing colon #24811

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Fix table operations failing for ROW type column name containing colon #24811

wants to merge 2 commits into from

Conversation

mayankvadariya
Copy link
Contributor

Table operations fail when sub-column in ROW type column contains
colon(:) in name in Glue.

Description

Additional context and related issues

Release notes

(x) Release notes are required, with the following suggested text:

## Section
* Fix table operations failing for ROW type sub-column name containing colon in Glue. ({issue}`24811`)

@mayankvadariya
Fix table operations failing for ROW type column name containing colon
28e0f3d 
Table operations fail when sub-column in ROW type column contains
colon(`:`) in name in Glue.
@mayankvadariya
fixup! Fix table operations failing for ROW type column name containi…
888b3e5 
…ng colon
@cla-bot cla-bot bot added the cla-signed label Jan 27, 2025
@github-actions github-actions bot added the hive Hive connector label Jan 27, 2025
ebyhr
ebyhr reviewed Jan 27, 2025
View reviewed changes
plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseS3AndGlueMetastoreTest.java
try {
assertUpdate("CREATE TABLE " + tableName + " (a row(\"b:c\" int), b row(\":b::c::\" int), \"::\" row(\":::\" int))");
assertUpdate("INSERT INTO " + tableName + " VALUES (ROW(ROW(11), ROW(22), ROW(33)))", 1);
assertQuery("SELECT a.\"b:c\", b.\":b::c::\", \"::\".\":::\" FROM " + tableName, "VALUES (11, 22, 33)");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This above TypeInfoUtils class comes from Apache Hive code. https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoUtils.java

Is this table readable by Hive?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Query fails with the same error in Hive.

0: jdbc:hive2://localhost:10000/> CREATE TABLE colon_test(
. . . . . . . . . . . . . . . . >     foo struct<`a:bc`:int, b:string>
. . . . . . . . . . . . . . . . > );
INFO  : Compiling command(queryId=hive_20250128022546_8f5f188f-c6d1-4e8a-8262-311ef3b5759a): CREATE TABLE colon_test(
foo struct<`a:bc`:int, b:string>
)
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20250128022546_8f5f188f-c6d1-4e8a-8262-311ef3b5759a); Time taken: 0.009 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20250128022546_8f5f188f-c6d1-4e8a-8262-311ef3b5759a): CREATE TABLE colon_test(
foo struct<`a:bc`:int, b:string>
)
INFO  : Starting task [Stage-0:DDL] in serial mode
ERROR : Failed
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: Error: type expected at the position 9 of 'struct<a:bc:int,b:string>' but 'bc' is found.
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1383) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1388) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableNonReplaceMode(CreateTableOperation.java:158) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:116) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:84) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:354) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:327) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:244) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:105) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:367) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:205) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:185) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236) ~[hive-service-4.0.1.jar:4.0.1]
	at org.apache.hive.service.cli.operation.SQLOperation.access$500(SQLOperation.java:90) ~[hive-service-4.0.1.jar:4.0.1]
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336) ~[hive-service-4.0.1.jar:4.0.1]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_342]
	at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_342]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) ~[hadoop-common-3.3.6.jar:?]
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:356) ~[hive-service-4.0.1.jar:4.0.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_342]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_342]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342]
	at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342]
Caused by: java.lang.IllegalArgumentException: Error: type expected at the position 9 of 'struct<a:bc:int,b:string>' but 'bc' is found.
	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.expect(TypeInfoUtils.java:404) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.expect(TypeInfoUtils.java:387) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseType(TypeInfoUtils.java:448) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseType(TypeInfoUtils.java:545) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseTypeInfos(TypeInfoUtils.java:361) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils.getTypeInfosFromTypeString(TypeInfoUtils.java:851) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.AbstractSerDe.parseColumnTypes(AbstractSerDe.java:117) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.AbstractSerDe.parseColumnTypes(AbstractSerDe.java:110) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.parseColumnTypes(LazySimpleSerDe.java:143) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:84) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.initialize(AbstractEncodingAwareSerDe.java:43) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.initialize(LazySimpleSerDe.java:120) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:97) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:80) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:356) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:349) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(Table.java:266) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1320) ~[hive-exec-4.0.1.jar:4.0.1]
	... 27 more
ERROR : DDLTask failed, DDL Operation: class org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: Error: type expected at the position 9 of 'struct<a:bc:int,b:string>' but 'bc' is found.
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1383) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1388) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableNonReplaceMode(CreateTableOperation.java:158) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:116) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:84) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:354) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:327) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:244) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:105) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:367) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:205) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:185) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236) ~[hive-service-4.0.1.jar:4.0.1]
	at org.apache.hive.service.cli.operation.SQLOperation.access$500(SQLOperation.java:90) ~[hive-service-4.0.1.jar:4.0.1]
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336) ~[hive-service-4.0.1.jar:4.0.1]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_342]
	at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_342]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) ~[hadoop-common-3.3.6.jar:?]
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:356) ~[hive-service-4.0.1.jar:4.0.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_342]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_342]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342]
	at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342]
Caused by: java.lang.IllegalArgumentException: Error: type expected at the position 9 of 'struct<a:bc:int,b:string>' but 'bc' is found.
	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.expect(TypeInfoUtils.java:404) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.expect(TypeInfoUtils.java:387) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseType(TypeInfoUtils.java:448) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseType(TypeInfoUtils.java:545) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils$TypeInfoParser.parseTypeInfos(TypeInfoUtils.java:361) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.typeinfo.TypeInfoUtils.getTypeInfosFromTypeString(TypeInfoUtils.java:851) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.AbstractSerDe.parseColumnTypes(AbstractSerDe.java:117) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.AbstractSerDe.parseColumnTypes(AbstractSerDe.java:110) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.parseColumnTypes(LazySimpleSerDe.java:143) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:84) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.AbstractEncodingAwareSerDe.initialize(AbstractEncodingAwareSerDe.java:43) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.initialize(LazySimpleSerDe.java:120) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:97) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:80) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:356) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:349) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.metadata.Table.checkValidity(Table.java:266) ~[hive-exec-4.0.1.jar:4.0.1]
	at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1320) ~[hive-exec-4.0.1.jar:4.0.1]
	... 27 more
ERROR : FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.IllegalArgumentException: Error: type expected at the position 9 of 'struct<a:bc:int,b:string>' but 'bc' is found.
INFO  : Completed executing command(queryId=hive_20250128022546_8f5f188f-c6d1-4e8a-8262-311ef3b5759a); Time taken: 0.003 seconds
Error: Error while compiling statement: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. java.lang.IllegalArgumentException: Error: type expected at the position 9 of 'struct<a:bc:int,b:string>' but 'bc' is found. (state=08S01,code=40000)
0: jdbc:hive2://localhost:10000/>

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want to allow creating tables that the data source can't read.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ebyhr ebyhr ebyhr left review comments

Assignees
No one assigned
Labels
cla-signed hive Hive connector
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mayankvadariya @ebyhr