fix: Handle missing valueType column after upgrade

[linliu-code]

Describe the issue this Pull Request addresses

#14108

Column valueType is added to ColumnStats record in recent change in master branch (1.1.x version), as causes backward compatibility issue since old ColumnStats records do not have this column. That is, when MDT tries to access this column from these old records say from 1.0.2 version, they would throw exception saying org.apache.avro.AvroRuntimeException: Not a valid schema field: valueType.

The reason that these old record would use their writer schema is because they are from log files of MDT that contains their writer schema in the log block header. When these log records are deserialized, the old writer schema is used.

Summary and Changelog

The solution is to check if valueType column exists in current records before access. If not exists, we return V1EmptyMetadata object instead.

I see we also return V1EmptyMetadata object when the value of valueType is NULL. Therefore, after V1EmptyMetadata is returned, the sequential operations should be handled properly.

Impact

Fix one compatibility issue after upgrade to version 9.

Risk Level

Low.

Documentation Update

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[nsivabalan]

Can we add a test fixture based test?

[hudi-bot]

CI report:

• c3891e7 UNKNOWN
• a059a11 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build