RFC-80: Updates for class structure and discussion items #14092
Conversation
…ion, update some terminology to match 1.0 classes
|
|
||
| ## Background | ||
| Currently, Hudi organizes data according to fileGroup granularity. The fileGroup is further divided into column clusters to introduce the columngroup concept. | ||
| Currently, Hudi organizes data according to fileGroup granularity. The fileGroup is further divided into column clusters to introduce the columngroup concept. Within a ColumnGroup, there is a ColumnSegment that allows for multiple file slices for the same column group to prevent large files. Inside each ColumnSegment, there are one or more FileSlices. |
There was a problem hiding this comment.
does RLI now point to a filgroup/colsegment
| - Base file: [file_id]\_[write_token]\_[begin_time][_cfName].[extension] | ||
| - Log file: [file_id]\_[begin_instant_time][_cfName].log.[version]_[write_token] | ||
| - Base file: [file_id]\_[write_token]\_[begin_time][_cgName_cgSegment].[extension] | ||
| - Log file: [file_id]\_[begin_instant_time][_cgName_cgSegment].log.[version]_[write_token] |
There was a problem hiding this comment.
cgName, cgSegment will be autogenerated by Hudi, right?
are these uuids..
There was a problem hiding this comment.
Just noticed this, given that begin_time is all digits, if we stick to the convention of begin_time being at the end of the file, but before the extension.
Since begin_time is a fixed length string, i feel it's more useful to put fixed length details before an extension. Reason being that one can just delimit by the period, then move forward N characters.
I assume cgName_cgSegment is variable length, this might make extracting begin_time harder in the future where users have to fall back to regex instead of just using a few keyboard shortcuts.
4 participants
Describe the issue this Pull Request addresses
Adds updates to allow multiple file slices for Column Group to control file sizing, also raises new questions for discussion.
Summary and Changelog
Impact
Risk Level
Documentation Update
Contributor's checklist