How could I know which payload class is using for record combine

[bithw1]

I have created a hudi table using following spark sql, but when I look at the .hoodie.properties or run show create table, there is no payload class there,

How could I know which payload class this table is using when doing record combine, i want to know the combine logic when I am using the table.

CREATE TABLE IF NOT EXISTS hudi_cow_15 (
  a INT,
  b INT,
  c INT
) 

USING hudi

tblproperties(
type='cow',
primaryKey='a',
preCombineField='c'
)

[deepakpanda93]

Hello @bithw1

For Spark SQL tables like yours, Hudi will use the default payload / record-merger configuration if you don’t explicitly set one in TBLPROPERTIES or writer options.

• The default Spark payload class is org.apache.hudi.common.model.OverwriteWithLatestAvroPayload when hoodie.datasource.write.payload.class is not set.
• This payload “picks the record with the greatest value (determined by calling .compareTo() on the value of precombine key) to break ties and simply picks the latest record while merging” – i.e. latest-write-wins based on your preCombineField (c in your table).

So for your table:

CREATE TABLE IF NOT EXISTS hudi_cow_15 (
  a INT,
  b INT,
  c INT
) USING hudi
TBLPROPERTIES(
  type='cow',
  primaryKey='a',
  preCombineField='c'
)

• Record key: a
• Ordering / preCombine field: c
• Combine logic (by default): among multiple records with the same a, the one with the largest c wins; that full record overwrites the previous one.

Using OverwriteWithLatestAvroPayload, Hudi performs:

if incomingRecord.preCombineField > existingRecord.preCombineField:
    keep incoming record
else:
    keep existing record

For your schema:

record with highest value of c overwrites others

If the user does not set it, Hudi does not write it into .hoodie.properties, Hudi automatically falls back to its default payload class

[bithw1]

Thanks @deepakpanda93 for the detailed explanation!

[bithw1]

@deepakpanda93 , I am using Hudi 0.15, and tried with cow

1. I tried the payload class org.apache.hudi.common.model.OverwriteWithLatestAvroPayload, new incoming record will overwrite old one(in storage),even the new incoming record's precombine value is less than the old one's
2. It looks to me the default paly class is DefaultHoodieRecordPayload, which will do the precombineField comparation

  public static final ConfigProperty<String> WRITE_PAYLOAD_CLASS_NAME = ConfigProperty
      .key("hoodie.datasource.write.payload.class")
      .defaultValue(DefaultHoodieRecordPayload.class.getName())
      .markAdvanced()
      .withDocumentation("Payload class used. Override this, if you like to roll your own merge logic, when upserting/inserting. "
          + "This will render any value set for PRECOMBINE_FIELD_OPT_VAL in-effective");