[HUDI-9025] perf: improve append performance by reducing avro schema comparisons

[TheR1sing3un]

For engine specific record merger mode, i.e. spark record：
For each record, we need to use cpu time to do some unnecessary operation, such as schema comparison, even if we have a global avro schema -> spark schema cache, but for the cache.get() operation, It will eventually call the avro schema::equals method, which will go through all the columns to compare. In the append scenario, our cache will hit every time, but each record will need to compare the avro schema completely, which will waste a lot of cpu time.

In my perf result:

HoodieInternalRowUtils::getCachedSchema cost almost 10% cpu time during doAppend.
And then we can analyze the results and figure out that basically all the time in this function is used to compare avro schema with cache::key.

As the number of columns increases, the proportion of this consumption will be higher.

Change Logs

1. improve append performance by reducing comparisons

Impact

add a new interface for engine specific record for using engine specific schema rather than avro schema

Risk level (write none, low medium or high below)

low

Documentation Update

none

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[hudi-bot]

CI report:

• e0e89f2 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build