[HUDI-9058] Use InProcessLockProvider for flink single writer ingesti…

[cshuo]

…on with async table services

Change Logs

For Flink ingestion cases the concurrency mode is single writer and with async table services, use InProcessLockProvider by default.

Impact

better lock performance.

Risk level (write none, low medium or high below)

low

Documentation Update

none

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[cshuo]

@hudi-bot run azure

[hudi-bot]

CI report:

• db50145 Azure: FAILURE Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[CTTY]

Hi @cshuo , thanks for putting out the fix. I remember @danny0405 and I had the discussion on this: using Inprocess lock provider may not be safe because for flink tables services will run in a different process. We may need to be careful about it.

An alternative would be making filesystem based lock provider work on storage like s3 and will help us better. cc @yihua

[cshuo]

@CTTY thks for the context.

using Inprocess lock provider may not be safe because for flink tables services will run in a different process

So in this PR, only writing pipeline with async service will use InProcess lock. When users use dedicated compaction/cluster job, usually, async table services is disabled, then the lock will fallback to FS based lock provider. WDYT

[danny0405]

@CTTY @cshuo yeah, the best way is to solve fs lock provider issue on s3, the s3 file creation should be atomic already, @cshuo Can you take some time to research the fs lock provider.