[Enhancement] [Refactor] Add TaskRunFIFOQueue to optimize task scheduler (backport #45241)

[mergify]

Why I'm doing:

• Pending task runs are stored in pendingTaskRunMap and pendingTaskRunQueue separately and it's not good for maintain/interact.

    // TODO: Refactor this to find a better way to store the task runs.
    // taskId -> pending TaskRun Queue, for each Task only support 1 running taskRun currently,
    // so the map value is priority queue need to be sorted by priority from large to small
    @SerializedName("pendingTaskRunMap")
    private final Map<Long, Queue<TaskRun>> pendingTaskRunMap = Maps.newConcurrentMap();

    private final TaskRunFIFOQueue pendingTaskRunQueue = new TaskRunFIFOQueue();

What I'm doing:

• Refactor pending task runs into TaskRunFIFOQueue class.
• Use an ordered set for better poll and extension.

/**
 * A thread safe FIFO queue for pending task runs, it uses an ordered set and hash map to schedule task runs.
 * - ordered set is used to sort task runs by priority and created time.
 * - hash map is used to store task runs by task id.
 */

Further

• Support more task properties, eg: event trigger delay perioid/task priority/task retry attempts.

Fixes #issue

What type of PR is this:

• BugFix
• Feature
• Enhancement
• Refactor
• UT
• Doc
• Tool

Does this PR entail a change in behavior?

• Yes, this PR will result in a change in behavior.
• No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• Interface/UI changes: syntax, type conversion, expression evaluation, display information
• Parameter changes: default values, similar parameters but with different default values
• Policy changes: use new policy to replace old one, functionality automatically enabled
• Feature removed
• Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• I have added test cases for my bug fix or my new feature
• This pr needs user documentation (for new or modified features or behaviors)
  • I have added documentation for my new feature or new function
• This is a backport pr

Bugfix cherry-pick branch check:

• I have checked the version labels which the pr will be auto-backported to the target branch
  • 3.3
  • 3.2
  • 3.1
  • 3.0
  • 2.5

---

This is an automatic backport of pull request #45241 done by [Mergify](https://mergify.com). ## Why I'm doing: - Pending task runs are stored in `pendingTaskRunMap` and `pendingTaskRunQueue ` separately and it's not good for maintain/interact. ``` // TODO: Refactor this to find a better way to store the task runs. // taskId -> pending TaskRun Queue, for each Task only support 1 running taskRun currently, // so the map value is priority queue need to be sorted by priority from large to small @SerializedName("pendingTaskRunMap") private final Map> pendingTaskRunMap = Maps.newConcurrentMap();

private final TaskRunFIFOQueue pendingTaskRunQueue = new TaskRunFIFOQueue();

## What I'm doing:
- Refactor pending task runs into TaskRunFIFOQueue class.
- Use an ordered set for better `poll` and extension.

/**

• A thread safe FIFO queue for pending task runs, it uses an ordered set and hash map to schedule task runs.
• • ordered set is used to sort task runs by priority and created time.
• • hash map is used to store task runs by task id.
    */

## Further
- Support more task properties, eg: event trigger delay perioid/task priority/task retry attempts.

Fixes #issue

## What type of PR is this:

- [ ] BugFix
- [ ] Feature
- [x] Enhancement
- [ ] Refactor
- [ ] UT
- [ ] Doc
- [ ] Tool

Does this PR entail a change in behavior?

- [ ] Yes, this PR will result in a change in behavior.
- [x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

- [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
- [ ] Parameter changes: default values, similar parameters but with different default values
- [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
- [ ] Feature removed
- [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

## Checklist:

- [x] I have added test cases for my bug fix or my new feature
- [ ] This pr needs user documentation (for new or modified features or behaviors)
  - [ ] I have added documentation for my new feature or new function
- [x] This is a backport pr

[mergify]

Cherry-pick of 5dd0ed3 has failed:

On branch mergify/bp/branch-3.2/pr-45241
Your branch is up to date with 'origin/branch-3.2'.

You are currently cherry-picking commit 5dd0ed391b.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   fe/fe-core/src/main/java/com/starrocks/scheduler/TaskRunFIFOQueue.java
	new file:   fe/fe-core/src/test/java/com/starrocks/scheduler/TaskRunFIFOQueueTest.java

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   fe/fe-core/src/main/java/com/starrocks/scheduler/TaskRun.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/scheduler/TaskRunManager.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/scheduler/TaskRunScheduler.java
	both modified:   fe/fe-core/src/test/java/com/starrocks/scheduler/TaskRunSchedulerTest.java

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[mergify]

@mergify[bot]: Backport conflict, please reslove the conflict and resubmit the pr

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ LiShuMing
❌ mergify[bot]
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud