fix: the problem that the pending tasks cannot be scheduled during the backfill action

[hansongChina]

What type of PR is this?

What this PR does / why we need it:

During the backfill action, in the loop of pending tasks, if a previous task fails to match a node or an exception occurs when calling the PrePredicateFn method, all subsequent tasks will stop being scheduled.

Which issue(s) this PR fixes:

Fixes #4028

Special notes for your reviewer:

Does this PR introduce a user-facing change?

[hwdef]

/lgtm

[hwdef]

/ok-to-test

[volcano-sh-bot]

New changes are detected. LGTM label has been removed.

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign hwdef
You can assign the PR to them by writing /assign @hwdef in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• pkg/scheduler/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[JesseStutler]

Great. But I'm wondering isn't there also a problem in the allocate action? After the break here:

volcano/pkg/scheduler/actions/allocate/allocate.go

Line 215 in 5d66534

break

, the job is not pushed back, and the remaining pods in this job are not scheduled

cc @Monokaix @lowang-bh @hwdef

[lowang-bh]

Great. But I'm wondering isn't there also a problem in the allocate action? After the break here:

volcano/pkg/scheduler/actions/allocate/allocate.go

Line 215 in 5d66534

break

, the job is not pushed back, and the remaining pods in this job are not scheduled
cc @Monokaix @lowang-bh @hwdef

It is ok when minMember equals to total replicas.
Otherwise, maybe it need to be considered as this:

volcano/pkg/scheduler/actions/allocate/allocate.go

Line 224 in a61fc90

if job.NeedContinueAllocating() {