Skip to content

Commit 9ff46a3

Browse files
zhouyifan279pan3793
authored andcommitted
[KYUUBI #4935] More than target num of executors may survive after FinalStageResourceManager did kill
### _Why are the changes needed?_ When FinalStageResourceManager chooses executors to be killed, it may add dead executors to the kill list. This will leave more than target num of executors survived and cause resource waste. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request Closes #4936 from zhouyifan279/kill-executor. Closes #4936 2aaa84c [zhouyifan279] [KYUUBI#4935][Improvement] More than target num of executors may survive after FinalStageResourceManager did kill Authored-by: zhouyifan279 <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
1 parent 787028e commit 9ff46a3

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

extensions/spark/kyuubi-extension-spark-3-3/src/main/scala/org/apache/spark/sql/FinalStageResourceManager.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -170,7 +170,7 @@ case class FinalStageResourceManager(session: SparkSession)
170170

171171
// Evict the rest executors according to the shuffle block size
172172
executorToBlockSize.toSeq.sortBy(_._2).foreach { case (id, _) =>
173-
if (executorIdsToKill.length < expectedNumExecutorToKill) {
173+
if (executorIdsToKill.length < expectedNumExecutorToKill && existedExecutors.contains(id)) {
174174
executorIdsToKill.append(id)
175175
}
176176
}

0 commit comments

Comments
 (0)