You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[KYUUBI #4935] More than target num of executors may survive after FinalStageResourceManager did kill
### _Why are the changes needed?_
When FinalStageResourceManager chooses executors to be killed, it may add dead executors to the kill list.
This will leave more than target num of executors survived and cause resource waste.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes#4936 from zhouyifan279/kill-executor.
Closes#49362aaa84c [zhouyifan279] [KYUUBI#4935][Improvement] More than target num of executors may survive after FinalStageResourceManager did kill
Authored-by: zhouyifan279 <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
Copy file name to clipboardExpand all lines: extensions/spark/kyuubi-extension-spark-3-3/src/main/scala/org/apache/spark/sql/FinalStageResourceManager.scala
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -170,7 +170,7 @@ case class FinalStageResourceManager(session: SparkSession)
170
170
171
171
// Evict the rest executors according to the shuffle block size
172
172
executorToBlockSize.toSeq.sortBy(_._2).foreach { case (id, _) =>
173
-
if (executorIdsToKill.length < expectedNumExecutorToKill) {
173
+
if (executorIdsToKill.length < expectedNumExecutorToKill&& existedExecutors.contains(id)) {
0 commit comments