Commit 0311f44
committed
[SPARK-54366][INFRA] Add
### What changes were proposed in this pull request?
This PR aims to add `free_disk_space` step to K8s integration test GitHub Action job.
### Why are the changes needed?
The K8s integration test CI is flaky due to `No space left on device` error.
- https://github.com/apache/spark/actions/runs/19354883389/job/55448531341
```
[info] 25/11/14 21:27:02 ERROR TaskSchedulerImpl: Lost executor 4 on 10.244.0.67: Unable to create executor due to /var/data/spark-163899dd-08da-4b76-b71d-c428207a3bdf/spark-1e9d976f-69b3-4274-af03-300cfc4d6fd5/-14621403551763155568738_cache
-> ./software.amazon.awssdk_bundle-2.29.52.jar:
No space left on device
```
Like the other four GitHub Action jobs, `free_disk_space_container` will mitigate this situation in this job.
**BEFORE**
```
$ git grep 'free_disk_space$'
.github/workflows/build_and_test.yml: ./dev/free_disk_space
.github/workflows/release.yml: ./dev/free_disk_space
```
**AFTER**
```
$ git grep 'free_disk_space$'
.github/workflows/build_and_test.yml: ./dev/free_disk_space
.github/workflows/build_and_test.yml: ./dev/free_disk_space
.github/workflows/release.yml: ./dev/free_disk_space
```
### Does this PR introduce _any_ user-facing change?
No, this is a test infra change.
### How was this patch tested?
Pass the CIs and check the log. The following is the log result of this PR.
- https://github.com/dongjoon-hyun/spark/actions/runs/19395758483/job/55495933312
**BEFORE CLEANUP**
```
+ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 72G 54G 18G 76% /
tmpfs 7.9G 84K 7.9G 1% /dev/shm
tmpfs 3.2G 1.1M 3.2G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda16 881M 62M 758M 8% /boot
/dev/sda15 105M 6.2M 99M 6% /boot/efi
/dev/sdb1 74G 4.1G 66G 6% /mnt
tmpfs 1.6G 12K 1.6G 1% /run/user/1001
```
**AFTER CLEANUP**
```
+ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 72G 21G 52G 29% /
tmpfs 7.9G 84K 7.9G 1% /dev/shm
tmpfs 3.2G 1.1M 3.2G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda16 881M 62M 758M 8% /boot
/dev/sda15 105M 6.2M 99M 6% /boot/efi
/dev/sdb1 74G 4.1G 66G 6% /mnt
tmpfs 1.6G 12K 1.6G 1% /run/user/1001
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #53077 from dongjoon-hyun/SPARK-54366.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>free_disk_space step to K8s integration test GitHub Action job1 parent d02a6d4 commit 0311f44
1 file changed
+5
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1325 | 1325 | | |
1326 | 1326 | | |
1327 | 1327 | | |
| 1328 | + | |
| 1329 | + | |
| 1330 | + | |
| 1331 | + | |
| 1332 | + | |
1328 | 1333 | | |
1329 | 1334 | | |
1330 | 1335 | | |
| |||
0 commit comments