Skip to content

Commit 0311f44

Browse files
committed
[SPARK-54366][INFRA] Add free_disk_space step to K8s integration test GitHub Action job
### What changes were proposed in this pull request? This PR aims to add `free_disk_space` step to K8s integration test GitHub Action job. ### Why are the changes needed? The K8s integration test CI is flaky due to `No space left on device` error. - https://github.com/apache/spark/actions/runs/19354883389/job/55448531341 ``` [info] 25/11/14 21:27:02 ERROR TaskSchedulerImpl: Lost executor 4 on 10.244.0.67: Unable to create executor due to /var/data/spark-163899dd-08da-4b76-b71d-c428207a3bdf/spark-1e9d976f-69b3-4274-af03-300cfc4d6fd5/-14621403551763155568738_cache -> ./software.amazon.awssdk_bundle-2.29.52.jar: No space left on device ``` Like the other four GitHub Action jobs, `free_disk_space_container` will mitigate this situation in this job. **BEFORE** ``` $ git grep 'free_disk_space$' .github/workflows/build_and_test.yml: ./dev/free_disk_space .github/workflows/release.yml: ./dev/free_disk_space ``` **AFTER** ``` $ git grep 'free_disk_space$' .github/workflows/build_and_test.yml: ./dev/free_disk_space .github/workflows/build_and_test.yml: ./dev/free_disk_space .github/workflows/release.yml: ./dev/free_disk_space ``` ### Does this PR introduce _any_ user-facing change? No, this is a test infra change. ### How was this patch tested? Pass the CIs and check the log. The following is the log result of this PR. - https://github.com/dongjoon-hyun/spark/actions/runs/19395758483/job/55495933312 **BEFORE CLEANUP** ``` + df -h Filesystem Size Used Avail Use% Mounted on /dev/root 72G 54G 18G 76% / tmpfs 7.9G 84K 7.9G 1% /dev/shm tmpfs 3.2G 1.1M 3.2G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock /dev/sda16 881M 62M 758M 8% /boot /dev/sda15 105M 6.2M 99M 6% /boot/efi /dev/sdb1 74G 4.1G 66G 6% /mnt tmpfs 1.6G 12K 1.6G 1% /run/user/1001 ``` **AFTER CLEANUP** ``` + df -h Filesystem Size Used Avail Use% Mounted on /dev/root 72G 21G 52G 29% / tmpfs 7.9G 84K 7.9G 1% /dev/shm tmpfs 3.2G 1.1M 3.2G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock /dev/sda16 881M 62M 758M 8% /boot /dev/sda15 105M 6.2M 99M 6% /boot/efi /dev/sdb1 74G 4.1G 66G 6% /mnt tmpfs 1.6G 12K 1.6G 1% /run/user/1001 ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #53077 from dongjoon-hyun/SPARK-54366. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent d02a6d4 commit 0311f44

File tree

1 file changed

+5
-0
lines changed

1 file changed

+5
-0
lines changed

.github/workflows/build_and_test.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1325,6 +1325,11 @@ jobs:
13251325
key: k8s-integration-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
13261326
restore-keys: |
13271327
k8s-integration-coursier-
1328+
- name: Free up disk space
1329+
run: |
1330+
if [ -f ./dev/free_disk_space ]; then
1331+
./dev/free_disk_space
1332+
fi
13281333
- name: Install Java ${{ inputs.java }}
13291334
uses: actions/setup-java@v4
13301335
with:

0 commit comments

Comments
 (0)