[CI] Upgrade test frameworks

[EnricoMi]

Upgrades:

• torch to 2.0.1 and 2.1.2
• tensorflow 2.13.0 and 2.14.0
• spark 3.5.0

[github-actions]

Unit Test Results

   522 files   -   202    522 suites   - 202   5h 57m 35s ⏱️ - 2h 8m 32s
   887 tests ±    0    639 ✅  -   129    195 💤 + 76   47 ❌ + 47   6 🔥 + 6 
11 590 runs   - 4 619  7 482 ✅  - 3 871  3 966 💤  - 890  130 ❌ +130  12 🔥 +12 

For more details on these failures and errors, see this check.

Results for commit e11592b. ± Comparison against base commit 9f88e1d.

This pull request skips 76 tests.

test.parallel.test_mxnet1.MX1Tests ‑ test_gluon_trainer
test.parallel.test_mxnet1.MX1Tests ‑ test_gpu_required
test.parallel.test_mxnet1.MX1Tests ‑ test_horovod_allreduce_cpu_gpu_error
test.parallel.test_mxnet1.MX1Tests ‑ test_horovod_grouped_allgather_cpu_gpu_error
test.parallel.test_mxnet1.MX1Tests ‑ test_horovod_grouped_allreduce_cpu_gpu_error
test.parallel.test_tensorflow.TensorFlowTests ‑ test_gpu_required
test.parallel.test_tensorflow.TensorFlowTests ‑ test_horovod_allgather_fused_gpu
test.parallel.test_tensorflow.TensorFlowTests ‑ test_horovod_allgather_gpu
test.parallel.test_tensorflow.TensorFlowTests ‑ test_horovod_allgather_grad_gpu
test.parallel.test_tensorflow.TensorFlowTests ‑ test_horovod_allgather_variable_size_fused_gpu
…

♻️ This comment has been updated with latest results.

[github-actions]

Unit Test Results (with flaky tests)

   874 files   -    14     874 suites   - 14   11h 20m 57s ⏱️ + 2h 22m 16s
   887 tests ±    0     639 ✅  -   129    195 💤 + 76   47 ❌ + 47   6 🔥 + 6 
19 064 runs   - 1 175  12 072 ✅  - 1 717  6 566 💤 +116  390 ❌ +390  36 🔥 +36 

For more details on these failures and errors, see this check.

Results for commit e11592b. ± Comparison against base commit 9f88e1d.

This pull request skips 76 tests.

test.parallel.test_mxnet1.MX1Tests ‑ test_gluon_trainer
test.parallel.test_mxnet1.MX1Tests ‑ test_gpu_required
test.parallel.test_mxnet1.MX1Tests ‑ test_horovod_allreduce_cpu_gpu_error
test.parallel.test_mxnet1.MX1Tests ‑ test_horovod_grouped_allgather_cpu_gpu_error
test.parallel.test_mxnet1.MX1Tests ‑ test_horovod_grouped_allreduce_cpu_gpu_error
test.parallel.test_tensorflow.TensorFlowTests ‑ test_gpu_required
test.parallel.test_tensorflow.TensorFlowTests ‑ test_horovod_allgather_fused_gpu
test.parallel.test_tensorflow.TensorFlowTests ‑ test_horovod_allgather_gpu
test.parallel.test_tensorflow.TensorFlowTests ‑ test_horovod_allgather_grad_gpu
test.parallel.test_tensorflow.TensorFlowTests ‑ test_horovod_allgather_variable_size_fused_gpu
…

♻️ This comment has been updated with latest results.

[EnricoMi]

@liangz1 @irasit @WeichenXu123 @chongxiaoc There is an issue with test test_dbfs_local_store in test/integration/test_spark.py when run against tensorflow 2.13. Can one of you have a look, please?

Here is a log of the failure:
https://github.com/horovod/horovod/actions/runs/6097139901/job/16544192396#step:207:168-176

[chongxiaoc]

hmm, this is specific logic from databricks about model serialization and deserialization.
Not familiar with this.

[EnricoMi]

Not familiar with this.

No worries, thanks for the response!

[EnricoMi]

Seeing std::bad_alloc when calling into horovod.torch.init():

[0]<stderr>:terminate called after throwing an instance of 'std::bad_alloc'
[0]<stderr>:  what():  std::bad_alloc

[thomas-bouvier]

I can't reproduce the bad alloc locally when compiling against PyTorch 2.1.0. Could I have more information about your environment? Was error triggered in the test pipeline?

[EnricoMi]

That can be seen in the CI: https://github.com/horovod/horovod/actions/runs/7364310753/job/20044572743#step:38:30

[0]<stderr>:terminate called after throwing an instance of 'std::bad_alloc'
[0]<stderr>:  what():  std::bad_alloc

As well as in the test results: See test.integration.test_interactiverun.InteractiveRunTests.

To reproduce:

git remote add upstream https://github.com/horovod/horovod.git
git fetch upstream
git checkout upstream/branch-ci-upgrade-test-frameworks-9
docker compose -f docker-compose.test.yml build test-cpu-gloo-py3_9-tf2_14_0-keras2_14_0-torch2_1_2-mxnet1_9_1-pyspark3_4_0
docker run --rm -it horovod-test-cpu-gloo-py3_9-tf2_14_0-keras2_14_0-torch2_1_2-mxnet1_9_1-pyspark3_4_0 /bin/bash -c " cd /horovod/test/parallel && (ls -1 test_*.py | xargs -n 1 horovodrun -np 2 -H localhost:2 --gloo /bin/bash /pytest.sh gloo)"

[EnricoMi]

I can't reproduce the bad alloc locally when compiling against PyTorch 2.1.0.

Have you compiled together with Gloo support? That might be related.

[thomas-bouvier]

I could reproduce the bad_alloc using the following:

docker run --rm -it horovod-test-cpu-gloo-py3_9-tf2_14_0-keras2_14_0-torch2_1_2-mxnet1_9_1-pyspark3_4_0 /bin/bash -c " cd /horovod/test/parallel && (ls -1 test_torch.py | xargs -n 1 horovodrun -np 1 -H localhost:1 --gloo /bin/bash /pytest.sh gloo)"

I can't identify the cause for now.

[EnricoMi]

Earlier you said you were able to compile against PyTorch 2.1.0 and run the tests successfully: #3979 (comment)

How was the setup different to the bad_alloc setup?

[thomas-bouvier]

I initially compiled Horovod myself using my own scripts. There are probably too many differences with the Docker image, so I propose to rely on the Docker recipe instead.

I spent some time debugging the bad malloc and it seems related to gloo indeed. Specifically, the device creation here, with the bad malloc being triggered here on the gloo side. Still investigating to understand why is that!

[EnricoMi]

May this be related to me incorrectly trying to compile Gloo with C++17: e11592b

[EnricoMi]

I spent some time debugging the bad malloc and it seems related to gloo indeed. Specifically, the device creation here, with the bad malloc being triggered here on the gloo side. Still investigating to understand why is that!

@maxhgerlach any idea from a quick look at these spots what this might be related to?

[EnricoMi]

@thomas-bouvier do you have any new findings regarding this bad_alloc?

[thomas-bouvier]

I couldn't make any progress since last time. Feel free to investigate on your side if you have some time. I will get back to it eventually.