[rapids] removed spark tests, updated to a more recent rapids release #1219

cjac · 2024-08-08T16:36:32Z

Tested with CUDA=11 and CUDA=12

cjac · 2024-08-08T16:37:21Z

I prefer this to #1218

cjac · 2024-08-09T01:09:54Z

/gcbrun

cjac · 2024-08-09T04:20:34Z

/gcbrun

cjac · 2024-08-09T04:53:29Z

/gcbrun

prince-cs · 2024-08-09T05:13:59Z

/gcbrun

prince-cs · 2024-08-09T05:27:40Z

Should we increase the machine type from n1-standard-4 to n1-standard-16

cjac · 2024-08-09T05:40:25Z

cuda11 has been manually tested with all versions.
dataproc 2.0 images all pass the automated tests and can be assumed to work with cuda12 as well
Trying cuda12 on 2.1 and 2.2 now.

cjac · 2024-08-09T06:49:32Z

/gcbrun

cjac · 2024-08-09T15:29:17Z

/gcbrun

cjac · 2024-08-09T16:20:29Z

tests are failing for

2.1-debian11
2.1-rocky8
2.1-ubuntu20
2.2-debian12
2.2-rocky9
2.2-ubuntu22

cjac · 2024-08-09T22:59:01Z

/gcbrun

cjac · 2024-08-10T00:44:07Z

/gcbrun

cjac · 2024-08-10T02:26:29Z

[edit: this was a misconfiguration in the systemd unit]

It looks like the dask infrastructure is out of date and I'll have to target 2023.12 instead.

root@cluster-1718310842-m:~# /opt/conda/miniconda3/envs/dask/bin/python /tmp/init/dask/verify_dask_standalone.py 
/opt/conda/miniconda3/envs/dask/lib/python3.11/site-packages/distributed/client.py:1394: VersionMismatchWarning: Mismatched versions found

+-------------+----------------+----------------+---------+
| Package     | Client         | Scheduler      | Workers |
+-------------+----------------+----------------+---------+
| dask        | 2024.6.2       | 2023.12.1      | None    |
| distributed | 2024.6.2       | 2023.12.1      | None    |
| python      | 3.11.9.final.0 | 3.11.8.final.0 | None    |
| tornado     | 6.4.1          | 6.3.3          | None    |
+-------------+----------------+----------------+---------+

cjac · 2024-08-10T02:38:24Z

I also need to reduce the python abi to 3.10

cjac · 2024-08-10T04:45:24Z

/gcbrun

cjac · 2024-08-10T20:06:18Z

/gcbrun

cjac · 2024-08-10T23:53:26Z

/gcbrun

cjac · 2024-08-11T00:52:47Z

/gcbrun

cjac · 2024-08-11T01:04:56Z

/gcbrun

cjac · 2024-08-11T01:23:47Z

/gcbrun

cjac · 2024-08-11T01:32:44Z

/gcbrun

cjac · 2024-10-25T22:16:35Z

All versions aside from 2.2-debian12 succeeded.

Build ID: 36ac2daa-1c2d-483c-8f0b-b3359fee06b9

Commit: e8a44fe

2.2-debian12 failed because of a conda mirror failure. I've opened conda/infrastructure#1051 to track. It was running a dask install, which it shouldn't have been doing. So I've removed the delta from master of the dask/ directory

cjac · 2024-10-25T22:16:45Z

/gcbrun

cjac · 2024-10-25T22:45:14Z

/gcbrun

cjac · 2024-10-26T00:02:48Z

/gcbrun

cjac · 2024-10-26T00:58:18Z

/gcbrun

cjac · 2024-10-26T01:57:02Z

Ach! Finally!

jakirkham · 2024-10-26T02:09:16Z

Hooray! 🥳

Thank you for your persistence here CJ! 🙏 👏

cjac · 2024-10-26T02:15:04Z

Hooray! 🥳

Thank you for your persistence here CJ! 🙏 👏

Thanks for noticing!

This reminds me of a tool song. I feel like an excess of optimism is the only thing that keeps me going sometimes ;-)

cjac · 2024-10-26T03:16:51Z

okay, result of trying dask>=2024.8 instead of dask>=2024.5:

+ /opt/conda/miniconda3/bin/mamba create -m -n dask-rapids -y --no-channel-priority -c conda-forge -c nvidia -c rapidsai 'cuda-version>=12,<13' rapids=24.08 'dask>=2024.8' dask-bigquery dask-ml dask-sql cudf numba 'python>=3.11'

Looking for: ["cuda-version[version='>=12,<13']", 'rapids=24.08', "dask[version='>=2024.8']", 'dask-bigquery', 'dask-ml', 'dask-sql', 'cudf', 'numba', "python[version='>=3.11']"]

Encountered problems while solving:
  - package rapids-24.08.00-cuda11_py310_240808_g86654f0_0 requires custreamz 24.08.*, but none of the providers can be installed
  
  
 + /opt/conda/miniconda3/bin/conda create -m -n dask-rapids -y --no-channel-priority -c conda-forge -c nvidia -c rapidsai 'cuda-version>=12,<13' rapids=24.08 'dask>=2024.8' dask-bigquery dask-ml dask-sql cudf numba 'python>=3.11'

Solving environment: ...working... failed

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package rapids-24.08.00-cuda11_py310_240808_g86654f0_0 requires custreamz 24.08.*, but none of the providers can be installed

Could not solve for environment specs
The following packages are incompatible
├─ dask >=2024.8  is requested and can be installed;                                                                                                                                                
└─ rapids 24.08**  is not installable because it requires
   └─ custreamz 24.08.* , which requires
      └─ rapids-dask-dependency 24.08.* , which requires
         └─ dask 2024.7.1 , which conflicts with any installable versions previously reported.

cjac · 2024-10-26T05:05:30Z

I'll use "rapids>=${RAPIDS_VERSION}" instead of "rapids=${RAPIDS_VERSION}" as the rapids_spec ; that way we'll pick up a newer version if it can be solved for. Nothing newer than 24.08 for rapids, but dask picked up 2024.7 rather than 2024.5, so I'll encode that into the current state of the patch.

… dask>=2024.7 ; correctly capturing retval of installer program

cjac · 2024-10-26T05:09:49Z

/gcbrun

cjac · 2024-10-26T06:07:19Z

Two in a row!

Deependra-Patel

I am not able to fully understand the PR tbh, there are too many changes clubbed together without reasoning why we changed the behavior, PR description is not helping either. We should have split this up.

Deependra-Patel · 2024-11-05T04:43:59Z

cloudbuild/presubmit.sh

@@ -70,6 +70,7 @@ determine_tests_to_run() {
    changed_dir="${changed_dir%%/*}/"
    # Run all tests if common directories modified
    if [[ ${changed_dir} =~ ^(integration_tests|util|cloudbuild)/$ ]]; then
+      continue # remove this before squash/merge


Deependra-Patel · 2024-11-05T04:47:18Z

rapids/test_rapids.py

-        boot_disk_size="200GB",
-        timeout_in_minutes=30)
+        boot_disk_size="50GB",
+        timeout_in_minutes=60)


This is too long timeout, is 30 mins not enough for a cluster to be up?

Yes, it takes much time as rapids requires gpu driver as well. So, in order to get rapids up and running we first need to install gpu drivers and that takes up quite some time.

Deependra-Patel · 2024-11-05T04:49:35Z

rapids/test_rapids.py

-                            ("STANDARD", ["m"], GPU_T4, "standalone"))
+
+  @parameterized.parameters(
+# If a new version of dask-yarn is released, add this test back in.


Are these not supported anymore? Also, let's update the init actions README as well.

Deependra-Patel · 2024-11-05T04:50:14Z

rapids/run-bazel-tests.sh

+if [[ -z "${IMAGE_VERSION}" ]] ; then
+       IMAGE_VERSION="$(jq -r .IMAGE_VERSION        env.json)" ; fi ; export IMAGE_VERSION
+
+#declare -a TESTS_TO_RUN=('dask:test_dask' 'rapids:test_rapids')


Dead code, let's remove this

Deependra-Patel · 2024-11-05T04:51:45Z

rapids/rapids.sh

  if [[ "${ROLE}" == "Master" ]]; then
    systemctl restart hadoop-yarn-resourcemanager.service
    # Restart NodeManager on Master as well if this is a single-node-cluster.
-    if systemctl status hadoop-yarn-nodemanager; then
+    if systemctl list-units | grep hadoop-yarn-nodemanager; then


Curious: Why was this change needed?

Deependra-Patel · 2024-11-05T04:57:53Z

rapids/bazel.screenrc

@@ -0,0 +1,17 @@
+#


Are these files committed by mistake?

Deependra-Patel · 2024-11-05T04:59:26Z

rapids/rapids.sh

-      local python_ver="3.10"
-  else
-      local python_ver="3.9"
+function configure_knox_for_dask() {


Why do we need to configure knox now, earlier it was not needed?

cjac requested a review from prince-cs August 8, 2024 16:36

cjac mentioned this pull request Aug 8, 2024

Fixing rapids test for presubmits #1218

Closed

cjac marked this pull request as draft August 8, 2024 16:37

cjac mentioned this pull request Aug 8, 2024

Parallel build support conda/conda-build#5449

Open

2 tasks

cjac changed the title ~~[rapids] tested with 24.06~~ [rapids] tested with CUDA11+22.08 and CUDA12+24.06 Aug 9, 2024

cjac changed the title ~~[rapids] tested with CUDA11+22.08 and CUDA12+24.06~~ [rapids] tested with CUDA11+22.06 and CUDA12+24.06 Aug 9, 2024

cjac changed the title ~~[rapids] tested with CUDA11+22.06 and CUDA12+24.06~~ [rapids] tested with CUDA11+22.06 and CUDA12+23.12 Aug 10, 2024

cjac changed the title ~~[rapids] tested with CUDA11+22.06 and CUDA12+23.12~~ [rapids,dask] tested with CUDA11+22.06 and CUDA12+23.12 Aug 10, 2024

cjac force-pushed the rapids-20240806 branch from d987ab1 to db71b01 Compare August 11, 2024 01:41

cjac added 2 commits October 25, 2024 15:44

Thread.yield()

5b93e3a

improved documentation

38ba6e3

default to non-private ip ; maybe that is why this last run failed

91907ae

revert dataproc_test_case.py to last known good

6d8c32b

using correct df command ; using greater or equal to rapids version ;…

7c8ce57

… dask>=2024.7 ; correctly capturing retval of installer program

cjac marked this pull request as ready for review October 26, 2024 06:39

cjac requested review from medb, prince-cs, davidrabinowitz, cnauroth, Deependra-Patel, abmodi, bradmiro, jayadeep-jayaraman and kuldeepkk-dev October 26, 2024 06:40

prince-cs approved these changes Oct 26, 2024

View reviewed changes

cjac merged commit 90b1bd1 into GoogleCloudDataproc:master Oct 26, 2024
2 checks passed

Deependra-Patel reviewed Nov 5, 2024

View reviewed changes

[rapids] removed spark tests, updated to a more recent rapids release #1219

[rapids] removed spark tests, updated to a more recent rapids release #1219

Conversation

cjac commented Aug 8, 2024 • edited Loading

cjac commented Aug 8, 2024

cjac commented Aug 9, 2024

cjac commented Aug 9, 2024

cjac commented Aug 9, 2024

prince-cs commented Aug 9, 2024

prince-cs commented Aug 9, 2024

cjac commented Aug 9, 2024 • edited Loading

cjac commented Aug 9, 2024

cjac commented Aug 9, 2024

cjac commented Aug 9, 2024

cjac commented Aug 9, 2024

cjac commented Aug 10, 2024

cjac commented Aug 10, 2024 • edited Loading

cjac commented Aug 10, 2024

cjac commented Aug 10, 2024

cjac commented Aug 10, 2024

cjac commented Aug 10, 2024

cjac commented Aug 11, 2024

cjac commented Aug 11, 2024

cjac commented Aug 11, 2024

cjac commented Aug 11, 2024

cjac commented Oct 25, 2024 • edited Loading

cjac commented Oct 25, 2024

cjac commented Oct 25, 2024

cjac commented Oct 26, 2024

cjac commented Oct 26, 2024

cjac commented Oct 26, 2024

jakirkham commented Oct 26, 2024

cjac commented Oct 26, 2024 • edited Loading

cjac commented Oct 26, 2024 • edited Loading

cjac commented Oct 26, 2024

cjac commented Oct 26, 2024

cjac commented Oct 26, 2024

Deependra-Patel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjac commented Aug 8, 2024 •

edited

Loading

cjac commented Aug 9, 2024 •

edited

Loading

cjac commented Aug 10, 2024 •

edited

Loading

cjac commented Oct 25, 2024 •

edited

Loading

cjac commented Oct 26, 2024 •

edited

Loading

cjac commented Oct 26, 2024 •

edited

Loading