GOAWAY issues with Ray on Conda #203
Description
Solution to issue cannot be found in the documentation.
- I checked the documentation.
Issue
In recent versions of Ray on Conda, a GOAWAY log message has appeared. The error is not present on Ray on PyPi.
Minimal repro:
import time
import ray
RAY_ADDRESS = "ray://<cluster_address>:80"
AUTH_TOKEN = "********"
ray_config = {
'address': f"{RAY_ADDRESS}",
'_metadata': [("authorization", f"Basic {AUTH_TOKEN}")],
'runtime_env': {
"log_to_driver": True,
"working_dir": "/home/ray"
}
}
ray.init(**ray_config)
@ray.remote
def trivial_task():
return True
@ray.remote
def read_local_file():
return open("hello.txt").read()
ray.get(trivial_task.remote())
ray.get(read_local_file.remote())
time.sleep(180)
Error logs:
/home/brentb/.conda/envs/ray/bin/python /home/brentb/.config/JetBrains/PyCharm2024.3/scratches/scratch_123.py
2025-01-31 10:06:20,432 INFO client_builder.py:244 -- Passing the following kwargs to ray.init() on the server: log_to_driver
SIGTERM handler is not set because current thread is not the main thread.
(pid=gcs_server) I0000 00:00:1738336294.337305 1666 chttp2_transport.cc:1182] ipv4:10.51.55.81:38939: Got goaway [2] err=UNAVAILABLE:GOAWAY received; Error code: 2; Debug Text: Cancelling all calls {created_time:"2025-01-31T10:11:34.337303243-05:00", http2_error:2, grpc_status:14}
This does not impact the operation/stability of Ray, but is leading to very noisy logs. This appears to be due to a difference in the grpc package between PyPi and Conda ray versions.
Installed packages
1 # packages in environment at /data01/conda/envs/ray-conda:
2 #
3 # Name Version Build Channel
4 _libgcc_mutex 0.1 conda_forge
5 _openmp_mutex 4.5 2_gnu
6 aiohappyeyeballs 2.4.6 pyhd8ed1ab_0
7 aiohttp 3.10.5 py311h9ecbd09_1
8 aiohttp-cors 0.7.0 pyhd8ed1ab_2
9 aiosignal 1.3.2 pyhd8ed1ab_0
10 annotated-types 0.7.0 pyhd8ed1ab_1
11 async-timeout 5.0.1 pyhd8ed1ab_1
12 attrs 25.1.0 pyh71513ae_0
13 brotli-python 1.1.0 py311hfdbb021_2
14 bzip2 1.0.8 h4bc722e_7
15 c-ares 1.34.4 hb9d3cd8_0
16 ca-certificates 2025.1.31 hbcca054_0
17 cachetools 5.5.1 pyhd8ed1ab_0
18 certifi 2025.1.31 pyhd8ed1ab_0
19 cffi 1.17.1 py311hf29c0ef_0
20 charset-normalizer 3.4.1 pyhd8ed1ab_0
21 click 8.1.8 pyh707e725_0
22 colorama 0.4.6 pyhd8ed1ab_1
23 colorful 0.5.6 pyhd8ed1ab_0
24 cpp-expected 1.1.0 hf52228f_0
25 cryptography 44.0.1 py311hafd3f86_0
26 distlib 0.3.9 pyhd8ed1ab_1
27 filelock 3.17.0 pyhd8ed1ab_0
28 fmt 11.0.2 h434a139_0
29 frozenlist 1.5.0 py311h2dc5d0c_1
30 google-api-core 2.24.1 pyhd8ed1ab_0
31 google-auth 2.38.0 pyhd8ed1ab_0
32 googleapis-common-protos 1.67.0 pyhd8ed1ab_0
33 grpcio 1.67.1 py311h9789449_1
34 h2 4.2.0 pyhd8ed1ab_0
35 hpack 4.1.0 pyhd8ed1ab_0
36 hyperframe 6.1.0 pyhd8ed1ab_0
37 idna 3.10 pyhd8ed1ab_1
38 importlib_resources 6.5.2 pyhd8ed1ab_0
39 jsonschema 4.23.0 pyhd8ed1ab_1
40 jsonschema-specifications 2024.10.1 pyhd8ed1ab_1
41 keyutils 1.6.1 h166bdaf_0
42 krb5 1.21.3 h659f571_0
43 ld_impl_linux-64 2.43 h712a8e2_2
44 libabseil 20240722.0 cxx17_hbbce691_4
45 libarchive 3.7.7 h4585015_3
46 libcurl 8.12.1 h332b0f4_0
47 libedit 3.1.20250104 pl5321h7949ede_0
48 libev 4.33 hd590300_2
49 libexpat 2.6.4 h5888daf_0
50 libffi 3.4.6 h2dba641_0
51 libgcc 14.2.0 h77fa898_1
52 libgcc-ng 14.2.0 h69a702a_1
53 libgomp 14.2.0 h77fa898_1
54 libgrpc 1.67.1 h25350d4_1
55 libiconv 1.18 h4ce23a2_0
56 liblzma 5.6.4 hb9d3cd8_0
57 libmamba 2.0.5 h49b8a8d_1
58 libnghttp2 1.64.0 h161d5f1_0
59 libnsl 2.0.1 hd590300_0
60 libprotobuf 5.28.3 h6128344_1
61 libre2-11 2024.07.02 hbbce691_2
62 libsolv 0.7.30 h3509ff9_0
63 libsqlite 3.48.0 hee588c1_1
64 libssh2 1.11.1 hf672d98_0
65 libstdcxx 14.2.0 hc0a3c3a_1
66 libstdcxx-ng 14.2.0 h4852527_1
67 libunwind 1.6.2 h9c3ff4c_0
68 libuuid 2.38.1 h0b41bf4_0
69 libxcrypt 4.4.36 hd590300_1
70 libxml2 2.13.5 h0d44e9d_1
71 libzlib 1.3.1 hb9d3cd8_2
72 lz4-c 1.10.0 h5888daf_1
73 lzo 2.10 hd590300_1001
74 mamba 2.0.5 h8871ed4_1
75 msgpack-python 1.1.0 py311hd18a35c_0
76 multidict 6.1.0 py311h2dc5d0c_2
77 ncurses 6.5 h2d0b736_3
78 nlohmann_json 3.11.3 he02047a_1
79 opencensus 0.11.3 pyhd8ed1ab_1
80 opencensus-context 0.1.3 py311h38be061_3
81 openssl 3.4.1 h7b32b05_0
82 packaging 24.2 pyhd8ed1ab_2
83 pip 25.0.1 pyh8b19718_0
84 pkgutil-resolve-name 1.3.10 pyhd8ed1ab_2
85 platformdirs 4.3.6 pyhd8ed1ab_1
86 prometheus_client 0.21.1 pyhd8ed1ab_0
87 proto-plus 1.26.0 pyhd8ed1ab_0
88 protobuf 5.28.3 py311hfdbb021_0
89 psutil 6.1.1 py311h9ecbd09_0
90 py-spy 0.4.0 h4c5a871_1
91 pyasn1 0.6.1 pyhd8ed1ab_2
92 pyasn1-modules 0.4.1 pyhd8ed1ab_1
93 pycparser 2.22 pyh29332c3_1
94 pydantic 2.10.6 pyh3cfb1c2_0
95 pydantic-core 2.27.2 py311h9e33e62_0
96 pyopenssl 25.0.0 pyhd8ed1ab_0
97 pysocks 1.7.1 pyha55dd90_7
98 python 3.11.11 h9e4cc4f_1_cpython
99 python_abi 3.11 5_cp311
100 pyu2f 0.1.5 pyhd8ed1ab_1
101 pyyaml 6.0.2 py311h2dc5d0c_2
102 ray-core 2.42.1 py311h4831e01_0 conda-forge
103 ray-default 2.42.1 py311ha0e88df_0 conda-forge
104 re2 2024.07.02 h9925aae_2
105 readline 8.2 h8228510_1
106 referencing 0.36.2 pyh29332c3_0
107 reproc 14.2.5.post0 hb9d3cd8_0
108 reproc-cpp 14.2.5.post0 h5888daf_0
109 requests 2.32.3 pyhd8ed1ab_1
110 rpds-py 0.22.3 py311h9e33e62_0
111 rsa 4.9 pyhd8ed1ab_1
112 setproctitle 1.3.4 py311h9ecbd09_0
113 setuptools 75.8.0 pyhff2d567_0
114 simdjson 3.11.6 h84d6215_0
115 six 1.17.0 pyhd8ed1ab_0
116 smart_open 7.1.0 pyhd8ed1ab_0
117 spdlog 1.15.1 hb29a8c4_0
118 tk 8.6.13 noxft_h4845f30_101
119 typing-extensions 4.12.2 hd8ed1ab_1
120 typing_extensions 4.12.2 pyha770c72_1
121 tzdata 2025a h78e105d_0
122 urllib3 2.3.0 pyhd8ed1ab_0
123 virtualenv 20.29.2 pyhd8ed1ab_0
124 wheel 0.45.1 pyhd8ed1ab_1
125 wrapt 1.17.2 py311h9ecbd09_0
126 yaml 0.2.5 h7f98852_2
127 yaml-cpp 0.8.0 h59595ed_0
128 yarl 1.9.4 py311h9ecbd09_1
129 zipp 3.21.0 pyhd8ed1ab_1
130 zstandard 0.23.0 py311hbc35293_1
131 zstd 1.5.6 ha6fb4c9_0Environment info
Info