Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intermittent "HTTP 502 BAD GATEWAY" error for repo.anaconda.com #825

Open
2 tasks done
l0b0 opened this issue Sep 22, 2023 · 6 comments
Open
2 tasks done

Intermittent "HTTP 502 BAD GATEWAY" error for repo.anaconda.com #825

l0b0 opened this issue Sep 22, 2023 · 6 comments
Labels
type::bug describes erroneous operation, use severity::* to classify the type

Comments

@l0b0
Copy link

l0b0 commented Sep 22, 2023

Checklist

  • I added a descriptive title
  • I searched open reports and couldn't find a duplicate

What happened?

I work on a project which runs a matrix of jobs to verify that it works on a number of platforms (2 subprojects * 3 operating systems * 4 Python versions, for a total of 24). Sometime probably within the last few weeks about 25% of these jobs have started failing with the same general error for a bunch of different packages. For example:

CondaHTTPError: HTTP 502 BAD GATEWAY for url https://conda.anaconda.org/conda-forge/linux-64/pthread-stubs-0.4-h36c2ea0_1001.tar.bz2

Is it possible that one of your mirrors is misbehaving, or that the project is being throttled and the server should be returning HTTP 429 Too Many Requests?

(I've added a caching step to the job just now to minimise the impact of unreliable mirrors.)

Conda Info

active environment : test
    active env location : /usr/share/miniconda/envs/test
            shell level : 2
       user config file : /home/runner/.condarc
 populated config files : /home/runner/.condarc
          conda version : 23.5.2
    conda-build version : not installed
         python version : 3.11.4.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.35=0
                          __linux=6.2.0=0
                          __unix=0=0
       base environment : /usr/share/miniconda  (writable)
      conda av data dir : /usr/share/miniconda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/runner/conda_pkgs_dir
       envs directories : /usr/share/miniconda/envs
                          /home/runner/.conda/envs
               platform : linux-64
             user-agent : conda/23.5.2 requests/2.29.0 CPython/3.11.4 Linux/6.2.0-1011-azure ubuntu/22.04.3 glibc/2.35
                UID:GID : 1001:127
             netrc file : None
           offline mode : False

Conda Config

==> /home/runner/.condarc <==
auto_update_conda: False
auto_activate_base: True
notify_outdated_conda: False
changeps1: False
pkgs_dirs:
  - /home/runner/conda_pkgs_dir
channels:
  - conda-forge
  - defaults
always_yes: True

Conda list

# packages in environment at /usr/share/miniconda/envs/test:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
blosc                     1.21.5               h0f2a231_0    conda-forge
boost-cpp                 1.78.0               h2c5509c_4    conda-forge
brotlipy                  0.7.0           py311hd4cff14_1005    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cachecontrol              0.12.14            pyhd8ed1ab_0    conda-forge
cairo                     1.16.0            h0c91306_1017    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py311hb3a22ac_4    conda-forge
cfitsio                   4.2.0                hd9d235c_0    conda-forge
charset-normalizer        3.2.0              pyhd8ed1ab_0    conda-forge
cleo                      2.0.1              pyhd8ed1ab_0    conda-forge
crashtest                 0.4.1              pyhd8ed1ab_0    conda-forge
cryptography              41.0.4          py311h63ff55d_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
distlib                   0.3.7              pyhd8ed1ab_0    conda-forge
dulwich                   0.20.50         py311h2582759_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
filelock                  3.12.4             pyhd8ed1ab_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
freexl                    2.0.0                h743c826_0    conda-forge
gdal                      3.6.4           py311h815a124_18    conda-forge
geos                      3.12.0               h59595ed_0    conda-forge
geotiff                   1.7.1               hee599c5_13    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
giflib                    5.2.1                h0b41bf4_3    conda-forge
hdf4                      4.2.15               h501b40f_6    conda-forge
hdf5                      1.14.2          nompi_h4f84152_100    conda-forge
html5lib                  1.1                pyh9f0ad1d_0    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.13.0             pyha770c72_0    conda-forge
importlib_metadata        4.13.0               hd8ed1ab_0    conda-forge
importlib_resources       6.0.1              pyhd8ed1ab_0    conda-forge
jaraco.classes            3.3.0              pyhd8ed1ab_0    conda-forge
jeepney                   0.8.0              pyhd8ed1ab_0    conda-forge
json-c                    0.17                 h7ab15ed_0    conda-forge
jsonschema                4.19.1             pyhd8ed1ab_0    conda-forge
jsonschema-specifications 2023.7.1           pyhd8ed1ab_0    conda-forge
kealib                    1.5.1                hcd42e92_5    conda-forge
keyring                   23.13.1         py311h38be061_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
lcms2                     2.15                 h7f713cb_2    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libaec                    1.0.6                hcb278e6_1    conda-forge
libblas                   3.9.0           18_linux64_openblas    conda-forge
libcblas                  3.9.0           18_linux64_openblas    conda-forge
libcurl                   8.3.0                hca28451_0    conda-forge
libdeflate                1.19                 hd590300_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_2    conda-forge
libgdal                   3.6.4               hdb503af_18    conda-forge
libgfortran-ng            13.2.0               h69a702a_2    conda-forge
libgfortran5              13.2.0               ha4646dd_2    conda-forge
libglib                   2.78.0               hebfc3b9_0    conda-forge
libgomp                   13.2.0               h807b86a_2    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libjpeg-turbo             2.1.5.1              hd590300_1    conda-forge
libkml                    1.3.0             h37653c0_1015    conda-forge
liblapack                 3.9.0           18_linux64_openblas    conda-forge
libnetcdf                 4.9.2           nompi_h80fb2b6_112    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.24          pthreads_h413a1c8_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     15.4                 hfc447b1_0    conda-forge
librttopo                 1.1.0               hb58d41b_14    conda-forge
libspatialite             5.0.1               h090f1da_29    conda-forge
libsqlite                 3.43.0               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_2    conda-forge
libtiff                   4.6.0                h29866fb_1    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.3.2                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxml2                   2.11.5               h232c23b_1    conda-forge
libzip                    1.10.1               h2629f0a_3    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lockfile                  0.12.2                     py_1    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
minizip                   4.0.1                h0ab5242_4    conda-forge
more-itertools            10.1.0             pyhd8ed1ab_0    conda-forge
msgpack-python            1.0.5           py311ha3edf6b_0    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.92                 h1d7d5a4_0    conda-forge
numpy                     1.26.0          py311h64a7726_0    conda-forge
openjpeg                  2.5.0                h488ebb8_3    conda-forge
openssl                   3.1.3                hd590300_0    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h36c2ea0_0    conda-forge
pkginfo                   1.9.6              pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
platformdirs              2.6.2              pyhd8ed1ab_0    conda-forge
poetry                    1.3.2           linux_pyhd8ed1ab_0    conda-forge
poetry-core               1.4.0              pyhd8ed1ab_0    conda-forge
poetry-plugin-export      1.3.1              pyhd8ed1ab_0    conda-forge
poppler                   23.08.0              ha962622_1    conda-forge
poppler-data              0.4.12               hd8ed1ab_0    conda-forge
postgresql                15.4                 h8972f4a_0    conda-forge
proj                      9.3.0                h1d62c97_1    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.11.5          hab00c5b_0_cpython    conda-forge
python_abi                3.11                    4_cp311    conda-forge
rapidfuzz                 2.15.1          py311hcafe171_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
referencing               0.30.2             pyhd8ed1ab_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
requests-toolbelt         0.10.1             pyhd8ed1ab_0    conda-forge
rpds-py                   0.10.3          py311h46250e7_0    conda-forge
secretstorage             3.3.3           py311h38be061_1    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
shellingham               1.5.3              pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
sqlite                    3.43.0               h2c6b66d_0    conda-forge
tiledb                    2.16.3               hf0b6e87_3    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tomlkit                   0.12.1             pyha770c72_0    conda-forge
trove-classifiers         2023.9.19          pyhd8ed1ab_0    conda-forge
typing-extensions         4.8.0                hd8ed1ab_0    conda-forge
typing_extensions         4.8.0              pyha770c72_0    conda-forge
tzcode                    2023c                h0b41bf4_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
urllib3                   1.26.15            pyhd8ed1ab_0    conda-forge
virtualenv                20.21.1            pyhd8ed1ab_0    conda-forge
webencodings              0.5.1              pyhd8ed1ab_2    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xerces-c                  3.2.4                hac6953d_3    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.6                h8ee46fc_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxrender           0.9.11               hd590300_0    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

Additional Context

No response
@l0b0 l0b0 added the type::bug describes erroneous operation, use severity::* to classify the type label Sep 22, 2023
@jezdez jezdez transferred this issue from conda/conda Sep 22, 2023
@jezdez
Copy link
Member

jezdez commented Sep 22, 2023

@l0b0 I've moved this to the right repo, since conda/conda is about the software.

@barabo Any chance to look at this?

@barabo
Copy link

barabo commented Sep 22, 2023

@jezdez - you can assign this to me, I'm looking into it

@barabo
Copy link

barabo commented Sep 22, 2023

@l0b0 👋

Could you point me to a couple of the specific failed GH actions? It would be helpful to see the earliest and the most recent ones, if you can dig those up for me.

@l0b0
Copy link
Author

l0b0 commented Sep 23, 2023

This job might be the first one, but it has a slightly different error:

HTTP 000 CONNECTION FAILED

I can't seem to find old failed runs for any of the merged PRs, so I strongly suspect GitHub must be discarding those logs. I'm also not sure when the problem started; the previous time I had a look at the project seems to have been 17 August 2023.

@jakirkham
Copy link
Member

Seeing this HTTP 000 CONNECTION FAILED error in CI:

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <[https://conda.anaconda.org/conda-forge/win-64/cudatoolkit-11.8.0-h09e9e62_13.conda>](https://conda.anaconda.org/conda-forge/win-64/cudatoolkit-11.8.0-h09e9e62_13.conda%3E)
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <[https://conda.anaconda.org/conda-forge/win-64/cudatoolkit-11.8.0-h09e9e62_13.conda>](https://conda.anaconda.org/conda-forge/win-64/cudatoolkit-11.8.0-h09e9e62_13.conda%3E)
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

@jakirkham
Copy link
Member

Have been seeing a variation on this intermittently recently when re-rendering in conda-forge

Here is a recent example taken from the GHA rerender service (with attached log):

requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.anaconda.org/download/conda-forge/conda-forge-pinning/2024.05.29.16.17.59/noarch/conda-forge-pinning-2024.05.29.16.17.59-hd8ed1ab_0.conda

@barabo barabo removed their assignment Jun 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type::bug describes erroneous operation, use severity::* to classify the type
Projects
🧭 Planning
Status: 🆕 New
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jezdez @l0b0 @jakirkham @barabo