Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Discuss state of r2 migration on Build WG meeting #3508

Open
MoLow opened this issue Oct 2, 2023 · 42 comments
Open

Discuss state of r2 migration on Build WG meeting #3508

MoLow opened this issue Oct 2, 2023 · 42 comments
Labels
build-agenda

Comments

@MoLow
Copy link
Member

MoLow commented Oct 2, 2023

Opening this issue just to track this in the meeting.
we are pretty much done with getting https://r2.nodejs.org/dist/ working.
we want to make sure we did not miss anything and discuss the next steps - mainly when we should pull the plug on the migration
the Cloudflare team asked for an estimated date so they can prepare (they are depending on https://nodejs.org/dist for installing node on their infra :), they want to avoid circular reference issue )

CC @flakey5 @ovflowd
@richardlau
Copy link
Member

Opening this issue just to track this in the meeting.

FYI unless #3503 changes the meeting times, the next scheduled Build WG meeting is in the inconvenient-for-everybody slot on Wednesday 18th October. The next meeting after that, in the earlier slot, is on Tuesday 7th November.

I guess one question is whether we want to migrate before or after Node.js 21 -- usually a semver major release results in a spike in traffic.

@ovflowd
Copy link
Member

ovflowd commented Oct 3, 2023

Opening this issue just to track this in the meeting.

FYI unless #3503 changes the meeting times, the next scheduled Build WG meeting is in the inconvenient-for-everybody slot on Wednesday 18th October. The next meeting after that, in the earlier slot, is on Tuesday 7th November.

I guess one question is whether we want to migrate before or after Node.js 21 -- usually a semver major release results in a spike in traffic.

Oh yeah waiting till November is definitely a no go IMO. It would make sense to discuss about R2 way before that.

@mhdawson
Copy link
Member

mhdawson commented Oct 6, 2023

I just got an error when trying to hit the link mentioned

image

@mhdawson
Copy link
Member

mhdawson commented Oct 6, 2023

Does it make any sense to switch over a portion of our downloads first to validate on a smaller subset versus everything. Is that possible/worth any additional work.

@mhdawson
Copy link
Member

mhdawson commented Oct 6, 2023

When I hit the page again, it now seems to work instead of the error

@ovflowd
Copy link
Member

ovflowd commented Oct 6, 2023

I just got an error when trying to hit the link mentioned

image

@flakey5 ☝️

@flakey5
Copy link
Member

flakey5 commented Oct 6, 2023

I just got an error when trying to hit the link mentioned

Looking into it now to see what happened

Does it make any sense to switch over a portion of our downloads first to validate on a smaller subset versus everything. Is that possible/worth any additional work.

As in, for example, requests going to /dist/... hit the worker and all requests going to /download/... hit the origin? In that case it's definitely possible and would just be up to the routing config. For deeper paths like /dist/latest hitting the worker while /dist/vX.X.X hits the origin it should be possible, but I'm not 100% sure on that.

@flakey5
Copy link
Member

flakey5 commented Oct 7, 2023

Believe I know what the error is, should be fixed in nodejs/release-cloudflare-worker#44

@mhdawson
Copy link
Member

As in, for example, requests going to /dist/... hit the worker and all requests going to /download/... hit the origin?

Something like that. Roll out in smaller steps than a big bang. Do you have a suggestion for a first good step?

@richardlau
Copy link
Member

richardlau commented Oct 10, 2023 •
edited

I'm running into errors trying to promote Node.js 18.18.1 😞:

# Creating SHASUMS256.txt ...
upload failed: nodejs/release/v18.18.1/docs/api/all.html to s3://dist-prod/nodejs/release/v18.18.1/docs/api/all.html SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/docs/api/all.html EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-aix-ppc64.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-aix-ppc64.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-aix-ppc64.tar.gz?uploadId=AK5en5fhbO8ZsJyjv_fBucFKF_nxOZWmnYVx8Q3garrFPWRyq2q8drTOSKCzgKUhPajKU0Amnxyb2bP2ibtu23YtoUzUWurgo-eyOrTCGb7-r84FCEeUXsm9nK2fPct0NOWqy39ojE_p7kl2lGHDPl76k0bmcmHag7Rz-srSGElHAqv-EdcbuQZaXdZQaYDidov2W1Oi3W1FvPMrrfF1Jw6aqMaNspg1MH6ESNXsyVdovcRmlDS89m2oSX2-zr_CiirdKWSA8o0X0bWPNWS2zoXujri0NFD7yT39wKmrGCbGG20JkHj5a1C7WnTmtFoJNuV398LIyXsQQbeCdxxNKgI&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-darwin-arm64.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-darwin-arm64.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-darwin-arm64.tar.gz?uploadId=AH2Q3Vic8_76OWmwnrMacUFjhcSDZH4Q5BH-k9Ox65vWGdaedlgo0IMLJra90QGpY-VFwr6MzXIRaganfleHJK_-vfcD-KxuBW3Pz3M3edpEqJaE-2qc3Y_inkMdP8i_gvN4atAFEybBvGyYEaw0z999v5miTeym7udzACkhiJGGJcv97e4LrFL9K6vZ3KjCyidIZDNrTgcUfGJAWYY9JmgMveCwjJ6OuZkZYYxCPHVpgNqLeAuFr7gTBFU8AvrQB5DGoVaKve-ElV1zqPIUDggDeyNv5prkhIPvEEv-gOT4dduZSxO3VbqTjvXrGYrSHtn7bjW4QZkmLgCaDQbnJHo&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-darwin-arm64.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-darwin-arm64.tar.xz An error occurred (InternalError) when calling the UploadPart operation (reached max retries: 2): We encountered an internal connectivity issue. Please try again.
upload failed: nodejs/release/v18.18.1/node-v18.18.1-darwin-x64.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-darwin-x64.tar.xz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-darwin-x64.tar.xz?uploadId=AHpycjrfMp3Ou8gBMPUA88nOYF1d23N-rQtud7r-4Q2nvFp9Y2HDYhBzCl76vriPZRJx4bIr--HetvrX_pyEBKaq2Md1QsDUvmH69wAYRnajN9I-v5I6sQrZbX5vv_FUxwA8Oj2qWTAL6nKZz-KFH5SCpSvtGe_i9YALGX2m7aata16Up0NtaDynsL_vaT_FQZxh6M9aOpO1mv9ucbxsOsFwXZhZOC6ZJhscmyGPUH47jwvJDwXZVGJTQAG8QvKjB4uZQPZyn4aJ5_EtupYx8K8X6yKK-3cTgyraNlFX9SWtn022WQiq8vfoJQWmqkwad2_qaee2m3NCIFLK2YXtBh4&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.gz?uploadId=AD0two2TSNT8hnMWVnQOf4M5G-xs2gouFzsq33f5bMnQKRgg0pKP0jF54WSjAvZ3a0vbyTGzDmVPFpuzhWzqc84hzrL6BlGn3yJa8qkxUNlPQA2TZ9b23KjSlcIixVEGpyWKtkADiZAWEpyYjOzTySL181clydXJAMMa_ewH155cVd_m6ysMSIYN5ltYOhsSfxYh080uL_AhymJCVlDfm7SBRLIRFOOXeOpmYGsHkCmLgfWZLNCPqAtIFqNd9YKIeo1GgQnPiDyDw4Fs4XjALtQu1hyk-qqkLvI7hW_XmpXmhm136441N6XhlSfRW0tmvV6yhtk81uLo8GtIEzseOiU&partNumber=4 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-headers.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-headers.tar.gz An error occurred (502) when calling the UploadPart operation (reached max retries: 2): Bad Gateway
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.xz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.xz?uploadId=ABMnDebWVxhBc1k04Wcg99_ZuWBywvH307mvd1RGbzBQ-Nl9Cd8WcOFPgqrceIebh0o8OfNfLQxbWSEgf0FcSJYFZwl9RBG4VaAOaz_kUCvpqyWh6rjc9ovLQZ7XwBgHxW8qk-aKrdjAK5FuAu7h6Eo8Kh5Q8MdtRcd9xaTTBdAMWiDNrUnGFdnCjhw53jW9tUEt1lMevfOKlvjkAI9_kT833WFcCwdacQBF1xAsKC9ugXPRhWQbyeXrOCxZvTQzxFbYVRaCFgrWvR6-wCaaae4amq7cYqL0EiP7DIUIRLvWvYZhKsvx5xDXdgLA_z20tr0pjedEN1TK6xeQQqu5UIw&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-armv7l.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-armv7l.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-armv7l.tar.gz?uploadId=APNla1tU948J1WlUYujXN074miBtOynCvXW_pEHKPz33APXaSBXYPv68C5aYgawgnL4EV_T_ziGWXPLnMPD8fLS8vPnxStqmIUI8HR9_d62NbT1oXH0-sRSm3s7HVYnf8QlKK2xmQPr4IFqspoWUgG-vbEegPG1f3qrYajeSx6oOmGu9H5mTi893neyUdcqIkI--VWUtxTCfazc9lSBeIlT3qFKGBlLd_lnj2iPCrbACchTTE46RI5OxDQzfL9Hy7Wb9ZGe-lg8TnTEg7w9NKMkywUTGa_d5PulEEq0wdSNKOtXd_GfXdpXyTwbziuR6BI3vhPk5gAe67X27b9Hl-Jo&partNumber=3 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.gz?uploadId=AIwdPt9ASHPR_MWigdIMmSVw177O9XCiEOxWvAFjsr_4EipKvxjx46ylZXRo7WL4m-ubBC1vBAo3V33m8WVu7C4_Yd7WCkMEltL23-FMyiVJg_jWmY2wOwBpFoeXn2VlTio_4O2JS2s72miIlWFdRZE6D_DrzBzQFJ65i29ubGX1Xl31Nf8ZDex0_E31pajt2IuO6ZR-8fygJS5Q7qGCjNzM7U2uBTt3Y4gcxZPFH_uE-LTHBVCzsA4-8bWqBWDpLzkDBCG-05GjlUfsvRQyku5vXNTvYTVQ2vypFuSnfQYACXHHb2w7zed8Gq1tzXqX6nAp0hqI89_Wl_XnRd8xvM8&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.gz?uploadId=AMlL9loRZNEFKsqDwdnyjRu_0Jnvw29aDfoZI2nAxWQtM8C95V5Q5LZ76xP13PIRGoCOB0bIRZFJG6p66tF3cO8fknqJ-bLjbzE-XEUGmnCplxMI0RERotWU9BgUDLqF2DiFa1qt8xIuZ4MvO1YjtXGlhK4ESAZYqHY5vdY9GaTFTHAIEX7OiqFO07oaLmQ85dGhIfj0PHI_johOm-RL705XuKo3ErIL45vc_9PdNExbae175azbJCJqBbiczwypHCz5lf0310Nhpo_HTGnN7Cj2qfVx22_o7htu-eMtrzDd5hYE5UbuTLE5EGi8FJIDMQ7_kY6i7WXG5zll_d4tEKI&partNumber=4 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.xz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.xz?uploadId=AD-QybSCbx5AtyxVFP_zOUXmdCl05J95Oyp1QFB_W6lf1vmlXOZjpdKAYd-4L_4FTLyjo34oX4rDdBnfl-BoPUzzcjQ4wCxmRARrDNErNZXh8WbAJg6-PQzrVYUKf3EqAP8N_8yWckdxrS0DiQ494XqNruNOVwvKVxPboc2onfJbWW_ZgZT9O9PcvdXuSaKsBI13Ai_G2wJYsDb_wX7Cs0QtfNudADtFaZ5Fgzk48SK83SrO_f9GbHWxHab767TjAqM7hXsVzTH0T3Uh8N81Qkcq7AgbeteYEWeGZ2D-iRcWE0SnL3jRAtMFITG_qxCy01tMShYHDFb2oAsMOc_Olng&partNumber=3 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.gz?uploadId=AAwOyFGuwNTBenVpMm0focOuZcnDW2g2d8M1FsVPr4FmPOjDRl4-Tv0ecly20EwmTfgYWGMqWwHxnR8472eQmadv-sG_KUXU-HRsndQKaZtihGUc6kbXE_xrR2tVYRQvRZvaKtsOFS3zfovVkB4-GHudNOqVGhNYyAHdkV1CM3Smnl8O2JCR5RO2Qfr0LE02iTaMk3vXDnJf65EoP41QBiI5KJXgiWkgJyMhor9WL31mDwVGZm8kOOdgBDOB_LZQF2qCk9LYWSCAy_8se9K2WtKqwWjkAn7XR2Xy9-W0eiSCg5DwJtHAswn3moQal2MPBVsd0OdZhmXPis7W2Nd0nWQ&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.xz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.xz?uploadId=ANEZzsnpRXUgADvxhKSOUYXW7t7hxbJ8qxUTqTT7C2YHpgAse6Zn_e3BGjDxZUsMPLqfSYdHXXyy7IuWw5OhqFIGpnExWcKufbbx4SYcrYEDiZApoFHWwDUcwnmkJ99IMCh74dnL7W53-EkZuP-yGpDur3L_X5I9UUVjPkdKLgNI9Eno-uhcuv4UKcuHeLmE9QeivLTMO8RRiuvSqP0DCygRWxgnpyePwD7nQrh2JkE2IAt8D9yacsRxVY0oC6KTDDzgj4ttDIkz15qTXiVbhWPF0e6vzoLMgtWgFYZrPjIexCqkbm5RA6qNAMj-qxIUw6fxwulYTFtaN_zqDtOrvvo&partNumber=3 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.xz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.xz?uploadId=AB0nJ81_nbe8AA148NLE4Upjz0ASRbpbhIQVzEe3E24H6xKq5QiNASuuto5fKmsoeWW6VkQ0HTYll-EkEM9jtNMe-N3uOXjSfMrn2k0Za5hsncZv67kxEDbK1_F6coNcGI_cbQ_m6Un1VmLflRDqSPEM4kxZrQ2pRArEsfLo66t3FPEQdQtngUm1IbxIz5LDivhSEIKnL2FrQw1av0_1qRR9X4MMCueOo5WJF803JmE5df94NPWOle5Sv4NxI3o1dPPj4IFbmYgE6hEcmtC7kANhOSnIlKm5XwToPOwOC60O0YNWw0fh0EczU5GIYufceUKmo6S1KqBqtVgqURkw93g&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-win-x64.7z to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x64.7z SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x64.7z?uploadId=AA-QUfH57CHLUnp4zMoteTRVdHCm62UUaGR1ThGNmJ6J5YDGsjxbKE1xGTbT95_u-T2ADPSqOyH69wRDwI4XEDWxm_Sw411XiuzpXfZeDREctWd_aJnLIcSJXrQtUkyn0FSWkbtMra6koyKOrM_d-rf2GDDMukpYfo6XYQheM04frt1NjefGOnP2sfanSP_Uys6ghF2iq3427OqfjJueTrIqRBhpuJY50Vi53ZUKAjgWiTYwtHtza3xFTryYv8jQ45oqOnmfS-UwLVTxgebJuDblOEupILAMfLTEN0eMLBFV0D-sDocHhgPguO2yxv4dUmuujYruXMN6ckAYawkPTuA&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-win-x86.7z to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x86.7z Connection was closed before we received a valid response from endpoint URL: "https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x86.7z?uploadId=AOJKmxdqJZpGbcdtMJZR0A4rJwZHQsOsBB7XWGnOh6-CQ0Xh8kLWXfr-gtf8aKszYSdssvfXaCWOE27QY0Y5KGMejyQpQHSvrnZ6_hEl3mO8-MdYAy0PFtfqIBo9HfKlohlXhVv3-QeWDXc_PbicVyydWbDnCiUJERwxFmkLPRqzWWfBMTlid151dqgD_WoXgpD-quF9OD2h5YcgJ_-xRd5orGJ-bAAuD-5t_DD-a_oLJ3UPB1oZ5mLXsYcBTmibIxa4fMj4Uhs0meYAiC3Eaajh7Vx4sDP3lWH-0YlFB6SVr0TkylKWpmbY2GR0VkZgb-H3GseeTAJGuKj4YtJ0FZA&partNumber=3".
upload failed: nodejs/release/v18.18.1/node-v18.18.1-win-x86.zip to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x86.zip SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x86.zip?uploadId=AAqI5JU0k7ReQ25MvcfNJaUi394xfgSlaL5ly2aKUaUiKVAhJPLiBULxLhUQ4rYvSw9I4Bf_8klqsz66njzUA2YfioWH9-s6tUzs6fPGnGhqlF2cs_v2usDXBHYDhqisiAkoNEg5lQCbo6ljVC-B0Saw2zY9jWE37ajXtO_fcI6zdDhAmRSnG9C6tTIx9UY3mFN4i_TaqXcu-fn1LEfNFr6ZKnuk7KUn_g5iX72bomuHzPe9ad3jimx5JJwQ9fnScOKdyL8_P2hxPXRpWn00Iej0Gssu-ljMqXBdlG4bKsS_On4ajXOfB-fR3EN0C8AacJt1CQzLoR5KJybGEU_yEwo&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-win-x64.zip to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x64.zip SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x64.zip?uploadId=AHG6FbmDzD4q4xehQgIRvTjTrsOsEQyFX4YVfVM0m3QPrzNtBts07XqdqRRNeWDA8gMp8lSYD2MzHC5kvpGRONr1JsPuqn8oIFq3ZNRlMU2I0dI57SfzWCPmPRhW7_zPn9_iutokUyHIaDPuj8kHhbM-oEXdtIi-gPo_aG6Xs3cja9iyfhH1VtA-t0TX89Po5XDjLfoc5U5no9TJsB4XXGGnWn1A76PoQuoc37ytb49DqSp2Hekte6rgN2ZLhglQXweap6qPqsMGAaFXUKv-ZMbw8wAe6Lwo22Gwi4gDzq-EtZUFyo6-rbaRf1xtB4I1-kGe5_63hSWdsKOM5uVg97U&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-x64.msi to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-x64.msi SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-x64.msi?uploadId=AP4c7XUSrPT5bxe2-O_NMN2I45wgE_UFZM94P_lN3Y17FjXmFG_llHaCObiHOmFyTxwyMvCSKOziSIPEa_4puQaaJ2vSyhTdIRo3AlrME9zcOJ0glO4CuQ3KeWz_WfAAEbECldzqIdw3cxuwhpnS3Jj7VPUwtoMFpBgDqi6_yaVE_vNM2Nd6d42jYaQb8f7tQruH6KgqUcXP9w8A3FFaeLkP1-qimKjpifH4yDbjHWgjcnwoCh24e5pEgusRPq7UM65p76pVQcuXqog13GXMErRbMILeEGEe9MLdZ9Cb9eLdiuZlhABrBLXRkD73ebEm-w-1YO2YhKEHyfqEZDlV8no&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-x86.msi to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-x86.msi SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-x86.msi?uploadId=AOQzX2Yx7ZNW_rN9FPslfmTV7YYQoOclj-ONBSslE5Poy0GVTxMObVa1kAl9RqqabxX6r_uptKjwWGaAGzN_DLAXV9A-JjvFW8i9d2xuBPWC7yO8-l-p6rtwTTjT-yu4YrVxjb54-gz1mBQDtXBj6VOFb0YzglG4MP3BnUSS6wGprnqMxUo5kZgVW7S4qWhqsG8f1fYlxtBKFp8tLaDp33QRoofnOJgxQui7jiAyF8J068U8cq2SmdJqulmR_B_biR6V7wdYo0LetMU0ZW5HV9-zXQYd-b5pIR6wT1pK_6NeoHk1-K0LDzCE264VmLtCFJiUnll6gJtOrPDS7ysZC1g&partNumber=4 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1.pkg to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1.pkg SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1.pkg?uploadId=AO6MuwqBn4EJDVxIKoUum-rAvZaXF8ETUfFLcu4gtm271pga0BuoEnEUC-Sd8ZRL2MFUE-i9XlCB2KMEN2B2Iji5D_hs78Nx1NSmDKKdYZSNfWACxUbl2K8SBbJmBUfRiGSLJv5fk0ui8ne9qOvllysIBy4MODDs4bbf2zgc3WQPNV_U1LUAxNbkW73NmQmAie1mnW5spoP7FQQgh4q0nFdYW27niJihu7OEGMxDS9eLwQptxrcFbSxajpTn4qja_Y8QJUbiDAu_spk7WmCkDAj78l8Ap1-d61_xpwsPMKwUDPQCTr1jXix-HmRFfAbGt5UW_Hph5MlJa2QTxRf9O1c&partNumber=9 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1.tar.gz?uploadId=ADBk6NL9Q2ehez8_McAh1Mf7CR4zyh1QK6kPXYmpj9MDU7Jz0HalsCPX-6mfqTSGcct3FBdnDSbadj3EuB1uIEUo8vdi4bzQvEuGSUsjeTvEjVdVBRZtQ40b_CcIVXZZnQJ7tBj66uejgkMt_zkpkqkgvyQNI6FNeWa0K9c1t8t6BzxBDRIPrq5Y-BkyzDpaUDi0KeELWe0mT9vXeYGTtLReoN3oHR2aSsmr0weiLE7agGf2bhW09EWaPJ8PY8DxrjEVxD5WpnGxipRTaNs4S8XX6O3vK7d9YdlcXxo49CZ0eR32ze8n3LC6o1W7fy6XSyysY27YjWcwkLTrO6dzJmY&partNumber=3 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x64/node.exe to s3://dist-prod/nodejs/release/v18.18.1/win-x64/node.exe SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x64/node.exe?uploadId=ALfJRer4UbzGgzZAAC8E447xGQBu2Jva9QUTLsObZH6BoPzSxx9PaSK9efBK-gkyZZUsj05NjUwuqznjS6ilQufuVQ4AG-jbAeVyfPNNuN1i5SabdlV0433dg38nJIIY7-OL9gUp-f18nkPULFmLeKvoMObMzV3BioYLKMJ9JZpMPavvahzupJBI__i1k5XWofooRqb3wVOsMvcXV49uT5ZZxsitP7hW5cIjZSz9mhDrfh4RmVUgAI8tHHyhGDPBIACqby_HB2ChdlMaCpUfgMOS8S22TiJ4DSavjiXqAT4qo7Iw4FITgcs-r6n_x3AvGJg4B8FgNR62NhuUHYvXL60&partNumber=3 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x64/node.lib to s3://dist-prod/nodejs/release/v18.18.1/win-x64/node.lib SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x64/node.lib?uploadId=AON6QbdBHLTVcjEC5SOke9-pPSp9FYvUZED-5SWbFU3Y-npTbO1Arj5_k0FP0Lapgfkg5l5pYx6OAZnTtVla4RMHfk-wdKMPIVtt6fUDkYLcfqIeBoCQVf5Pm8fJra-YY6qJsng1YrpjKe9xi2lSepRbChKOvOB75cuHXoyFLXDFEa0xQCAkR-aMhans_W_Cad8YqH3gwVLTJhyCS1kStiIWvluKOuRx9zt7qpIkQHcP5v5fTJWEPyBLMM3aYWeoUahni1JGaVBgkvY03fWkBMQWmc5Jckl4sc0PtBAU9el5RfFpPxdqhgzIQQyUTDHrNU8DGqI0ruROAf6IQzuDbbs&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x64/node_pdb.7z to s3://dist-prod/nodejs/release/v18.18.1/win-x64/node_pdb.7z SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x64/node_pdb.7z?uploadId=AA5kO1d2ZgdVAlWKzAF3NGddqb8WIGngKT8D4VrJxExlYz2B2A7NaHYOqFUzzMi9aWgA3ofzf7cf2jjT5JhrXytavKVsR0QBv3l72ti2E04xFnjhJQNWYSfFzgzH6dYwQtlxC05KAFjOWrIkU5NPge1CBuTpYtfJQgMaYGYh8yjH9Y5YP4bCmnhlnKB6nE9C-rvQ8cfc79-hvxPMelBJWP8gsQx6D8V3tBGrPboIQKUSZMt2BxjxaRvQxbR44wjzp6x_ohxV4t7XFbJdGpKAJDVpClzHZ9L0dxG8SBVFB7GD9fRXg36rafyl3Wy07oDHNbNREtHDpVLDJcbcyDRTANk&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x64/node_pdb.zip to s3://dist-prod/nodejs/release/v18.18.1/win-x64/node_pdb.zip SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x64/node_pdb.zip?uploadId=AH8ieQkTJSvGmxBlv6PbrBh1NjuAXibvLkqgvCa_13-pCedouLVWSccfKhd2hDc-IeNQzNuB19Z2DY-7zh3K5i5ULhGW11_xPoQF4KbwuCaE3y2sBEk-Ylg5E5CzIY3RhJfyd2zA1KnjrrVE29o9qPtt-Lyss6IwXWDtdCcFu0bNjh-fcen5-iZXq4fjg1Ir75j8K2dZZGlLhsp9e5BvOVcXsPghcBkK5vJBSgho5DktNcaUi8YeoSAsGyxNSvAfjyBMw5SM0tXGTf_0QO-KNlWkExvAPsC-7EpQA4iyTTVpU2SV0wUfwiH5D2P6jOgQxvCkQojycFCAps-962m2-v8&partNumber=5 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x86/node.exe to s3://dist-prod/nodejs/release/v18.18.1/win-x86/node.exe SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x86/node.exe?uploadId=AHnm_19Q4ivEGat7YNepjZOnNDPl6CSp5Npjev_9XgvK-Kbmp_7eGtTyp7X6qd26LA_Bvoh3eEERG4kjrN1U6ouSb6vMBXtjTqdsaSZzr49c98ca9p5eNWQ7NeBnbD99F3y_y555xMmlEZ3vVzUH0bSWAgwuegLNBNXWNDSlWLPyqyf0TV1kO6n0VsmnFa7YsHc6QNQsijh-9RlDmr_sqMLJwrsSjuP87muUS5spJSGrguWYPsc_Q2q0lQBv6__lWWi1UUTsakjeT-6CxjZvHXT69R1GBfSIFjGW3bdW_jyWbzPzxxNCU97zvy5B96ARhuuNN0hPMK3pshpd_Bv4nZc&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x86/node.lib to s3://dist-prod/nodejs/release/v18.18.1/win-x86/node.lib SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x86/node.lib?uploadId=ANnBZCq-6ijn_8ZcDCdA32R3ZA_kVl4hw_gL98dmCXJcrvQMuEOBB0NU39VZZihB6XyvBb9ZN3b0wlSMCwhdlW3rOvEHWJzkZJKmmTLK9RviH-yl4whPjmvkITyBXBAC8wyLRjCuLpqs_y2SuhDLtxMacpe9D3uWqIaSXAuvPnS3WqOLABzJas2BE2VPnK4AVkvon5YlSoud2EYnR1quqe0uzmMdfXNYXlKAyAD2ukQDf_FF1n29SekhxbcLlBLioCI-SNZHarYlCmPKPLxjNyNQgCos_ntwvk8b0LF-Lowu7D-LLQViNPAElzMfF50ZEXrUsBTw1DmTICqkFj54HeI&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x86/node_pdb.7z to s3://dist-prod/nodejs/release/v18.18.1/win-x86/node_pdb.7z SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x86/node_pdb.7z?uploadId=AJ3AQL3aZCrLsy7myySv_L61kLKVTa_7t_U68o-pUOHUYY1lS29itps-etR8ZS8xFrjwqmpXOUiaOxIugul14pijacW7diecbA3fLm_kV8e6ROwI0bU2d0N-oF4Ao68CJNqbrfaQuUg8Xx_XmPcQVcgqTiSNbYXn33VRqJM9ngZTDl5OZvPtFysq6bff6EL7n6NojFy9oxeXbdmBAILyCIk2Y3N8DRSNXFhH_yima17OVJ-cqW12xt43rV_0NGP-Xl8F1H16UETzzSx_S2gn4zpN_pnsl6md4QOLj0z1GTND1UPiWED3P_95nDjeshrFzs7InJgN8AlZ1MdDf5A3VD8&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x86/node_pdb.zip to s3://dist-prod/nodejs/release/v18.18.1/win-x86/node_pdb.zip SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x86/node_pdb.zip?uploadId=ABmd5cHDQxWUZ4F_1XW_YTuBDpYxAU5QZWK7bFdb6xzkbWidqPfoWiWIjMybXNAtG_VH9rHPUT_mybY_mpGqCwOIWJzaydvvXFH44kIx5dRDPwv2HMMhwQbl4A3W8MVGkhVUnRSAqosRt7Mc2wLG-b7tPD5MrmkLxX2HnMrQK9Lbuz_zqoQbaF_WHhnD9EPMmN5_dPMflpmEKtvqaJTdako82Ws2wV7Wsxal0ZoGVrQaARlG71rbzGJTfp2lqMEat7CknovUyWnxketGWognkKyhj4EMX8zfQKN7zndS4Z4gNCiA7J5el4fssVPeYeBKsQ_dlFHOtBI6_EmowyUUkx0&partNumber=5 EOF occurred in violation of protocol (_ssl.c:2427)

The errors are very slow -- presumably the uploads are being attempted and erroring one by one.

This is just running tools/release.sh as per https://github.com/nodejs/node/blob/main/doc/contributing/releases.md#15-promote-and-sign-the-release-builds.

@flakey5
Copy link
Member

flakey5 commented Oct 10, 2023

Something like that. Roll out in smaller steps than a big bang. Do you have a suggestion for a first good step?

First step could be /metrics, that way if anything does go wrong with how we route to the worker we can fix it without disrupting too much

I'm running into errors trying to promote Node.js 18.18.1

@MoLow

@richardlau
Copy link
Member

FYI I've had to comment out the calls to upload_to_cloudflare.sh from the promotion scripts so that tools/release.sh can actually sign the releases (the errors from the upload were causing the release script to bail out).

_promote.sh-1267-# https://github.com/nodejs/build/issues/3508
_promote.sh:1313:#    . ${__dirname}/upload_to_cloudflare.sh $site $subdir
_promote.sh-1371-    fi
--
resha_release.sh-505-# https://github.com/nodejs/build/issues/3508
resha_release.sh:551:# . ${__dirname}/upload_to_cloudflare.sh $site $2
resha_release.sh-601-
--

@ovflowd
Copy link
Member

ovflowd commented Oct 10, 2023

Why would there be SSL validation failures? @flakey5 do you want to share this with the Cloudflare team? Afaik we have one Cloudflare Engineer and our Account Manager on the #nodejs-build channel

@jasnell
Copy link
Member

jasnell commented Oct 10, 2023

Check the certs provided by the server. As a temporary workaround also try setting allow unauthorized to skip the cert validation. It's possibly with either a mismatch in the cname expected by the cert or a CA issue.

@flakey5
Copy link
Member

flakey5 commented Oct 10, 2023

The cert is valid from September 6th to December 5th. I'm not really sure what the issue with it is, there's no status page or anyone talking about a similar issue in the Cloudflare Discord so I'm more inclined to think it's something with the origin server itself. But iirc this was working on the origin server already, so I don't know what happened.

Could we retry syncing it just to see if it's reproducible?

@MoLow
Copy link
Member Author

MoLow commented Oct 11, 2023

btw it seems the artifacts did partially upload,
https://r2.nodejs.org/dist/v18.18.1/, https://r2.nodejs.org/docs/latest-v18.x/api/ and https://r2.nodejs.org/docs/latest-hydrogen/api/ are updated but https://r2.nodejs.org/dist/latest-v18.x/ and https://r2.nodejs.org/dist/latest-hydrogen/ are not

@richardlau
Copy link
Member

@MoLow Are those separate uploads or are those latest-* directories supposed to be handled by https://github.com/nodejs/release-cloudflare-worker? I see nodejs/release-cloudflare-worker#47 has updated the JSON to point at 18.18.1.

@ovflowd
Copy link
Member

ovflowd commented Oct 11, 2023

@MoLow Are those separate uploads or are those latest-* directories supposed to be handled by nodejs/release-cloudflare-worker? I see nodejs/release-cloudflare-worker#47 has updated the JSON to point at 18.18.1.

I also wonder if the redirect updates should be done automatically? Because creating the manual need of updating those redirects feels weird. Also, @MoLow or @flakey5 are the caches for those latest-* folders updated/cleared if needed?

@flakey5
Copy link
Member

flakey5 commented Oct 11, 2023 •
edited

I also wonder if the redirect updates should be done automatically?

The redirect update is scheduled for every night at midnight (https://github.com/nodejs/release-cloudflare-worker/blob/3fcba3ef2149a639a77b3369109426b3f7a600af/.github/workflows/update-links.yml#L6) in addition to being able to be manually triggered. Someone on the web infra team does need to approve the prs still though.

are the caches for those latest-* folders updated/cleared if needed?

Right now yes since the entire zone's cache is still being purged. Will probably need to update the worker's cache purge endpoint though to work with the redirects we're doing.

@MoLow
Copy link
Member Author

MoLow commented Oct 12, 2023

Are those separate uploads or are those latest-* directories supposed to be handled by nodejs/release-cloudflare-worker? I see nodejs/release-cloudflare-worker#47 has updated the JSON to point at 18.18.1.

yes, as @flakey5 said this is entirely automated.
this is very weird as the nightly builds seem to be uploaded correctly.

@mhdawson mhdawson mentioned this issue Oct 16, 2023
Closed
@MoLow MoLow mentioned this issue Oct 16, 2023
Closed
@flakey5 flakey5 mentioned this issue Nov 5, 2023
Closed
@MoLow
Copy link
Member Author

MoLow commented Nov 8, 2023

@nodejs/build @flakey5 @ovflowd what is the current status than?

@MoLow MoLow mentioned this issue Nov 8, 2023
Open
@flakey5
Copy link
Member

flakey5 commented Nov 8, 2023

We're ready for the worker to be deployed on

  • nodejs.org/metrics*
  • nodejs.org/docs*
  • nodejs.org/api*
  • nodejs.org/download*

Not nodejs.org/dist* since there may still be a circular dependency with R2's CI, I'm waiting to hear back from them on it. nodejs.org/download/release does link to the same folder as nodejs.org/download/release, but it should be fine.

@targos
Copy link
Member

targos commented Nov 16, 2023 •
edited

Before doing the next release (next week), I'll try to investigate why binaries were not synced to R2 when I released v21.2.0.

  • Compare script contents between server and github -> identical
  • Compare what happens when we call release.sh vs release.sh -s
  • Add echo statements in promotion script

@ovflowd
Copy link
Member

ovflowd commented Nov 16, 2023

Right, let's maybe also reference this on > nodejs/release-cloudflare-worker#74

@targos
Copy link
Member

targos commented Nov 22, 2023

  • ./tools/release.sh -s VERSION:
    • checktag VERSION
    • sign VERSION (runs dist-sign)
  • ./tools/release.sh:
    • run dist-promotable command on www to list releases in staging and ask user if wants to promote (y/n)
    • checktag VERSION
    • run dist-promote command on www
    • sign VERSION (runs dist-sign)

The last step of sign is to copy the shasum files to the server:

https://github.com/nodejs/node/blob/1858341377c268c5ffce9345517dc07b0e6c240a/tools/release.sh#L145-L151

The sync to R2 should only be done after that. Currently, it runs too early:

https://github.com/nodejs/node/blob/1858341377c268c5ffce9345517dc07b0e6c240a/tools/release.sh#L106

https://github.com/nodejs/build/blob/0734aaac357b1cff2965638b1b6e92afc86592e5/ansible/www-standalone/tools/dist/dist-sign

build/ansible/www-standalone/tools/promote/resha_release.sh

Line 28 in 0734aaa

. ${__dirname}/upload_to_cloudflare.sh $site $2

@targos
Copy link
Member

targos commented Nov 22, 2023

Status: the script is currently runnning aws s3 cp /home/dist/nodejs/release/v20.10.0/ s3://dist-prod/nodejs/release/v20.10.0/ ... and that's taking a long time (with no feedback to the console).

@targos
Copy link
Member

targos commented Nov 22, 2023

And it just started to fail with the "EOF occurred in violation of protocol" error 😭

@targos
Copy link
Member

targos commented Nov 22, 2023

@richardlau I think your hunch was right. The release puts a high load on the server and it takes to long to send the files to R2 (we probably hit a timeout).

@targos
Copy link
Member

targos commented Nov 22, 2023

I disabled the upload script, signed the release, and re-synced the folder manually to R2.

@ovflowd
Copy link
Member

ovflowd commented Nov 22, 2023

What should we do then? Yo be able to upload these files during release? Or should we run this after the release or should we maybe maybe shard or thread this?

@mhdawson mhdawson mentioned this issue Nov 27, 2023
Closed
@targos
Copy link
Member

targos commented Nov 29, 2023 •
edited

I tried again today for the release of v18.19.0. It failed to sync with R2 again. This time not because of timeout/ssl errors, but because some file name is too long?

Full error output: https://gist.github.com/targos/039dc834d68fb1480bc8a96e6fb757d4

Last line is

Completed 161.7 KiB/161.7 KiB (630.1 KiB/s) with 1 file(s) remaining\rupload: nodejs/release/index.tab to s3://dist-prod/nodejs/release/index.tab: File name too long

I tried to run the signing step multiple times but it always failed with the same error.
I disabled the sync in resha_release.sh for now.

@ovflowd
Copy link
Member

ovflowd commented Nov 29, 2023

I tried again today for the release of v18.19.0. It failed to sync with R2 again. This time not because of timeout/ssl errors, but because some file name is too long?

Full error output: gist.github.com/targos/039dc834d68fb1480bc8a96e6fb757d4

Last line is

Completed 161.7 KiB/161.7 KiB (630.1 KiB/s) with 1 file(s) remaining\rupload: nodejs/release/index.tab to s3://dist-prod/nodejs/release/index.tab: File name too long

I tried to run the signing step multiple times but it always failed with the same error. I disabled the sync in resha_release.sh for now.

How is that file name too long O.o

@targos
Copy link
Member

targos commented Nov 29, 2023

The actual error is probably unrelated to index.tab. R2 contains the latest version of that file.

@mhdawson mhdawson mentioned this issue Nov 29, 2023
Closed
@targos targos mentioned this issue Dec 5, 2023
Closed
@targos targos pinned this issue Dec 5, 2023
@targos
Copy link
Member

targos commented Dec 5, 2023

We discussed this during the last Build WG meeting.

Suggested approach to implement:

  • In addition to copying assets to the DO server (www), release jobs should also push directly to R2
    • Like on www, they should be pushed to a staging place, and moved to production when promoted
  • During promotion, only the shasums and signatures would need to be pushed directly to R2 (by the script run by the releaser)

@richardlau
Copy link
Member

Suggested approach to implement:

  • In addition to copying assets to the DO server (www), release jobs should also push directly to R2

    • Like on www, they should be pushed to a staging place, and moved to production when promoted

On this particular point, currently the uploads to www are done in the Makefile and vcbuild.bat as a series of scp and ssh commands.

Generally, for each asset:

  1. Use scp to upload the asset to www.
  2. Use ssh to touch a corresponding *.done file. I believe these are to prevent the promotion scripts from promoting a partially uploaded file.

To upload to R2 we either need to install the S3 client and secret onto every release machine, or we might be able to use ssh to run the upload from www after we have uploaded the asset (e.g. something like ssh $(STAGINGSERVER) "upload-to-cloudflare.sh ...").

@ovflowd
Copy link
Member

ovflowd commented Dec 5, 2023

The 2nd option feels simpler for the time being, and sorta ensures we always have one copy of the assets somewhere besides Cloudflare

@ovflowd
Copy link
Member

ovflowd commented Dec 5, 2023

But the 1st option feels better for the longterm probably?

@richardlau
Copy link
Member

The 2nd option feels simpler for the time being, and sorta ensures we always have one copy of the assets somewhere besides Cloudflare

And on that note, one reason for now to keep uploading to the DO droplet (besides being cautious) is that the backup server (actual backup server and NOT the fail over www server) currently rsyncs with the DO droplet. Today, each binary asset is:

  • Initially uploaded to www (DO droplet).
  • Uploaded to R2 (from www, via the promotion script or manually synced (due to issues)).
  • Rsynced to unencrypted (the fail over server) from www.
  • Rsynced to backup (non-public server) from www.

IIRC the rsyncs to unencrypted and backup are run on timers on those machines (i.e. they are pulled and not pushed from www).

@ovflowd
Copy link
Member

ovflowd commented Dec 5, 2023

qq: failover would the joyent server?

@richardlau
Copy link
Member

qq: failover would the joyent server?

Yes.

@MoLow
Copy link
Member Author

MoLow commented Dec 5, 2023

We discussed this during the last Build WG meeting.

Suggested approach to implement:

  • In addition to copying assets to the DO server (www), release jobs should also push directly to R2

    • Like on www, they should be pushed to a staging place, and moved to production when promoted

  • During promotion, only the shasums and signatures would need to be pushed directly to R2 (by the script run by the releaser)

FWIW, I started implementing something similar in #3501

@mhdawson mhdawson mentioned this issue Dec 18, 2023
Closed
@mhdawson mhdawson mentioned this issue Jan 8, 2024
Open
@mhdawson mhdawson mentioned this issue Jan 29, 2024
Open
@mhdawson mhdawson mentioned this issue Feb 19, 2024
Open
@mhdawson mhdawson mentioned this issue Mar 11, 2024
Closed
@mhdawson mhdawson mentioned this issue Apr 1, 2024
Closed
@mhdawson mhdawson mentioned this issue Apr 22, 2024
Closed
@mhdawson mhdawson mentioned this issue May 13, 2024
Open
@richardlau richardlau unpinned this issue May 14, 2024
@richardlau richardlau pinned this issue May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
build-agenda
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@jasnell @targos @richardlau @MoLow @mhdawson @ovflowd @flakey5