Runtime download of compile assets (denort-x86_64-unknown-linux-gnu.zip) fails

[thoughtpolice]

Version: Deno 2.1.9

(Refile of #26672)

While running deno compile under GitHub Actions, I am consistently running into the error described in #26672, repeated here below:

https://github.com/thoughtpolice/qq/actions/runs/13169912728/job/36758221145#step:6:75

[2025-02-06T01:28:15.048+00:00] Local command: env -- 'BUCK_SCRATCH_PATH=buck-out/v2/tmp/qq/9a8cb533f7a5e67f/deno_compile' buck-out/v2/gen/qq-toolchains/4cdf265ac4da48bd/deno/__2.1.9-x86_64-unknown-linux-gnu__/2.1.9-x86_64-unknown-linux-gnu/deno compile --output buck-out/v2/gen/qq/4cdf265ac4da48bd/src/tools/buck2-logs/__upload__/upload.exe --unstable-kv --allow-read --allow-net src/tools/buck2-logs/upload.ts
[2025-02-06T01:28:15.048+00:00] Stdout:
Archive:  /home/runner/work/qq/qq/buck-out/v2/tmp/qq/9a8cb533f7a5e67f/deno_compile/.tmpeYFk7K/denort.zip
[2025-02-06T01:28:15.048+00:00] Stderr:
Download https://jsr.io/@std/async/meta.json
Download https://jsr.io/@std/cli/meta.json
Download https://jsr.io/@std/fs/meta.json
Download https://jsr.io/@std/io/meta.json
Download https://jsr.io/@std/async/0.224.2_meta.json
Download https://jsr.io/@std/cli/1.0.5_meta.json
Download https://jsr.io/@std/fs/0.229.3_meta.json
Download https://jsr.io/@std/io/0.224.3_meta.json
Download https://jsr.io/@std/async/0.224.2/debounce.ts
Download https://jsr.io/@std/cli/1.0.5/parse_args.ts
Download https://jsr.io/@std/fs/0.229.3/exists.ts
Download https://jsr.io/@std/io/0.224.3/to_readable_stream.ts
Download https://jsr.io/@std/io/0.224.3/_constants.ts
Download https://jsr.io/@std/io/0.224.3/_common.ts
Download https://jsr.io/@std/io/0.224.3/types.ts
Check file:///home/runner/work/qq/qq/src/tools/buck2-logs/upload.ts
Compile file:///home/runner/work/qq/qq/src/tools/buck2-logs/upload.ts to buck-out/v2/gen/qq/4cdf265ac4da48bd/src/tools/buck2-logs/__upload__/upload.exe
Download https://dl.deno.land/release/v2.1.9/denort-x86_64-unknown-linux-gnu.zip
unpacking via zip crate failed: invalid Zip archive: No valid central directory found
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of /home/runner/work/qq/qq/buck-out/v2/tmp/qq/9a8cb533f7a5e67f/deno_compile/.tmpeYFk7K/denort.zip or
        /home/runner/work/qq/qq/buck-out/v2/tmp/qq/9a8cb533f7a5e67f/deno_compile/.tmpeYFk7K/denort.zip.zip, and cannot find /home/runner/work/qq/qq/buck-out/v2/tmp/qq/9a8cb533f7a5e67f/deno_compile/.tmpeYFk7K/denort.zip.ZIP, period.
error: Writing deno compile executable to temporary file 'buck-out/v2/gen/qq/4cdf265ac4da48bd/src/tools/buck2-logs/__upload__/upload.exe.tmp-99[59](https://github.com/thoughtpolice/qq/actions/runs/13169912728/job/36758221145#step:6:60)80d3bdf44ad2'

Caused by:
    Failed to unpack archive.

This only seems to happen:

• In GitHub Actions,
• On Ubuntu Linux runners

This has something like a 20-40% hit rate in my experience, so it's really clogging up my CI pipelines, where I run compile on some Deno apps in the tree.

I'm not sure how to easily reproduce this, but I guess you could fork my repo and just make a bunch of meaningless commits to try and diagnose?

[nktpro]

FYI, it's a much better practice to pre-download and cache the denort binary in CI pipelines instead of having it dynamically downloaded on the fly. It really should be treated like the deno binary itself (immutable - part of your CI runner image). Then, you can set DENORT_BIN env var when executing deno compile.

DENORT_BIN=/path/to/denort deno compile ...

[thoughtpolice]

FYI, it's a much better practice to pre-download and cache the denort binary in CI pipelines instead of having it dynamically downloaded on the fly. It really should be treated like the deno binary itself (immutable - part of your CI runner image). Then, you can set DENORT_BIN env var when executing deno compile.

DENORT_BIN=/path/to/denort deno compile ...

I'm totally happy to do that, thanks! I wasn't aware this was an option; is there any documentation on this?

Really, it feels like this should be the default anyway; just downloading code after installation doesn't feel very good.

[nktpro]

@thoughtpolice we're on the same page. Since the inception of denort, I ran into the exact issue you did. I had to look into the code to find this option. This definitely should be publicly documented.

[tugrulates]

I could produce a minimal reproduction case, showing that this is a concurrency issue.

// test.ts, no `deno.json`
console.log("test");

# .github/workflows/ci.yml
on:
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Setup
        uses: denoland/setup-deno@v2

      - name: Build
        run: |
          deno compile -A -o test1 test.ts &
          deno compile -A -o test2 test.ts &
          deno compile -A -o test3 test.ts &
          deno compile -A -o test4 test.ts &
          deno compile -A -o test5 test.ts &
          deno compile -A -o test6 test.ts &
          deno compile -A -o test7 test.ts &
          deno compile -A -o test8 test.ts &
          deno compile -A -o test9 test.ts

This produces many failures like below.

Compile file:///home/runner/work/test/test/test.ts to test7
Download https://dl.deno.land/release/v2.1.9/denort-x86_64-unknown-linux-gnu.zip
unpacking via zip crate failed: invalid Zip archive: No valid central directory found
Archive:  /tmp/.tmpJ6VhvC/denort.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of /tmp/.tmpJ6VhvC/denort.zip or
        /tmp/.tmpJ6VhvC/denort.zip.zip, and cannot find /tmp/.tmpJ6VhvC/denort.zip.ZIP, period.
error: Writing deno compile executable to temporary file 'test2.tmp-e43bb7d7c9d278d0'
Caused by:
    Failed to unpack archive.

I doubt it is related to anything with GitHub runners. When the runtime is cached in a prior step with compile, the problem goes away. Thus, the issue is just difficult to reproduce locally.

Does not reproduce at v1.40.4, but reproduces at v1.40.5 where the denort binary was introduced with #22205.

[thoughtpolice]

Ah, good catch! That would explain it since my Buck builds probably try to run multiple copies of deno compile at once. Shared mutable state strikes again, it seems.

[dsherret]

Interesting. Does it happen in canary (deno upgrade canary)? Probably a82326d will fix this in the next release.

[tugrulates]

It is indeed reliably fixed with a82326d.

Compile is broken at HEAD for something else. Filed #28044 for it.

[dsherret]

Thanks for testing!