Bazel v5.1.0 installation fails on Tools.Tests.ps1

[DonKoning]

Description

On both Ubuntu 2004 and Windows 2022 our image builds fail with the following error:

2022/06/01 12:40:51 could not resolve the version 'latest' to an actual version number: unable to determine latest version: could not list Bazel versions in GCS bucket: could not list GCS objects at https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/: could not fetch https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/: Get "https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/": net/http: TLS handshake timeout

====================================================================
    vhd: bazel v5.1.0 [Approved]
    vhd: bazel package files install completed. Performing other installation steps.
    vhd: Content of C:\ProgramData\chocolatey\lib\bazel\tools\params.txt:
    vhd: https://github.com/bazelbuild/bazel/releases/download/5.1.0/bazel-5.1.0-windows-x86_64.zip
    vhd: ac2309f165d4ea4811f5710b182e3640c7eee09af21c79eb0c438dcad8ea8560
    vhd:
    vhd: url:  https://github.com/bazelbuild/bazel/releases/download/5.1.0/bazel-5.1.0-windows-x86_64.zip
    vhd: hash: ac2309f165d4ea4811f5710b182e3640c7eee09af21c79eb0c438dcad8ea8560
    vhd: Downloading bazel 64 bit
    vhd:   from 'https://github.com/bazelbuild/bazel/releases/download/5.1.0/bazel-5.1.0-windows-x86_64.zip'
    vhd:
    vhd: Download of bazel-5.1.0-windows-x86_64.zip (42.77 MB) completed.
    vhd: Hashes match.
    vhd: Extracting C:\Users\packer\AppData\Local\Temp\chocolatey\bazel\5.1.0\bazel-5.1.0-windows-x86_64.zip to C:\ProgramData\chocolatey\lib\bazel...
    vhd: C:\ProgramData\chocolatey\lib\bazel
    vhd: bazel installed to C:\ProgramData\chocolatey\lib\bazel
    vhd:
    vhd: See also https://bazel.build/docs/windows.html
    vhd:  ShimGen has successfully created a shim for bazel.exe
    vhd:  The install of bazel was successful.
    vhd:   Software installed to 'C:\ProgramData\chocolatey\lib\bazel'
    vhd:
    vhd: Chocolatey installed 1/1 packages.
    vhd:  See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
    vhd: Package installed: bazel|5.1.0
    vhd:
    vhd: added 1 package, and audited 2 packages in 1s
    vhd:
    vhd: found 0 vulnerabilities
    vhd: Pester v5.3.3
    vhd:
    vhd: Starting discovery in 1 files.
    vhd: Discovery found 44 tests in 1.18s.
    vhd: Filter 'FullName' set to ('Bazel').
    vhd: Filters selected 2 tests to run.
    vhd: Running tests.
    vhd:
    vhd: Running tests from 'C:\image\Tests\Tools.Tests.ps1'
    vhd: Describing Bazel
    vhd:   [-] bazel 10.95s (10.76s|187ms)
    vhd:    Command 'bazel --version' has finished with exit code
    vhd:        **2022/06/01 12:40:51 could not resolve the version 'latest' to an actual version number: unable to determine latest version: could not list Bazel versions in GCS bucket: could not list GCS objects at https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/: could not fetch https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/: Get "https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/": net/http: TLS handshake timeout**
    vhd:    at "$ToolName --version"| Should -ReturnZeroExitCode, C:\image\Tests\Tools.Tests.ps1:22
    vhd:    at <ScriptBlock>, C:\image\Tests\Tools.Tests.ps1:22
    vhd:   [-] bazelisk 10.14s (10.14s|1ms)
    vhd:    Command 'bazelisk --version' has finished with exit code
    vhd:        2022/06/01 12:41:01 could not resolve the version 'latest' to an actual version number: unable to determine latest version: could not list Bazel versions in GCS bucket: could not list GCS objects at https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/: could not fetch https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/: Get "https://www.googleapis.com/storage/v1/b/bazel/o?delimiter=/": net/http: TLS handshake timeout
    vhd:    at "$ToolName --version"| Should -ReturnZeroExitCode, C:\image\Tests\Tools.Tests.ps1:22
    vhd:    at <ScriptBlock>, C:\image\Tests\Tools.Tests.ps1:22
    vhd: Tests completed in 24.66s
    vhd: Tests Passed: 0, Failed: 2, Skipped: 0 NotRun: 42
====================================================================

When I run a packet capture on the image VM and run the Bazel part of the C:\image\Tests\tools.tests.ps1 script, I see that there's no response to the Client hello message (probably since it's using TLS1), for reasons I don't understand).

Contents of the TLS package:

Tried to force TLS1.2 (Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12), but it didn't work.

Virtual environments affected

• Ubuntu 18.04
• Ubuntu 20.04
• Ubuntu 22.04
• macOS 10.15
• macOS 11
• macOS 12
• Windows Server 2019
• Windows Server 2022

Image version and build link

2.201.1

Is it regression?

No response

Expected behavior

Image build succeeds.

Actual behavior

Image build fails every time

Repro steps

Environment:

• vwan hub-spoke topology with AzFw
• Running deployment from Azure DevOps to existing spoke vNet/subnet using private scale set agent
• NetworkWatcher traffic analyzer reports no flows blocked by NSG rules
• AzFw logs show no traffic being blocked

[DonKoning]

Additional findings:

• When I run 'bazel --version' command in a cmd-prompt, I get the version number. There's no network traffic.
• When I run the 'bazel --version' command in a PowerShell terminal (v5.1 or 7.2.4), I get the 'TLS handshake timeout' error message
• When I download bazel on a Windows 2019 server and run the 'bazel --version' command in a PowerShell session (v7.2.2) I get "2022/06/01 15:09:31 Downloading https://releases.bazel.build/5.1.1/release/bazel-5.1.1-windows-x86_64.exe...
  bazel 5.1.1". A packet capture learns that this time TLSv1.3 is used.

Thoughts?

[IkHaffner]

Hi team, any update on this? Issue persists.