Log Helix queue health summary on job submission (opt-in)

[chcosta]

Summary

Surfaces the new queueStats block that POST /api/jobs now returns from the Helix service, so anyone submitting a Helix job through the Arcade SDK can see the current health of their target queue at submit time without having to dig into the portal.

The feature is opt-in via a new MSBuild property and a matching fluent-API method, so it adds zero log noise to builds that haven't opted in.

User-visible change

Opt-in

<PropertyGroup>
  <EnableShowHelixQueueStats>true</EnableShowHelixQueueStats>
</PropertyGroup>

or /p:EnableShowHelixQueueStats=true. Direct Microsoft.DotNet.Helix.JobSender callers use .WithQueueStats() on IJobDefinition.

What lands in the build log

Healthy queue:

note : Helix queue health reporting is a preview feature; data and format may change.
Helix queue 'windows.10.amd64.open' health:
  Estimated wait : 0s   (queue depth: 0, avg run: 3m 26s)
  Snapshot taken : 2026-05-29 16:14 Pacific Daylight Time
  Questions about Helix queue health? Reach the dnceng First Responders channel: <teams url>

At-capacity queue (estimated wait exceeds the 30-minute SLA):

note : Helix queue health reporting is a preview feature; data and format may change.
Helix queue 'osx.15.amd64.open' health [AT CAPACITY]:
  Estimated wait : 1h 51m   (queue depth: 5864, avg run: 1m 24s)
  Snapshot taken : 2026-05-29 16:01 Pacific Daylight Time
warning : Helix queue 'osx.15.amd64.open' estimated wait of 1h 51m exceeds the 30-minute SLA - the queue is at capacity or unhealthy. Jobs may take longer than usual to start.
  Questions about Helix queue health? Reach the dnceng First Responders channel: <teams url>

Stale snapshot (> 15 min old): a second warning : line is emitted noting the age and threshold; the snapshot line also gets a (stale) tag.

Noise-control choices

• Off by default. Anyone who hasn't opted in sees no change in their logs.
• The preview-feature banner (note :) prints once per process.
• The First Responders Teams link prints once per process.
• Per-job summary is the same compact 2 lines plus optional 1ΓÇô2 warnings; emoji-free; ASCII dashes so it renders on any console codepage.
• Snapshot time is converted to the user's local timezone with the TZ name (e.g. Pacific Daylight Time) ΓÇö no more guessing UTC offsets.
• SLA warning uses the MSBuild warning : prefix so it shows up in PR build summaries and AzDO timeline issues.

Implementation

File	Change
src/Microsoft.DotNet.Helix/Client/CSharp/QueueStats.cs (new)	Newtonsoft.Json model bound to the queueStats JSON (queueName, depth, averageRunDuration, estimatedWait, estimatedWaitMethod, generatedAt).
src/Microsoft.DotNet.Helix/Client/CSharp/JobCreationResult.cs (new)	Partial-class extension exposing QueueStats. Keeps the generated generated-code/Models/JobCreationResult.cs untouched.
src/Microsoft.DotNet.Helix/JobSender/IJobDefinition.cs	Added WithQueueStats() fluent method.
src/Microsoft.DotNet.Helix/JobSender/JobDefinition.cs	Implements the new fluent method. After JobApi.NewAsync(...) returns, if the flag is on, calls LogQueueStats(...) which formats and emits the summary, SLA + stale warnings, and once-per-process banner / Teams link.
src/Microsoft.DotNet.Helix/Sdk/SendHelixJob.cs	New EnableShowHelixQueueStats task input piped into .WithQueueStats().
src/Microsoft.DotNet.Helix/Sdk/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets	Plumbs EnableShowHelixQueueStats="$(EnableShowHelixQueueStats)" onto the <SendHelixJob /> element.
src/Microsoft.DotNet.Helix/Sdk/tools/Microsoft.DotNet.Helix.Sdk.props	Default value EnableShowHelixQueueStats=false (mirrors the existing EnableHelixJobMonitor pattern).

Validation

Unit tests

• Microsoft.DotNet.Helix.Sdk.Tests ΓÇö 121/121 pass with these changes.

End-to-end against live helix.dot.net

I packed the modified packages locally (11.0.0-dev) into arcade/artifacts/packages/Debug/NonShipping/ and consumed them from a separate scratch repo (c:\repos\arcade-helix-stats-validate) that imports Microsoft.DotNet.Helix.Sdk via a fork-pinned NuGet.config. From that repo:

1. Gate OFF (default) ΓÇö submitted a tiny job to windows.10.amd64.open. Build log contained only the existing Sending Job to ... / Created job list at ... / Sent Helix Job; see work items at ... lines. Zero queue-health output, confirming the opt-in gate works.

2. Gate ON via /p:EnableShowHelixQueueStats=true ΓÇö same submit, against the same queue. Build log included the full preview banner + health summary + First Responders link shown above. Queue happened to be healthy (depth 0), so no SLA warning.

3. Gate ON against a backlogged queue (osx.15.amd64.open) ΓÇö exercised separately via a local programmatic probe consuming Microsoft.DotNet.Helix.JobSender directly. Output included the [AT CAPACITY] tag and the warning : ... exceeds the 30-minute SLA - ... line shown above. Confirmed the once-per-process gating on both the banner and the Teams link by submitting twice in the same process and observing the second submission's output omits both.

Stale-snapshot path

Live snapshots from the service were < 1 min old during validation, so the (stale) tag and "snapshot is N old" warning weren't observed live ΓÇö both are exercised by code review of JobDefinition.cs and gated by SnapshotStaleThreshold = TimeSpan.FromMinutes(15).

Out of scope

The Helix service is shipping additional queueStats fields (e.g. P50Wait, P95Wait, HealthyCores, ObserverUiUrl, EstimatedWaitConfidence). This PR intentionally only binds the fields needed to render the current preview summary; surfacing the richer telemetry can land in a follow-up once the field set stabilizes.

[Copilot]

Pull request overview

Adds an opt-in “Helix queue health” summary that surfaces queue wait/depth/snapshot info returned by the Helix POST /api/jobs response, so Arcade/Helix job submitters can see queue conditions at submit time without visiting the Helix portal.

Changes:

• Introduces a new MSBuild property (EnableShowHelixQueueStats, default false) and plumbs it through SendHelixJob into IJobDefinition.WithShowQueueStats(...).
• Extends the Helix client model (JobCreationResult) with a queueStats block and adds a QueueStats JSON model.
• Implements logging in JobDefinition.SendAsync to print a compact summary plus SLA/staleness warnings (once-per-process banner and support link).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/Microsoft.DotNet.Helix/Sdk/tools/Microsoft.DotNet.Helix.Sdk.props	Adds defaulted opt-in property EnableShowHelixQueueStats=false.
src/Microsoft.DotNet.Helix/Sdk/tools/Microsoft.DotNet.Helix.Sdk.MonoQueue.targets	Passes EnableShowHelixQueueStats into the SendHelixJob task invocation.
src/Microsoft.DotNet.Helix/Sdk/SendHelixJob.cs	Adds a task parameter and forwards it to IJobDefinition.WithShowQueueStats(...).
src/Microsoft.DotNet.Helix/JobSender/IJobDefinition.cs	Adds the fluent API method WithShowQueueStats(bool).
src/Microsoft.DotNet.Helix/JobSender/JobDefinition.cs	Implements the flag and logs queue health after job submission.
src/Microsoft.DotNet.Helix/Client/CSharp/QueueStats.cs	Adds the QueueStats JSON-bound model for the new response block.
src/Microsoft.DotNet.Helix/Client/CSharp/JobCreationResult.cs	Partial-class extension to expose QueueStats from the job creation response.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

[garath]

Using local time seems inconsistent. Is UTC better? I haven't looked around but I feel that UTC is much more common in our builds.

[chcosta]

It's true that UTC is more common, and how we store data on the server. Given that this is a QoL (and client-side) type change, I personally, found it annoying to constantly do the translation. If there's a strong opinion, I can move this back to UTC.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.