Implement a retry mechanism for activation in internal tests

[GabLeRoux]

When running internal tests for the Unity orb with parallel executions on sample projects, activation failures occur intermittently. The error message suggests a failed login due to incorrect credentials, but retries usually resolve the issue. This issue is likely caused by rate-limiting on Unity's servers, as many requests are being sent for the same user simultaneously, especially when using a free Unity license.

Steps to Reproduce:

1. Run tests that involve Unity activation in parallel.
2. Observe failed activation attempts with error UnityConnectLoginRequest: Failed to login and HTTP error 401.

Expected Behavior:
The activation should retry a few times before failing to account for transient issues, such as rate-limiting.

Suggested Solution:

• Implement a simple retry mechanism with configurable timeouts and delays for Unity activation in the orb.
• Retry logic should be applied during activation attempts to handle transient errors gracefully.
• The retry mechanism should be configurable for users of the orb.

Example output of a failing activation job:

Detected OS: Linux.
Activating Unity with username, password, and serial.
Unity Editor version:    2022.3.33f1 (b2c853adf198)
Branch:                  2022.3/staging
Build type:              Release
Batch mode:              YES
System name:             Linux
Distro version:          #63~20.04.1-Ubuntu SMP Mon Mar 25 10:28:36 UTC 2024
Kernel version:          5.15.0-1057-aws
Architecture:            x86_64
Available memory:        70213 MB
CreateDirectory '/root/.cache/unity3d' failed:  (current dir: /root/project)
UnityConnectLoginRequest: Failed to login - please check your username or password
No sufficient permissions while processing request "https://core.cloud.unity3d.com/api/login", HTTP error code 401
.UnityConnectLoginRequest: Failed to login - please check your username or password
No sufficient permissions while processing request "https://core.cloud.unity3d.com/api/login", HTTP error code 401
.[UnityConnect::TryLogin] Authentication Retry (1) ...
UnityConnectLoginRequest: Failed to login - please check your username or password
No sufficient permissions while processing request "https://core.cloud.unity3d.com/api/login", HTTP error code 401
.[UnityConnect::TryLogin] Authentication Retry (2) ...
[Licensing::Module] Trying to connect to existing licensing client channel...
[Licensing::IpcConnector] Connection attempt to the License Client on channel: "LicenseClient-root" failed because channel doesn't exist; code: "0x80000002"
[Licensing::Module] Successfully launched the LicensingClient (PId: 242)
[Licensing::IpcConnector] Successfully connected to the License Client on channel: "LicenseClient-root" at "2024-11-26T01:34:05.070975Z"
[SignatureVerifier] Application signature verification not supported on this platform.
[Licensing::Client] Handshaking with LicensingClient:
  Version:                 1.15.4+eee609c
  Session Id:              c6049a0ec2d4484092ed96cf8db671b5
  Correlation Id:          80dc6ec6e327cde68703b107e8e8f6a3
  External correlation Id: 3782874392433401156
  Machine Id:              D7nTUnjNAmtsUMcnoyrqkgIbYdM=
[Licensing::Module] Successfully connected to LicensingClient on channel: "LicenseClient-root" (connect: 0.57s, validation: 0.05s, handshake: 0.00s)
[Licensing::IpcConnector] Successfully connected to the License Notification on channel: "LicenseClient-root-notifications" at "2024-11-26T01:34:05.124646Z"
[Licensing::Module] Connected to LicensingClient (PId: 242, launch time: 0.00, total connection time: 0.62s)
[Licensing::Module] Error: Access token is unavailable; failed to update
[Licensing::Module] Error: Failed to activate entitlement license
[Licensing::Module] Error: Access token is unavailable; failed to update
[Licensing::Module] Error: Failed to activate ULF license
Failed to generate Unity license file. Please verify your inputs.
Failed to activate Unity. Please check your inputs or open an issue.

Exited with code exit status 1

Here is an example implementation from a related repository:

https://gitlab.com/game-ci/unity3d-gitlab-ci-example/-/blob/v4.0.0/ci/before_script.sh?ref_type=tags#L7-34

---

Implementing this would be ideal because CircleCI only allows us to retry a given job when all of the jobs are done. Right now, if a job failed, one of the job takes 35 minutes to run so we need to wait until the end just to retry one. 😢