[META] Integrating multiple Jenkins CI instances for unified access

[jordarlu]

Is your feature request related to a problem? Please describe

The current OpenSearch Jenkins CI is critical to the success of OpenSearch project. It acts as a gateway for PR’s, distribution builds, benchmark tests, releases, security scans and more.

There are requests to add new tasks (for example adding integration test jobs against on a new Github Repo) to the current public OpenSearch Jenkins CI, keeping in mind the possibility of transitioning to community management soon.

The current Jenkins CI has been facing resource constraints, mainly caused by the increased specific tasks. This has led to service disruptions, for example increasing the frequency of build jobs, which in turn, has strained our other tasks within the Jenkins to run.

In response to these challenges, instead of relying on a current single Jenkins CI to handle all tasks, we're moving towards a distributed approach. This new strategy proposes the deployment of multiple Jenkins CI instances, each dedicated to managing specific types of tasks.

Describe the solution you'd like

The proposal in high level is to have multiple Jenkins instances, and each instance is responsible for a specific work/job/load ; for example we will have Jenkins for Build, Jenkins for Gradle Check, Jenkins for Benchmark, etc. Before splitting up jenkins the plan is to move the CDK package and deployment process to github, Github actions will be used instead of internal pipelines in the new approach.

Sub tasks:

• Review on the approaches to implement migration of the internal CDK package #502
• Configure agent-node usage based on jenkins instance type #509
• Build and deploy CI stacks using npm through github actions #517
• Enhance the mechanism to overcome the usage of stack context parameters #527

Acceptance criteria :

• Migrate the internal CDK package and the deployment steps to github.
• Jenkins CI shall be able to be deployed on multiple AWS accounts
• By default, every AWS account should be able to deploy one instance of Jenkins CI
• The Jenkins CI instances system should be publicly accessible as it is today
• Support for BYOJ (Bring Your Own Jenkins) shall be available, allowing integration any Jenkins CI with the current Build Jenkins CI that offers a public access point, provided it is accessible via the existing/current build CI URL path
• If any Jenkins CI account is compromised, it won't impact any other Jenkins CI
• If one Jenkins CI has been loaded, it won't impact any other Jenkins CI
• Different accounts, teams, or individuals can manage Jenkins CI instances without sharing sensitive information or secrets or minimizing the needs for sharing information crossing accounts

Describe alternatives you've considered

Keep using the current Jenkins CI

Additional context

N/A

[jordarlu]

We've opted to proceed with having the identification of Jenkins instances into the URL path.
The existing URL, https://build.ci.opensearch.org/, will serve as the primary access point for the main Jenkins CI. For accessing additional Jenkins instances, we will utilize specific paths within the URL, denoted as /path/.

For example: