feat: Jenkins Airgap Rancher and Airgap Harvester Pipeline Support

* brings in Jenkins pipeline
* adjusts Debian based provisioning
* adds additional credential
* additional settings.yml.sample configuration
* additional pipeline dependent plugin introduced
* fixes existing provisioning issues due to the nature of hashicorp
  products

Resolves: feat/air-gap-harvester-air-gap-rancher-cicd-pipelining-feat

AirGap-Harvester-Jenkins-Node-Infra/jeninks-airgap-harvester-installer-based-node/README.md

@@ -0,0 +1,46 @@
+# Introduction
+
+This is the Ansible to stand up a Jenkins Node that will use harvester-installer to create the artifacts for Harvester, for a pipeline in Jenkins that is capable of running AirGap Harvester & AirGap Rancher provisioning over Vagrant that leverages ipxe-examples (airgap version).
+
+# Setup New Jenkins
+
+To setup [Jenkins] on a target host.
+
+1. Make sure [Ansible] is installed. You can install the latest version
+   of [Ansible] using [Python PIP].
+2. Copy `settings.yml.sample` to `settings.yml`.
+3. Edit `settings.yml` by providing the required configurations. The
+   configurations are self-documented.
+4. Edit `inventory.harvester-ci` to make sure the host IP and Ansible user are
+   correct. **NOTE:** the Ansible user must have SSH access to the CI host and
+   have sudo permissions.
+5. Run the `install_jenkins.ym` playbook. For example:
+
+```console
+ansible-playbook -i inventor.harvester-ci --private-key <ansible user private key> install_jenkins.yml
+```
+
+# Add a Jenkins Slave
+
+To add a Jenkins Slave.
+
+1. Make sure [Ansible] is installed. You can install the latest version
+   of [Ansible] using [Python PIP].
+2. Copy `settings.yml.sample` to `settings.yml`.
+3. Edit `settings.yml` by providing the required configurations. The
+   configurations are self-documented.
+4. Edit `inventory.harvester-ci` to make sure the host IP and Ansible user are
+   correct. **NOTE:** the Ansible user must have SSH access to the CI host and
+   have sudo permissions.
+5. Install the required packages on the Jenkins Slave host by running the
+   `install_jenkins_slave.yml` playbook. For example:
+
+```console
+ansible-playbook -i inventory.harvester-ci --private-key <ansible user private key> install_jenkins_slave.yml
+```
+
+6. Manually add the new node from Jenkins Master.
+
+[Ansible]: https://www.ansible.com/
+[Jenkins]: https://www.jenkins.io/
+[Python PIP]: https://pip.pypa.io/en/stable/

AirGap-Harvester-Jenkins-Node-Infra/jeninks-airgap-harvester-installer-based-node/install_jenkins.yml

@@ -0,0 +1,31 @@
+---
+- name: Install Jenkins on localhost
+  hosts: harvester-ci
+  become: yes
+  vars:
+    GITHUB_PROJECT: harvester/harvester-installer
+
+  tasks:
+    - name: Include settings
+      include_vars:
+        file: settings.yml
+
+    - name: Check for supported OS version
+      fail:
+        msg: "OS must be openSUSE or Ubuntu"
+      when: (ansible_distribution|lower != 'opensuse leap' and
+             ansible_distribution|lower != 'ubuntu')
+
+    - name: Setup PKI
+      include_role:
+        name: pki
+      when: JENKINS_PROXY_ENABLE_SSL
+
+    - name: Setup Nginx proxy
+      include_role:
+        name: nginx
+      when: JENKINS_USE_PROXY
+
+    - name: Install Jenkins
+      include_role:
+        name: jenkins

AirGap-Harvester-Jenkins-Node-Infra/jeninks-airgap-harvester-installer-based-node/install_jenkins_slave.yml

@@ -0,0 +1,19 @@
+---
+- name: Install Jenkins Slave
+  hosts: harvester-ci-slave
+  become: yes
+
+  tasks:
+    - name: Include settings
+      include_vars:
+        file: settings.yml
+
+    - name: Check for supported OS version
+      fail:
+        msg: "OS must be openSUSE or Ubuntu"
+      when: (ansible_distribution|lower != 'opensuse leap' and
+             ansible_distribution|lower != 'ubuntu')
+
+    - name: Install Jenkins Slave
+      include_role:
+        name: jenkins_slave

AirGap-Harvester-Jenkins-Node-Infra/jeninks-airgap-harvester-installer-based-node/inventory.harvester-ci

@@ -0,0 +1,6 @@
+[harvester-ci]
+master ansible_host= ansible_user=root
+
+[harvester-ci-slave]
+slave ansible_host= ansible_user=root
+

AirGap-Harvester-Jenkins-Node-Infra/jeninks-airgap-harvester-installer-based-node/roles/jenkins/defaults/main.yml

@@ -0,0 +1,7 @@
+---
+JENKINS_ADMIN_USERNAME: admin
+JENKINS_ADMIN_PASSWORD: jenkins
+JENKINS_ADMIN_EMAIL: [email protected]
+JENKINS_DEV_USERNAME: harvester
+JENKINS_DEV_PASSWORD: harvester
+JENKINS_PUBLIC_ENDPOINT: http://localhost:8080

AirGap-Harvester-Jenkins-Node-Infra/jeninks-airgap-harvester-installer-based-node/roles/jenkins/tasks/install_docker.yml

@@ -0,0 +1,9 @@
+---
+- name: jenkins | install_docker | Install Docker
+  include_tasks: install_docker_on_{{ ansible_os_family }}.yml
+
+- name: jenkins | install_docker | Add jenkins user to docker group
+  user:
+    name: jenkins
+    groups: docker
+    append: yes

AirGap-Harvester-Jenkins-Node-Infra/jeninks-airgap-harvester-installer-based-node/roles/jenkins/tasks/install_docker_on_Debian.yml

@@ -0,0 +1,22 @@
+---
+- name: jenkins | install_docker_on_Debian |
+        Install prerequisite packages for Docker
+  apt:
+    name: [apt-transport-https, ca-certificates, curl, software-properties-common]
+    state: latest
+
+- name: jenkins | install_docker_on_Debian | Add Docker apt repo key
+  apt_key:
+    url: https://download.docker.com/linux/ubuntu/gpg
+    state: present
+
+- name: jenkins | install_docker_on_Debian | Add Docker apt repo
+  apt_repository:
+    repo: "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"
+    state: present
+    update_cache: true
+
+- name: jenkins | install_docker_on_Debian | Install docker-ce package
+  apt:
+    name: docker-ce
+    state: latest

AirGap-Harvester-Jenkins-Node-Infra/jeninks-airgap-harvester-installer-based-node/roles/jenkins/tasks/install_docker_on_Suse.yml

@@ -0,0 +1,11 @@
+---
+- name: jenkins | install_docker_on_Suse | Install Docker packages
+  community.general.zypper:
+    name: [docker, python3-docker-compose]
+    state: latest
+
+- name: jenkins | install_docker_on_Suse | Enable docker service
+  service:
+    name: docker
+    enabled: yes
+    state: started

AirGap-Harvester-Jenkins-Node-Infra/jeninks-airgap-harvester-installer-based-node/roles/jenkins/tasks/install_jenkins.yml

@@ -0,0 +1,137 @@
+---
+- name: jenkins | install_jenkins | Set Jenkins bootstrap credential
+  set_fact:
+    JENKINS_BOOTSTRAP_USERNAME: jenkins_bootstrap_user
+    JENKINS_BOOTSTRAP_PASSWORD: "{{ lookup('password', '/dev/null chars=ascii_letters,digits length=30') }}"
+
+- name: jenkins | install_jenkins | Install Jenkins
+  include_tasks: install_jenkins_on_{{ ansible_os_family }}.yml
+
+- name: jenkins | install_jenkins | Wait for Jenkins to be ready
+  uri:
+    url: "http://localhost:8080/cli/"
+    status_code: 200
+  register: get_jenkins_cli_result
+  until: get_jenkins_cli_result.status == 200
+  retries: 10
+  delay: 20
+
+- name: jenkins | install_jenkins | Install Jenkins CLI
+  get_url:
+    url: "http://localhost:8080/jnlpJars/jenkins-cli.jar"
+    dest: "/opt/jenkins-cli.jar"
+  register: jarfile_get
+  until: "'OK' in jarfile_get.msg or '304' in jarfile_get.msg or 'file already exists' in jarfile_get.msg"
+  retries: 5
+  delay: 10
+
+- name: jenkins | install_jenkins | Create jenkins CLI to install plugins
+  template:
+    src: jenkins.j2
+    dest: /usr/bin/jenkins
+    mode: 0755
+    force: yes
+  vars:
+    JENKINS_AUTH_USERNAME: "{{ JENKINS_BOOTSTRAP_USERNAME }}"
+    JENKINS_AUTH_PASSWORD: "{{ JENKINS_BOOTSTRAP_PASSWORD }}"
+
+- name: jenkins | install_jenkins | Install Jenkins plugins
+  shell: >
+    /usr/bin/jenkins install-plugin {{ item }}
+  with_items:
+    - ansible
+    - authorize-project
+    - build-timeout
+    - blueocean
+    - bootstrap5-api
+    - configuration-as-code
+    - credentials-binding
+    - docker-workflow
+    - email-ext
+    - ghprb
+    - git
+    - github-branch-source
+    - github-oauth
+    - htmlpublisher
+    - job-dsl
+    - ldap
+    - mailer
+    - matrix-auth
+    - pam-auth
+    - pipeline-github-lib
+    - pipeline-stage-view
+    - pipeline-utility-steps
+    - ssh-slaves
+    - timestamper
+    - workflow-aggregator
+    - workflow-cps
+    - workflow-job
+    - ws-cleanup
+
+- name: jenkins | install_jenkins | Remove Jenkins security bootstrap scripts
+  file:
+    path: /var/lib/jenkins/init.groovy.d/basic-security.groovy
+    state: absent
+
+- name: jenkins | install_jenkins | Create Jenkins ansible_playbooks directory
+  file:
+    path: /var/lib/jenkins/ansible_playbooks
+    state: directory
+    owner: jenkins
+    group: jenkins
+    mode: 0755
+  when: false
+
+- name: jenkins | install_jenkins | Create Jenkins config as code directory
+  file:
+    path: /var/lib/jenkins/casc_configs
+    state: directory
+    owner: jenkins
+    group: jenkins
+    mode: 0755
+
+# supports airgap rancher airgap harvester pipeline
+- name: jenkins | install_jenkins | Build Jenkins .ssh directory
+  ansible.builtin.file:
+    path: /var/lib/jenkins/.ssh
+    state: directory
+    owner: jenkins
+    group: jenkins
+
+- name: jenkins | install_jenkins | Build known_hosts file
+  ansible.builtin.file:
+    path: /var/lib/jenkins/.ssh/known_hosts
+    state: present
+
+- name: jenkins | install_jenkins | Create Jenkins config as code file
+  template:
+    src: config_jenkins_as_code.yaml.j2
+    dest: /var/lib/jenkins/casc_configs/config_jenkins_as_code.yaml
+    owner: jenkins
+    group: jenkins
+    mode: 0644
+
+- name: jenkins | install_jenkins | Copy pipeline jobs
+  template:
+    src: "{{ item }}.j2"
+    dest: "/var/lib/jenkins/casc_configs/{{ item }}"
+    owner: jenkins
+    group: jenkins
+    mode: 0755
+  with_items:
+    - airgap_rancher_airgap_harvester_pipelinejob.groovy
+
+- name: jenkins | install_jenkins | Restart jenkins
+  service:
+    name: jenkins
+    state: restarted
+
+- name: jenkins | install_jenkins | Create jenkins CLI
+  template:
+    src: jenkins.j2
+    dest: /usr/bin/jenkins
+    mode: 0755
+    force: yes
+  vars:
+    JENKINS_AUTH_USERNAME: "{{ JENKINS_ADMIN_USERNAME }}"
+    JENKINS_AUTH_PASSWORD: "{{ JENKINS_ADMIN_PASSWORD }}"

AirGap-Harvester-Jenkins-Node-Infra/jeninks-airgap-harvester-installer-based-node/roles/jenkins/tasks/install_jenkins_on_Debian.yml

@@ -0,0 +1,85 @@
+---
+- name: jenkins | install_jenkins_on_Debian |
+        Install prerequisite packages for Jenkins
+  apt:
+    name: [ca-certificates, openjdk-11-jdk, curl, apt-transport-https, gnupg,
+           python3, python3-pip, figlet, sshpass]
+    state: latest
+    update_cache: yes
+
+- name: jenkins | install_jenkins_on_Debian | Install Ansible
+  pip:
+    name: [pip, ansible]
+    state: latest
+
+- name: jenkins | install_jenkins_on_Debian | Add Jenkins apt repo key
+  apt_key:
+    url: https://pkg.jenkins.io/debian-stable/jenkins.io.key
+    state: present
+
+- name: jenkins | install_jenkins_on_Debian | Add Jenkins apt repo
+  apt_repository:
+    repo: 'deb http://pkg.jenkins.io/debian-stable binary/'
+    state: present
+    update_cache: true
+
+- name: jenkins | install_jenkins_on_Debian | Install Jenkins package
+  apt:
+    name: jenkins
+    state: latest
+
+- name: jenkins | install_jenkins_on_Debian | Stop jenkins service
+  service:
+    name: jenkins
+    state: stopped
+
+# NOTE(gyee): running this tasks repeatedly will add the same Java args
+# multiple time. But that should be fine so as long as the values are
+# consistent. However, if we are using this task to reconfigure any of the
+# args with a different value then the result may not be correct.
+# If we ever need to reconfigure Jenkins, it may be best to just do re-install.
+- name: jenkins | install_jenkins_on_Debian | Add Java options
+  lineinfile:
+    path: /etc/default/jenkins
+    regexp: '^(JAVA_ARGS=\")(.*)$'
+    line: '\1-Djenkins.install.runSetupWizard=false -Dcasc.jenkins.config=/var/lib/jenkins/casc_configs \2'
+    state: present
+    backrefs: yes
+    mode: 0644
+
+- name: jenkins | install_jenkins_on_Debian | Bind to localhost
+  lineinfile:
+    path: /etc/default/jenkins
+    regexp: '^(JENKINS_ARGS=\")(.*)$'
+    line: '\1--httpListenAddress=127.0.0.1 \2'
+    state: present
+    backrefs: yes
+    mode: 0644
+  when: JENKINS_USE_PROXY
+
+- name: jenkins | install_jenkins_on_Debian |
+        Create init.groovy.d to bootstrap Jenkins
+  file:
+    path: /var/lib/jenkins/init.groovy.d
+    state: directory
+    owner: jenkins
+    group: jenkins
+    mode: 0775
+
+- name: jenkins | install_jenkins_on_Debian |
+        Configure Jenkins bootstrap credential
+  template:
+    src: basic-security.groovy.j2
+    dest: /var/lib/jenkins/init.groovy.d/basic-security.groovy
+    owner: jenkins
+    group: jenkins
+    mode: 0755
+
+- name: jenkins | install_jenkins_on_Debian | Restart Jenkins
+  systemd:
+    name: jenkins
+    state: restarted
+
+- name: jenkins | install_jenkins_on_Debian | Enable Jenkins port 8080
+  shell: ufw allow 8080
+