[RFE] QUADS Self-Scheduling Phase 1 via API

[sadsfae]

This is the skeleton feature that will enable future self-scheduling via the UI here: #98

• Design flexible API scheduling framework using dedicated RBAC roles / token auth (perhaps we pre-define a set number of "slots" like we do cloud environments as users)

• Design a threshold activation mechanism based on both cloud capacity % and per-model % usage, e.g. a host level metadata attribute like ss_enabled: true/false

  • We can manage this with a cron-interval tool to check / set as needed like selfservice_manager.py
  • We can provide a global argparse override like --enable-ss true/false

• Design what the tenant workflow might look like, e.g.

• Query available self scheduling
• If enabled, what models/systems?
• Use API to obtain a "self scheduling" cloud/user
• Perform set commands to define cloud, add systems (schedule times are pre-set, e.g. starting now and lasting X days or a week"
• Possibly include API mechanisms to auto-extend so long as thresholds meet.

[jtaleric]

This is awesome!

Just to give a high-level what we are looking to do with Quads with our baremetal CPT, here is a quick and dirty visual ;-)

Our use-case for integration would be for leases <= 24 hours -- depending on the cluster size we are requesting. Basically these allocations will be ephemeral in nature. Not ideal for long investigations, but great for quick response to hardened performance automation.

let us know if this fits the model you describe here! Thanks Will!

[josecastillolema]

Thanks @sadsfae and @jtaleric ! I think this is a great step in the right direction to enable dynamic allocations in the scale lab.

After discussing with @jtaleric the way I have thought of the process is:

• Step 1 - The request: This would be a new Prow step that would query the QUADS API for the leftovers (servers that are not statically assigned in today's dynamic inventory via the usual reservation process). The reservation request would look something like:

  • NUM_SERVERS: the number of servers being requested
  • SERVERS_LABELS: Just some examples: sriov_intel, sriov_mellanox, dell_640, disk_nvme, disk_ssd. I don't know if this labels exist today or will have to be implemented.

  If the servers are available:

  • interact with the QUADS API to create a new cloud with those servers: this would setup the networking, BMCs, VLANs, generate the corresponding OCPINV, etc.
  • interact with the QUADS API (or the foreman one, not sure) to deploy the bastion server

• Step 2 - Deploy OCP corresponds to the current openshift-qe-installer-bm-deploy which we would need to improve by implementing a dynamic inventory

• Step 3 - Run benchmarks, no planned changes here, this is already implemented

• Step 4 - Return the machines, a new Prow step to cleanup, delete the cloud, etc.

[sadsfae]

Great feedback so far, keep it coming. We have a work-in-progress design document we're mostly done with but needs internal review before we share it but I gave @jtaleric and @radez a preview.

I think what you'd care about is the tenant API request workflow, it would look ideally something like this broken down into a GET request to obtain eligible systems and three API POSTs to get your own workload delivered.

You'd receive JSON returns for each of the API requests below that can be fed into any automation you have, note the jump to using the generated API token, etc.

1) Query available, eligible systems (OPEN)

• API GET to return list of systems eligible for self-scheduling (if self-scheduling is unavailable, return as such)

2) Request assignment (OPEN)

• API POST to Request a new self-service assignment
  • Sends: Workload description, tenant kerberos name e.g. dradez (mapped to email), optionally public VLAN requirement (we would auto-allocate a free one) and Q-in Q design
  • Receives: Permanent unique ID for their user (creates ID if it doesn’t exist)
  • Receives: Newly generated API auth token (for this request only)
  • Receives: JIRA ticket url for new JIRA issue (--cloud-ticket for this request)

3) Acquire assignment (Authenticated)

• API POST to Acquire an assignment
  Sends: Generated token, JIRA ticket number and permanent unique ID
  Receives: JSON return with cloud number, ticket, description

4) Deploy Assignment (Authenticated)

• Curl API POST to Deploy their assignment

  • Sends: Token, unique ID and list of hosts they want
  • Receives: JSON status return on success/failure

• After step 4 you can query the open public API too to view your list of machines as well

• After step 4 you can also poll the release status of the cloud environment and have your automation gate on that before proceeding.

More Details WIP

(excuse the screenshot, didn't convert to markdown)

[sadsfae]

Thanks @sadsfae and @jtaleric ! I think this is a great step in the right direction to enable dynamic allocations in the scale lab.

After discussing with @jtaleric the way I have thought of the process is:

• Step 1 - The request: This would be a new Prow step that would query the QUADS API for the leftovers (servers that are not statically assigned in today's dynamic inventory via the usual reservation process). The reservation request would look something like:

  • NUM_SERVERS: the number of servers being requested
  • SERVERS_LABELS: Just some examples: sriov_intel, sriov_mellanox, dell_640, disk_nvme, disk_ssd. I don't know if this labels exist today or will have to be implemented.

This already exists in our metadata models and filters for --ls-available

https://github.com/redhat-performance/quads/blob/latest/docs/quads-host-metadata-search.md

This can also be expanded upon and import anything of value via lshw and our Python metadata import tool:

https://github.com/redhat-performance/quads/blob/22bb54a2bf989103532057486542eef52fe29d1a/src/quads/tools/lshw2meta.py

If the servers are available:

• interact with the QUADS API to create a new cloud with those servers: this would setup the networking, BMCs, VLANs, generate the corresponding OCPINV, etc.
• interact with the QUADS API (or the foreman one, not sure) to deploy the bastion server

We would consume all the frills and features of a normal deliberately scheduled future QUADS assignment so all that would be inclusive. As to what you'd receive it would not differ at all from what someone who receives a future assignment scheduled by us would receive, only that you can request it yourself with a few API calls (we need several because we do a lot including talking to JIRA and Foreman on the backend).

• Step 2 - Deploy OCP corresponds to the current openshift-qe-installer-bm-deploy which we would need to improve by implementing a dynamic inventory
• Step 3 - Run benchmarks, no planned changes here, this is already implemented
• Step 4 - Return the machines, a new Prow step to cleanup, delete the cloud, etc.

The world is your oyster, but we're just auto-delivering the hardware/networks here. Any additional hour zero work is in your purview only to action like deploying OCP and running your workloads. You will be able to release the systems (or extend them) with the same set of API's though.

One thing to keep in mind is our provisioning release time is what it is, there's no significant speedups beyond our usage of asyncio (already in codebase and current QUADS) and what having multiple, concurrent gunicorn threads/listeners provides. Bare-metal / IPMI / boot / mechanics are just slow, prone to weird issues and often needs hands-on so only thing I'd add here is keep your expectations reasonable, we're not going to have within-the-hour deployments 100% of the time, it may take a few hours to get your systems once the dust clears, longer if hands-on is required to push them through validation like any normal QUADS assignment.

[josecastillolema]

Thanks for the great explanation @sadsfae ,
some newie questions about the scale lab internals:

• When a new cloud is assigned and goes through validation, does all its host get provisioned through foreman? Is this needed?
  • Assuming that the answer to the previous quesiton is yes, and considering our automatic deployment scenario where we only need the bastion node deployed, and the other servers will be handled by the bastion installer, would it be possible to skip the provisioning of the rest of the nodes or the provisioning itself is needed in order to do the validation?
• Could the validation of the servers be done when the previous cloud assignment finishes instead of when the new cloud assignment is released, or is it dependent on the new assignment (i.e.: vlans, etc.)

[sadsfae]

Thanks for the great explanation @sadsfae , some newie questions about the scale lab internals:

• When a new cloud is assigned and goes through validation, does all its host get provisioned through foreman? Is this needed?

Yes, it is absolutely needed. We have no other way to ensure systems data, settings, etc. are cleaned. More importantly we need a way to physically test/validate network functionality. We have a series of fping tests that occur and other validation testing to ensure the hardware is working 100%, traffic passes on ports and also that data and settings from previous tenants are removed. We have to deploy our own Foreman RHEL because it sets up templates for all of the VLAN interfaces with deliberate IP schemes to facilitate this testing.

• Assuming that the answer to the previous quesiton is yes, and considering our automatic deployment scenario where we only need the bastion node deployed, and the other servers will be handled by the bastion installer, would it be possible to skip the provisioning of the rest of the nodes or the provisioning itself is needed in order to do the validation?

Yes but also no. There is no way to perform any validation at all without wiping the systems for a new tenant. We have an option called "no wipe" which skips all provisioning and even reboots, it simply performs the network VLAN and switch automation needed to "move" an assignment to a new cloud. We can allow this as an option via the API phases but you would have to be 100% sure they were the same systems you used before and nobody else used them since you used them last. otherwise you get stuck with effectively the same running OS/systems as whomever had those systems before you, then you'd have to burn extra time provisioning them.

No-wipe is used more frequently in expansion scenarios when we can't be sure there isn't conflicting broadcast services on the internal VLANS that would hijack kickstart (like DHCP/PXE from an internal tenant installer or service) or on occasion when we need to resurrect an expired assignment, it's really not designed to be done on new assignments unless you can be sure of the systems integrity which I don't see really being a tenable or reliable situation in a self-service pool of changing hardware.

We also have no way of knowing what system you may pick ahead of time in the future for your bastion node before you get your systems, as those systems availability would only based on what's available at the current time. The Foreman-deployed OS, while perfectly fine for a generic RHEL is more a vehicle for us to validate hardware and network functionality, ensure clean baselines and catch any hands-on issue that might occur with bare-metal which happens frequently with 1000's of something more than we'd like. There is just no getting around this and it would cause a lot more headache to try skipping it. A environment doesn't get released until all systems pass our sets of validation phases.

• Could the validation of the servers be done when the previous cloud assignment finishes instead of when the new cloud assignment is released, or is it dependent on the new assignment (i.e.: vlans, etc.).

When machines finish their allocation they roll directly to the next tenant anyway if they have an active schedule but they still need to be wiped/validated/tested and then pass our hardware/network validation and release gating, meaning they do immediately go into this process anyway if they have another place to be.

What you're asking in general doesn't save a lot of time anyway. Kickstart via Foreman are a fairly fast part of the QUADS workflow (network automation is the most speedy which takes 5-10 seconds or so per machine).

It takes around 5-8 minutes or less not counting reboots for a modern system to fully kickstart with our local SDN mirror and it's all done in parallel via the asyncio-capable Foreman QUADS library. Even if we used image-based deployments it would still take a comparable amount of time because disk images have to be written and reboots still have to happen and it's more inflexible because we'd have to maintain many different flavors of images to account for hardware variety. In general this step is fundamental to ensuring we deliver and properly validate 100% functional systems to tenants.