MCO-1131: Merge implementation machineOSBuild and machineOSConfig in MCO #4327

inesqyx · 2024-04-17T13:30:56Z

A further cleanup and rebase w/ the main branch on MCO-1042: ocb-api implementation in MCO. It is kept separated instead of merging into @cdoern 's PR because (1) The functionality of @cdoern 's PR has been verified, keeping a good copy of it without direct modifying it would help with future debugging (2) A big rebase to the kube/FG bump change in the master branch has happened in this PR. Bringing it back to @cdoern 's PR will involve a another round of conflict resolution and un-verified changes into the original solution.

For reviewers

If you are interested in seeing what cleanup has been made after @cdoern 's work, please see: MCO-1042-cleanup-only cdoern/machine-config-operator#3
Otherwise, please review this PR, which will highly likely be the one to be merged, TY!

For testing

Please refer to this PR for API details: MCO-993: MachineOSBuild API api#1773
Please refer to this doc for a quick how-to-use/test guide: https://docs.google.com/document/d/1t-Hi9NYNe_Om-Um-iNTQ7k40Sx4KPAAiQQlx8tliVDw/edit?usp=sharing
Please refer to this graph for an API arch visual: https://miro.com/app/board/uXjVKWAyliE=/?share_link_id=11371293891

Current Progress

Code Cleanup: Done, ready for review
Functionality Testing: WIP, not ready for testing
Test: WIP, not ready for review

openshift-ci-robot · 2024-04-17T13:31:00Z

@inesqyx: This pull request references MCO-1042 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

A further cleanup and rebase w/ the main branch on MCO-1042: ocb-api implementation in MCO. It is kept separated instead of merging into @cdoern 's PR because (1) The functionality of @cdoern 's PR has been verified, keeping a good copy of it without direct modifying it would help with future debugging (2) A big rebase to the kube/FG bump change in the master branch has happened in this PR. Bringing it back to @cdoern 's PR will involve a another round of conflict resolution and un-verified changes into the original solution.

For reviewers

If you are interested in seeing what cleanup has been made after @cdoern 's work, please see: MCO-1042-cleanup-only cdoern/machine-config-operator#3

Otherwise, please review this PR, which will highly likely be the one to be merged, TY!

For testing

Please refer to this PR for API details: MCO-993: MachineOSBuild API api#1773

Please refer to this doc for a quick how-to-use/test guide: https://docs.google.com/document/d/1t-Hi9NYNe_Om-Um-iNTQ7k40Sx4KPAAiQQlx8tliVDw/edit?usp=sharing

Please refer to this graph for an API arch visual: https://miro.com/app/board/uXjVKWAyliE=/?share_link_id=11371293891

Current Progress

Code Cleanup: Done, ready for review

Functionality Testing: WIP, not ready for testing

Test: WIP, not ready for review

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2024-04-17T13:32:19Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

pkg/controller/node/node_controller.go

pkg/controller/build/build_controller_test.go

dkhater-redhat · 2024-04-17T18:13:02Z

cmd/machine-os-builder/start.go

- <-ctx.Done()
+ // is this... allowed?
+ // since users can specify different settings per pool, we need to run a controller PER pool. Otherwise, settings will be conflated, as will failures and builds.
+ for _, ctrl := range controllers {


so if im understanding this correctly, we now potentially return multiple build controllers, each tied to a different MachineOSConfig? remind me why we made this change again?

pkg/apihelpers/machineosbuild_apihelpers.go

dkhater-redhat · 2024-04-17T18:47:15Z

pkg/controller/common/layered_node_state.go

+ }
+
+ // If the pool is not layered, this annotation should not exist.
+ return val == "" || !ok


do we need distinguish between a legitimately empty/missing value and an erroneous state. i remember running into an issue like this on the revert work and needed to differentiate

@cdoern look at this!!

pkg/controller/common/layered_node_state.go

pkg/controller/common/mos_state.go

yuqi-zhang

Leaving some comments from our live debug session

pkg/controller/node/status.go

pkg/controller/node/node_controller.go

yuqi-zhang · 2024-04-22T18:56:00Z

pkg/controller/node/node_controller.go

+ return nil, nil, err
+ }
+
+ for _, config := range configList.Items {


consider looking for duplicate pool name, and either error or parse

yuqi-zhang · 2024-04-22T18:56:30Z

pkg/controller/node/node_controller.go

+
+ for _, build := range buildList.Items {
+ if build.Spec.MachineOSConfig.Name == ourConfig.Name {
+ if build.Spec.DesiredConfig.Name == pool.Spec.Configuration.Name {


consider disambigufy since we may have the same machineconfig render, but different base image in a build

pkg/controller/node/status.go

install/0000_80_machine-config_01_machineosbuild-TechPreviewNoUpgrade.crd.yaml

yuqi-zhang · 2024-04-24T00:12:33Z

Need a make go-deps I think

inesqyx · 2024-04-24T12:39:17Z

/retest-required

cheesesashimi

@inesqyx / @umohnani8 / @cdoern I took the closest look at this that I could before my PTO. After writing e2e tests and getting better acquainted with this code, I was better able to find a few issues that should be at least discussed, if not addressed completely.

cmd/machine-os-builder/start.go

cheesesashimi · 2024-04-27T03:34:05Z

cmd/machine-os-builder/start.go

- go ctrl.Run(ctx, 5)
- <-ctx.Done()
+ // is this... allowed?
+ // since users can specify different settings per pool, we need to run a controller PER pool. Otherwise, settings will be conflated, as will failures and builds.


question: What happens when a MachineOSConfig is added after we've started the BuildControllers? How does it get a BuildController instance assigned to it? Conversely, if one of several MachineOSConfigs is then deleted, how does the BuildController associated with it get stopped?

pkg/controller/build/build_controller.go

pkg/controller/build/image_build_request.go

pkg/controller/node/node_controller.go

pkg/controller/build/build_controller.go

pkg/controller/build/assets/Dockerfile.on-cluster-build-template

cheesesashimi · 2024-04-27T17:25:01Z

cmd/machine-os-builder/start.go


- return build.NewWithCustomPodBuilder(cfg, buildClients), nil
+ for range machineOSConfigs.Items {
+ controllersToStart = append(controllersToStart, build.NewWithCustomPodBuilder(cfg, buildClients))


issue (blocking): If the intent is to have a single BuildController handle a single MachineOSConfig, how does a BuildController instance know which MachineOSConfig to handle? It doesn't look like the instantiated BuildController has any knowledge about a particular MachineOSConfig.

Until we have a clear answer on how these get bound to a specific BuildController instance, we may want to only start a single BuildController instance. The reason is because when multiple MachineOSConfigs are created, the multiple BuildController instances don't know which MachineOSConfig to target, so they target whichever one has an event associated with it. This results in a lot of log messages like this:

I0506 20:45:50.806184 1 build_controller.go:471] Error syncing machineosbuild pool-0: machineosbuilds.machineconfiguration.openshift.io "pool-0-rendered-pool-0-aff68db9c6081b0d56cd746090f860c0-builder" already exists I0506 20:45:50.847069 1 build_controller.go:484] Started syncing build "pool-0" (2024-05-06 20:45:50.847055006 +0000 UTC m=+543.347506299) I0506 20:45:50.876267 1 build_controller.go:486] Finished syncing machineOSBuilder "pool-0" (29.201665ms) I0506 20:45:50.878236 1 build_controller.go:471] Error syncing machineosbuild pool-0: machineosbuilds.machineconfiguration.openshift.io "pool-0-rendered-pool-0-aff68db9c6081b0d56cd746090f860c0-builder" already exists I0506 20:45:50.971138 1 build_controller.go:484] Started syncing build "pool-0" (2024-05-06 20:45:50.971123152 +0000 UTC m=+543.471574548) I0506 20:45:50.984172 1 build_controller.go:486] Finished syncing machineOSBuilder "pool-0" (13.039368ms) E0506 20:45:50.984291 1 build_controller.go:476] machineosbuilds.machineconfiguration.openshift.io "pool-0-rendered-pool-0-aff68db9c6081b0d56cd746090f860c0-builder" already exists I0506 20:45:50.984316 1 build_controller.go:477] Dropping machineosbuild "pool-0" out of the queue: machineosbuilds.machineconfiguration.openshift.io "pool-0-rendered-pool-0-aff68db9c6081b0d56cd746090f860c0-builder" already exists

Basically, all of the BuildControllers react to a change, create a new object, etc. The first one to effect the change "wins". The rest end up returning errors until they drop the request from their queues. While this doesn't appear to have any effect on how builds are started and lifecycled, each of those messages has numerous API calls associated with it. In resource-constrained environments such as SNO, that can be problematic since we have more API calls than are necessary. As a side-effect, other controllers that watch for MachineOSBuilds / MachineOSConfigs can also react multiple time and cause unintended side-effects.

EDIT: I tested a version of this code that only has a single BuildController instance and everything seemed to work fine without causing the additional API requests.

Signed-off-by: Urvashi Mohnani <[email protected]>

This adds the capability for BuildController to use the RHEL entitlement secrets to allow cluster admins to inject RHEL content into their builds that they are entitled to receive. This also allows the injection / consumption of content into /etc/yum.repos.d as well as /etc/pki/rpm-gpg. There are a few notes about the implementation that I would like to have at a higher level: - Because we run rootless Buildah, we're more prone to running into SELinux complications. This makes it more difficult to directly mount the contents of /etc/yum.repos.d, /etc/pki/entitlement, and /etc/pki/rpm-gpg directly into the build context. With that in mind, we copy everything into a series of temp directories first, and then mount those temp directories into the build context as a volume. - We also create an emptyDir which is mounted into the build pod at /home/build/.local/share/containers. It is unclear why this is necessary, but as mentioned before, I suspect that this is due to SELinux issues. - The e2e test suite now has the capability to stream the container logs from the build pod to the filesystem as there is useful information contained within those logs if the e2e test fails. In OpenShift CI, this location will be determined by the ARTIFACT_DIR env var. If this env var is not present, it will default the current directory. - For right now, etc-pki-entitlement flow (specifically, the TestEntitledBuild test) is being skipped in OpenShift CI because the test clusters do not have that cred available. The test suite will automatically detect the presence (or lack thereof) of that cred in the openshift-config-managed namespace and run the test if it is available. However, the TestYumRepos test targets a very similar flow and can do its own setup and teardown regardless of creds preexisting. Additionally, I took care to ensure that this does not break OKD by taking the following actions: - I observed that the addition of the /home/build/.local/share/containers volume mount to the build pod prevented the wait-for-done container to start when running on FCOS. With this in mind, I modified the build pod instantiation to not connect this volume mount to the wait-for-done container. - I added a TestOnClusterBuildsOnOKD e2e test which will only run against an OKD cluster. Conversely, I excluded other tests from running against an OKD cluster since those tests make assumptions about things that would only be present within an OCP cluster.

sergiordlr · 2024-05-08T13:39:26Z

Pre-merge testing has been tracked here: https://issues.redhat.com/browse/MCO-1149

/label qe-approved

openshift-ci-robot · 2024-05-08T13:40:05Z

@inesqyx: This pull request references MCO-1131 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

A further cleanup and rebase w/ the main branch on MCO-1042: ocb-api implementation in MCO. It is kept separated instead of merging into @cdoern 's PR because (1) The functionality of @cdoern 's PR has been verified, keeping a good copy of it without direct modifying it would help with future debugging (2) A big rebase to the kube/FG bump change in the master branch has happened in this PR. Bringing it back to @cdoern 's PR will involve a another round of conflict resolution and un-verified changes into the original solution.

For reviewers

If you are interested in seeing what cleanup has been made after @cdoern 's work, please see: MCO-1042-cleanup-only cdoern/machine-config-operator#3

Otherwise, please review this PR, which will highly likely be the one to be merged, TY!

For testing

Please refer to this PR for API details: MCO-993: MachineOSBuild API api#1773

Please refer to this doc for a quick how-to-use/test guide: https://docs.google.com/document/d/1t-Hi9NYNe_Om-Um-iNTQ7k40Sx4KPAAiQQlx8tliVDw/edit?usp=sharing

Please refer to this graph for an API arch visual: https://miro.com/app/board/uXjVKWAyliE=/?share_link_id=11371293891

Current Progress

Code Cleanup: Done, ready for review

Functionality Testing: WIP, not ready for testing

Test: WIP, not ready for review

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

cheesesashimi · 2024-05-09T14:58:54Z

pkg/controller/build/build_controller.go

+// Determines if we should run a build, then starts a build pod to perform the
+// build, and updates the MachineConfigPool with an object reference for the
+// build pod.
+func (ctrl *Controller) startBuildForMachineConfigPool(mosc *mcfgv1alpha1.MachineOSConfig, mosb *mcfgv1alpha1.MachineOSBuild) error {


suggestion (non-blocking): We should change the name of this method to something like startBuild or similar since the MachineConfigPool is no longer the source of truth for build state.

(We can address this later though.)

cheesesashimi · 2024-05-09T15:02:45Z

pkg/controller/build/build_controller.go

- IsBuildRunning(*mcfgv1.MachineConfigPool) (bool, error)
-}, oldPool, curPool *mcfgv1.MachineConfigPool) (bool, error) {
- ps := newPoolState(curPool)
+func (ctrl *Controller) doesMOSBExist(mosc *mcfgv1alpha1.MachineOSConfig) (*mcfgv1alpha1.MachineOSBuild, bool) {


praise: Nicely done with this!

We'll probably want to make a more canonical shared implementation of this in the future since anything that works with MachineOSConfigs and MachineOSBuilds will need to perform similar operations. That doesn't need to be done at this time though.

cheesesashimi · 2024-05-09T15:04:35Z

pkg/controller/build/build_controller.go


- if !poolStateSuggestsBuild {
- return false, nil
+func (ctrl *Controller) getConfigAndBuildForPool(pool *mcfgv1.MachineConfigPool) (*mcfgv1alpha1.MachineOSConfig, *mcfgv1alpha1.MachineOSBuild, error) {


praise: Again, nicely done with this!

As mentioned before, we'll probably want to hoist this out of this package in the future. But that does not need to be done right now.

cheesesashimi · 2024-05-09T15:06:30Z

pkg/controller/build/build_controller_test.go

- return false
- }
- }
+ // for _, condition := range getMachineConfigPoolBuildConditions() {


suggestion (non-blocking): If this is commented out, we should probably just remove it.

cheesesashimi · 2024-05-09T15:08:10Z

pkg/controller/build/image_build_controller.go

@@ -1,335 +0,0 @@
-package build


note: I'm not sad to see this go away 😄

cheesesashimi · 2024-05-09T15:10:38Z

pkg/controller/build/image_build_request.go

- RunAsUser: &uid,
- RunAsGroup: &gid,
+ // Octal: 0755.
+ var mountMode int32 = 493


note (non-blocking): One can use an octal literal by doing something like var mountMode int32 = 0o755. Not sure if that would work here, but it's just a thought.

(This does not need to be addressed right now though.)

cheesesashimi · 2024-05-09T15:12:04Z

pkg/controller/build/image_build_request.go

@@ -513,7 +433,11 @@ func (i ImageBuildRequest) toBuildahPod() *corev1.Pod {
 Command: append(command, buildahBuildScript),
 ImagePullPolicy: corev1.PullAlways,
 SecurityContext: securityContext,
- VolumeMounts: volumeMounts,
+ // Only attach the buildah-cache volume mount to the buildah container.


note: In the future, I want to be more decisive about what volume mounts, env vars, etc., are attached to which container.

(Does not need to be addressed right now)

pkg/controller/build/build_controller.go

cheesesashimi · 2024-05-09T15:23:26Z

pkg/controller/common/layered_pool_state_test.go

@@ -1,96 +1,86 @@
 package common

-import (
- "testing"
+// func TestMachineOSBuildState(t *testing.T) {


note: I want to get these tests running again.

(Does not need to be addressed right now because the existing test suites / etc. exercise most of the paths here. I don't think the code coverage analysis takes that into consideration though.)

cheesesashimi · 2024-05-09T15:29:48Z

pkg/controller/node/node_controller.go

@@ -1079,8 +1214,8 @@ func getAllCandidateMachines(pool *mcfgv1.MachineConfigPool, nodesInPool []*core
 }

 // getCandidateMachines returns the maximum subset of nodes which can be updated to the target config given availability constraints.
-func getCandidateMachines(pool *mcfgv1.MachineConfigPool, nodesInPool []*corev1.Node, maxUnavailable int) []*corev1.Node {
- nodes, capacity := getAllCandidateMachines(pool, nodesInPool, maxUnavailable)
+func getCandidateMachines(pool *mcfgv1.MachineConfigPool, config *mcfgv1alpha1.MachineOSConfig, build *mcfgv1alpha1.MachineOSBuild, nodesInPool []*corev1.Node, maxUnavailable int, layered bool) []*corev1.Node {


suggestion (non-blocking): Make getCandidateMachines() accept a struct like candidateMachineOpts or similar. This makes it very clear at each call-site what each parameter is for.

(This does not need to be done right now though.)

cheesesashimi · 2024-05-09T15:31:06Z

pkg/controller/node/node_controller_test.go

@@ -1046,44 +1053,49 @@ func TestShouldMakeProgress(t *testing.T) {
 expectTaintsAddPatch: false,
 },
 {
- description: "node not at desired config, patch on annotation and taints",
+ description: "node not at desired config, patch on annotation and taints", // failing


question: Is this test still failing? We may want to remove the // failing comment if so.

cheesesashimi · 2024-05-09T15:34:03Z

pkg/controller/node/status.go

+ var updating int32
+ for _, set := range mcs.Status.PinnedImageSets {
+ if set.CurrentGeneration != set.DesiredGeneration {
+ updating++


suggestion (non-blocking): Once we encounter the first condition where set.CurrentGeneration != set.DesiredGeneration, we could probably return true early since we don't need to examine the rest of the conditions. Only once we've examined all of the conditions and found none to be unequal, we can return false.

(Does not need to be addressed right now.)

cheesesashimi · 2024-05-09T15:36:43Z

pkg/operator/operator.go

+ }
+
+ // this is for the FG
+ // informers = append(informers, moscInformer.Informer())


question: Is this informer still being instantiated?

EDIT: Instantiating it is unnecessary because all of the functions that handle the Machine OS Builder check for the presence of the featuregate before doing anything.

cheesesashimi · 2024-05-09T16:39:04Z

/lgtm
/approval

openshift-ci · 2024-05-09T16:39:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cheesesashimi, inesqyx, sinnykumari

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [cheesesashimi,sinnykumari]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2024-05-10T11:59:53Z

[ART PR BUILD NOTIFIER]

This PR has been included in build ose-machine-config-operator-container-v4.16.0-202405100946.p0.gdeebbcb.assembly.stream.el9 for distgit ose-machine-config-operator.
All builds following this will include this PR.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Apr 17, 2024

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 17, 2024

sinnykumari reviewed Apr 17, 2024

View reviewed changes

pkg/controller/node/node_controller.go Show resolved Hide resolved

sinnykumari reviewed Apr 18, 2024

View reviewed changes

pkg/controller/node/node_controller.go Show resolved Hide resolved

inesqyx force-pushed the MCO-1042-2 branch from 38424a1 to dff9755 Compare April 18, 2024 16:40

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 18, 2024

dkhater-redhat reviewed Apr 18, 2024

View reviewed changes

inesqyx force-pushed the MCO-1042-2 branch from dff9755 to 929c161 Compare April 22, 2024 14:57

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 22, 2024

yuqi-zhang reviewed Apr 22, 2024

View reviewed changes

inesqyx marked this pull request as ready for review April 23, 2024 06:13

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 23, 2024

openshift-ci bot requested review from cdoern and dkhater-redhat April 23, 2024 06:14

inesqyx force-pushed the MCO-1042-2 branch 4 times, most recently from 7399b1c to f368c74 Compare April 23, 2024 20:44

cheesesashimi mentioned this pull request Apr 23, 2024

MCO-1100: enable RHEL entitlements in on-cluster layering with OCL API #4333

Closed

sinnykumari force-pushed the MCO-1042-2 branch from b65f9d1 to 976bc65 Compare April 24, 2024 09:29

inesqyx force-pushed the MCO-1042-2 branch from 976bc65 to fbbb0ab Compare April 24, 2024 09:49

inesqyx force-pushed the MCO-1042-2 branch 2 times, most recently from 805b215 to ba183da Compare April 25, 2024 19:05

cheesesashimi suggested changes Apr 27, 2024

View reviewed changes

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 27, 2024

cheesesashimi reviewed Apr 27, 2024

View reviewed changes

umohnani8 and others added 4 commits May 8, 2024 07:57

Test fixes for the new OCB API

084a4e3

Signed-off-by: Urvashi Mohnani <[email protected]>

Build controller test fixes for the new OCB API

1da14cc

Final cleanup

e8e945c

inesqyx force-pushed the MCO-1042-2 branch from e4035e3 to e8e945c Compare May 8, 2024 11:58

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label May 8, 2024

cheesesashimi reviewed May 9, 2024

View reviewed changes

pkg/controller/build/image_build_controller.go

@@ -1,335 +0,0 @@

package build

Copy link

Member

cheesesashimi May 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: I'm not sad to see this go away 😄

cheesesashimi reviewed May 9, 2024

View reviewed changes

pkg/controller/build/build_controller.go Show resolved Hide resolved

cheesesashimi reviewed May 9, 2024

View reviewed changes

openshift-ci bot assigned cheesesashimi May 9, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 9, 2024

openshift-merge-bot bot merged commit deebbcb into openshift:master May 9, 2024
15 checks passed

This was referenced May 13, 2024

adds ocb api e2e tests #4328

Closed

MCO-1100: enable RHEL entitlements in on-cluster layering #4312

Closed

inesqyx mentioned this pull request Jul 24, 2024

MCO-790: Set up leader election in machine-os-builder #4302

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MCO-1131: Merge implementation machineOSBuild and machineOSConfig in MCO #4327

MCO-1131: Merge implementation machineOSBuild and machineOSConfig in MCO #4327

inesqyx commented Apr 17, 2024

openshift-ci-robot commented Apr 17, 2024 •

edited by openshift-ci bot

Loading

openshift-ci bot commented Apr 17, 2024

dkhater-redhat Apr 17, 2024

dkhater-redhat Apr 17, 2024

cdoern Apr 23, 2024

yuqi-zhang left a comment

yuqi-zhang Apr 22, 2024

yuqi-zhang Apr 22, 2024

yuqi-zhang commented Apr 24, 2024

inesqyx commented Apr 24, 2024

cheesesashimi left a comment

cheesesashimi Apr 27, 2024

cheesesashimi Apr 27, 2024

cheesesashimi May 6, 2024

sergiordlr commented May 8, 2024

openshift-ci-robot commented May 8, 2024 •

edited by openshift-ci bot

Loading

cheesesashimi May 9, 2024

cheesesashimi May 9, 2024 •

edited

Loading

cheesesashimi May 9, 2024

cheesesashimi May 9, 2024

cheesesashimi May 9, 2024

cheesesashimi May 9, 2024

cheesesashimi May 9, 2024

cheesesashimi May 9, 2024 •

edited

Loading

cheesesashimi May 9, 2024

cheesesashimi May 9, 2024

cheesesashimi May 9, 2024

cheesesashimi May 9, 2024 •

edited

Loading

cheesesashimi commented May 9, 2024

openshift-ci bot commented May 9, 2024

openshift-bot commented May 10, 2024

MCO-1131: Merge implementation machineOSBuild and machineOSConfig in MCO #4327

MCO-1131: Merge implementation machineOSBuild and machineOSConfig in MCO #4327

Conversation

inesqyx commented Apr 17, 2024

openshift-ci-robot commented Apr 17, 2024 • edited by openshift-ci bot Loading

openshift-ci bot commented Apr 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuqi-zhang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuqi-zhang commented Apr 24, 2024

inesqyx commented Apr 24, 2024

cheesesashimi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sergiordlr commented May 8, 2024

openshift-ci-robot commented May 8, 2024 • edited by openshift-ci bot Loading

Choose a reason for hiding this comment

cheesesashimi May 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cheesesashimi May 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cheesesashimi May 9, 2024 • edited Loading

Choose a reason for hiding this comment

cheesesashimi commented May 9, 2024

openshift-ci bot commented May 9, 2024

openshift-bot commented May 10, 2024

openshift-ci-robot commented Apr 17, 2024 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented May 8, 2024 •

edited by openshift-ci bot

Loading

cheesesashimi May 9, 2024 •

edited

Loading

cheesesashimi May 9, 2024 •

edited

Loading

cheesesashimi May 9, 2024 •

edited

Loading