[release-4.20] OCPBUGS-76339: machine-config-daemon: openshift: Exposure of Sensitive Data in Log Files in the Machine Configuration Daemon. [openshift-4] by openshift-cherrypick-robot · Pull Request #5631 · openshift/machine-config-operator

openshift-cherrypick-robot · 2026-02-06T15:16:21Z

This is an automated cherry-pick of #5624

/assign dkhater-redhat

openshift-ci-robot · 2026-02-06T15:16:32Z

@openshift-cherrypick-robot: Jira Issue OCPBUGS-76271 has been cloned as Jira Issue OCPBUGS-76339. Will retitle bug to link to clone.
/retitle [release-4.20] OCPBUGS-76339: machine-config-daemon: openshift: Exposure of Sensitive Data in Log Files in the Machine Configuration Daemon. [openshift-4]

Details

In response to this:

This is an automated cherry-pick of #5624

/assign dkhater-redhat

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2026-02-06T15:16:46Z

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-76339, which is invalid:

expected dependent Jira Issue OCPBUGS-76271 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

This is an automated cherry-pick of #5624

/assign dkhater-redhat

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

dkhater-redhat · 2026-02-06T15:27:16Z

/approve

dkhater-redhat · 2026-02-06T17:01:16Z

/retest-required

dkhater-redhat · 2026-02-09T21:12:48Z

/retest-required

ptalgulk01 · 2026-02-10T11:00:56Z

Pre-merge verified
Environment Setup
Version-4.20.0-0-2026-02-10-094806-test-ci-ln-7czc6ht-latest
Platform- AWS

Verification Steps:

Step 1: Verify Baseline State

$ oc get node ip-10-0-0-253.us-west-1.compute.internal  -o jsonpath='{.metadata.annotations.machineconfiguration\.openshift\.io/state}' && echo
Done
                                                                                                             
$  oc exec machine-config-daemon-hzf8m  -- curl -s localhost:8797/metrics 2>/dev/null | grep mcd_config_drift
# HELP mcd_config_drift timestamp for config drift
# TYPE mcd_config_drift gauge
mcd_config_drift 0

Step 2: Create Config Drift on Sensitive File
Modified the sensitive kubelet configuration file to trigger drift detection:

$ oc debug node/ip-10-0-0-253.us-west-1.compute.internal -- chroot  /host bash -c "echo '# SENSITIVE DATA LEAK TEST - PR5601' >>/etc/kubernetes/kubelet.conf" 
Starting pod/ip-10-0-0-253us-west-1computeinternal-debug-67xxk ...
To use host binaries, run `chroot /host`

Removing debug pod ...

Step 3: Verify Drift Detection

$ oc exec machine-config-daemon-hzf8m  -- curl -s localhost:8797/metrics 2>/dev/null | grep mcd_config_drift
# HELP mcd_config_drift timestamp for config drift
# TYPE mcd_config_drift gauge
mcd_config_drift 1.7707205746145768e+09

$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-f2e0652a5e53632feb9cf8bfdd7032da   True      False      False      3              3                   3                     0                      37m
worker   rendered-worker-93bec14cbf915cb962de836f208f588e   False     True       True       3              2                   2                     1                      37m

Step 4: Analyze MCD Logs for Sensitive Data Exposure

$ oc logs machine-config-daemon-hzf8m  --since=5m 2>&1     
Defaulted container "machine-config-daemon" out of: machine-config-daemon, kube-rbac-proxy
E0210 10:49:34.614555    2836 on_disk_validation.go:251] content mismatch for file "/etc/kubernetes/kubelet.conf" (expected 4827 bytes, got 4863 bytes)
E0210 10:49:34.614598    2836 daemon.go:1503] content mismatch for file "/etc/kubernetes/kubelet.conf"
E0210 10:49:34.614626    2836 writer.go:231] Marking Degraded due to: "content mismatch for file \"/etc/kubernetes/kubelet.conf\""
I0210 10:49:35.626941    2836 daemon.go:793] Transitioned from state: Done -> Degraded
I0210 10:49:35.627038    2836 daemon.go:796] Transitioned from degraded/unreconcilable reason  -> content mismatch for file "/etc/kubernetes/kubelet.conf"
W0210 10:49:35.631858    2836 daemon.go:2553] current+desiredConfig is rendered-worker-93bec14cbf915cb962de836f208f588e but state is Degraded
I0210 10:49:35.652346    2836 command_runner.go:24] Running captured: rpm-ostree kargs
E0210 10:49:35.786996    2836 on_disk_validation.go:251] content mismatch for file "/etc/kubernetes/kubelet.conf" (expected 4827 bytes, got 4863 bytes)
E0210 10:49:35.787045    2836 daemon.go:987] Preflight config drift check failed: unexpected on-disk state validating against rendered-worker-93bec14cbf915cb962de836f208f588e: content mismatch for file "/etc/kubernetes/kubelet.conf"
E0210 10:49:35.787059    2836 writer.go:231] Marking Degraded due to: "unexpected on-disk state validating against rendered-worker-93bec14cbf915cb962de836f208f588e: content mismatch for file \"/etc/kubernetes/kubelet.conf\""
I0210 10:49:37.799919    2836 daemon.go:796] Transitioned from degraded/unreconcilable reason content mismatch for file "/etc/kubernetes/kubelet.conf" -> unexpected on-disk state validating against rendered-worker-93bec14cbf915cb962de836f208f588e: content mismatch for file "/etc/kubernetes/kubelet.conf"

Step 5: Verify No Sensitive Content Leaked
Searched for patterns that would indicate sensitive data exposure:

$ oc logs machine-config-daemon-hzf8m --since=5m 2>&1    | grep  -E "uint8|want \+got|apiVersion|authentication|cmp.Diff"
$ oc logs machine-config-controller-fd49f477b-gxzhq--since=5m 2>&1    | grep  -E "uint8|want \+got|apiVersion|authentication|cmp.Diff"     
$ oc logs machine-config-controller-fd49f477b-gxzh --since=5m 2>&1    | grep  -E "uint8|want \+got|apiVersion|authentication|cmp.Diff"

Cleanup & Restoration

$ oc debug node/ip-10-0-0-253.us-west-1.compute.internal -- chroot  /host  bash -c "sed -i '/# SENSITIVE DATA LEAK TEST - PR5601/d' /etc/kubernetes/kubelet.conf" 
Starting pod/ip-10-0-0-253us-west-1computeinternal-debug-rcq2r ...
To use host binaries, run `chroot /host`

Removing debug pod ...

$ oc exec machine-config-daemon-hzf8m  -- curl -s localhost:8797/metrics 2>/dev/null | grep mcd_config_drift
# HELP mcd_config_drift timestamp for config drift
# TYPE mcd_config_drift gauge
mcd_config_drift 0

$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-f2e0652a5e53632feb9cf8bfdd7032da   True      False      False      3              3                   3                     0                      37m
worker   rendered-worker-93bec14cbf915cb962de836f208f588e   True      False      False      3              3                   3                     0                      37m

/label qe-approved
/verified by @ptalgulk01

openshift-ci-robot · 2026-02-10T11:01:03Z

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-76339, which is invalid:

expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

This is an automated cherry-pick of #5624

/assign dkhater-redhat

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2026-02-10T11:01:10Z

@ptalgulk01: This PR has been marked as verified by @ptalgulk01.

Details

In response to this:

Pre-merge verified
Environment Setup
Version-4.20.0-0-2026-02-10-094806-test-ci-ln-7czc6ht-latest
Platform- AWS

Verification Steps:

Step 1: Verify Baseline State

$ oc get node ip-10-0-0-253.us-west-1.compute.internal  -o jsonpath='{.metadata.annotations.machineconfiguration\.openshift\.io/state}' && echo
Done
                                                                                                            
$  oc exec machine-config-daemon-hzf8m  -- curl -s localhost:8797/metrics 2>/dev/null | grep mcd_config_drift
# HELP mcd_config_drift timestamp for config drift
# TYPE mcd_config_drift gauge
mcd_config_drift 0

Step 2: Create Config Drift on Sensitive File
Modified the sensitive kubelet configuration file to trigger drift detection:

$ oc debug node/ip-10-0-0-253.us-west-1.compute.internal -- chroot  /host bash -c "echo '# SENSITIVE DATA LEAK TEST - PR5601' >>/etc/kubernetes/kubelet.conf" 
Starting pod/ip-10-0-0-253us-west-1computeinternal-debug-67xxk ...
To use host binaries, run `chroot /host`

Removing debug pod ...

Step 3: Verify Drift Detection

$ oc exec machine-config-daemon-hzf8m  -- curl -s localhost:8797/metrics 2>/dev/null | grep mcd_config_drift
# HELP mcd_config_drift timestamp for config drift
# TYPE mcd_config_drift gauge
mcd_config_drift 1.7707205746145768e+09

$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-f2e0652a5e53632feb9cf8bfdd7032da   True      False      False      3              3                   3                     0                      37m
worker   rendered-worker-93bec14cbf915cb962de836f208f588e   False     True       True       3              2                   2                     1                      37m

Step 4: Analyze MCD Logs for Sensitive Data Exposure

$ oc logs machine-config-daemon-hzf8m  --since=5m 2>&1     
Defaulted container "machine-config-daemon" out of: machine-config-daemon, kube-rbac-proxy
E0210 10:49:34.614555    2836 on_disk_validation.go:251] content mismatch for file "/etc/kubernetes/kubelet.conf" (expected 4827 bytes, got 4863 bytes)
E0210 10:49:34.614598    2836 daemon.go:1503] content mismatch for file "/etc/kubernetes/kubelet.conf"
E0210 10:49:34.614626    2836 writer.go:231] Marking Degraded due to: "content mismatch for file \"/etc/kubernetes/kubelet.conf\""
I0210 10:49:35.626941    2836 daemon.go:793] Transitioned from state: Done -> Degraded
I0210 10:49:35.627038    2836 daemon.go:796] Transitioned from degraded/unreconcilable reason  -> content mismatch for file "/etc/kubernetes/kubelet.conf"
W0210 10:49:35.631858    2836 daemon.go:2553] current+desiredConfig is rendered-worker-93bec14cbf915cb962de836f208f588e but state is Degraded
I0210 10:49:35.652346    2836 command_runner.go:24] Running captured: rpm-ostree kargs
E0210 10:49:35.786996    2836 on_disk_validation.go:251] content mismatch for file "/etc/kubernetes/kubelet.conf" (expected 4827 bytes, got 4863 bytes)
E0210 10:49:35.787045    2836 daemon.go:987] Preflight config drift check failed: unexpected on-disk state validating against rendered-worker-93bec14cbf915cb962de836f208f588e: content mismatch for file "/etc/kubernetes/kubelet.conf"
E0210 10:49:35.787059    2836 writer.go:231] Marking Degraded due to: "unexpected on-disk state validating against rendered-worker-93bec14cbf915cb962de836f208f588e: content mismatch for file \"/etc/kubernetes/kubelet.conf\""
I0210 10:49:37.799919    2836 daemon.go:796] Transitioned from degraded/unreconcilable reason content mismatch for file "/etc/kubernetes/kubelet.conf" -> unexpected on-disk state validating against rendered-worker-93bec14cbf915cb962de836f208f588e: content mismatch for file "/etc/kubernetes/kubelet.conf"

Step 5: Verify No Sensitive Content Leaked
Searched for patterns that would indicate sensitive data exposure:

$ oc logs machine-config-daemon-hzf8m --since=5m 2>&1    | grep  -E "uint8|want \+got|apiVersion|authentication|cmp.Diff"
$ oc logs machine-config-controller-fd49f477b-gxzhq--since=5m 2>&1    | grep  -E "uint8|want \+got|apiVersion|authentication|cmp.Diff"     
$ oc logs machine-config-controller-fd49f477b-gxzh --since=5m 2>&1    | grep  -E "uint8|want \+got|apiVersion|authentication|cmp.Diff"

Cleanup & Restoration

$ oc debug node/ip-10-0-0-253.us-west-1.compute.internal -- chroot  /host  bash -c "sed -i '/# SENSITIVE DATA LEAK TEST - PR5601/d' /etc/kubernetes/kubelet.conf" 
Starting pod/ip-10-0-0-253us-west-1computeinternal-debug-rcq2r ...
To use host binaries, run `chroot /host`

Removing debug pod ...

$ oc exec machine-config-daemon-hzf8m  -- curl -s localhost:8797/metrics 2>/dev/null | grep mcd_config_drift
# HELP mcd_config_drift timestamp for config drift
# TYPE mcd_config_drift gauge
mcd_config_drift 0

$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-f2e0652a5e53632feb9cf8bfdd7032da   True      False      False      3              3                   3                     0                      37m
worker   rendered-worker-93bec14cbf915cb962de836f208f588e   True      False      False      3              3                   3                     0                      37m

/label qe-approved
/verified by @ptalgulk01

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

isabella-janssen · 2026-02-10T13:26:45Z

/jira refresh

openshift-ci-robot · 2026-02-10T13:26:53Z

@isabella-janssen: This pull request references Jira Issue OCPBUGS-76339, which is valid.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.20.z) matches configured target version for branch (4.20.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
release note type set to "Release Note Not Required"
dependent bug Jira Issue OCPBUGS-76271 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-76271 targets the "4.21.z" version, which is one of the valid target versions: 4.21.0, 4.21.z
bug has dependents

Requesting review from QA contact:
/cc @sergiordlr

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

isabella-janssen

/lgtm
/label backport-risk-assessed

openshift-ci · 2026-02-10T13:28:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dkhater-redhat, isabella-janssen, openshift-cherrypick-robot

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [dkhater-redhat,isabella-janssen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dkhater-redhat · 2026-02-10T17:51:09Z

/retest-required

dkhater-redhat · 2026-02-11T16:36:45Z

/test e2e-hypershift

dkhater-redhat · 2026-02-12T15:43:40Z

/test unit

openshift-ci · 2026-02-12T15:51:07Z

@openshift-cherrypick-robot: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot · 2026-02-12T15:56:12Z

@openshift-cherrypick-robot: Jira Issue Verification Checks: Jira Issue OCPBUGS-76339
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-76339 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

This is an automated cherry-pick of #5624

/assign dkhater-redhat

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

dkhater-redhat · 2026-02-12T21:55:45Z

/cherry-pick release-4.19

openshift-cherrypick-robot · 2026-02-12T21:56:39Z

@dkhater-redhat: new pull request created: #5658

Details

In response to this:

/cherry-pick release-4.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

fixes exposure of sensitive data in log files

5549421

openshift-cherrypick-robot mentioned this pull request Feb 6, 2026

[release-4.21] OCPBUGS-76271: machine-config-daemon: openshift: Exposure of Sensitive Data in Log Files in the Machine Configuration Daemon. [openshift-4] #5624

Merged

openshift-cherrypick-robot assigned dkhater-redhat Feb 6, 2026

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Feb 6, 2026

openshift-ci bot requested review from RishabhSaini and umohnani8 February 6, 2026 15:16

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 6, 2026

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Feb 10, 2026

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Feb 10, 2026

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Feb 10, 2026

openshift-ci bot requested a review from sergiordlr February 10, 2026 13:27

isabella-janssen reviewed Feb 10, 2026

View reviewed changes

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Feb 10, 2026

openshift-ci bot assigned anuragthehatter, imatza-rh, lyman9966 and mhanss Feb 10, 2026

openshift-ci bot assigned ptalgulk01, rbbratta, rioliu-rh, sergiordlr and zhaozhanqi Feb 10, 2026

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 10, 2026

openshift-merge-bot bot merged commit 0999f17 into openshift:release-4.20 Feb 12, 2026
16 checks passed

openshift-cherrypick-robot mentioned this pull request Feb 12, 2026

[release-4.19] OCPBUGS-76831: machine-config-daemon: openshift: Exposure of Sensitive Data in Log Files in the Machine Configuration Daemon. [openshift-4] #5658

Open

Conversation

openshift-cherrypick-robot commented Feb 6, 2026

Uh oh!

openshift-ci-robot commented Feb 6, 2026

Uh oh!

openshift-ci-robot commented Feb 6, 2026

Uh oh!

dkhater-redhat commented Feb 6, 2026

Uh oh!

dkhater-redhat commented Feb 6, 2026

Uh oh!

dkhater-redhat commented Feb 9, 2026

Uh oh!

ptalgulk01 commented Feb 10, 2026

Uh oh!

openshift-ci-robot commented Feb 10, 2026

Uh oh!

openshift-ci-robot commented Feb 10, 2026

Uh oh!

isabella-janssen commented Feb 10, 2026

Uh oh!

openshift-ci-robot commented Feb 10, 2026

Uh oh!

isabella-janssen left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Feb 10, 2026

Uh oh!

dkhater-redhat commented Feb 10, 2026

Uh oh!

dkhater-redhat commented Feb 11, 2026

Uh oh!

dkhater-redhat commented Feb 12, 2026

Uh oh!

openshift-ci bot commented Feb 12, 2026

Uh oh!

Uh oh!

openshift-ci-robot commented Feb 12, 2026

Uh oh!

dkhater-redhat commented Feb 12, 2026

Uh oh!

openshift-cherrypick-robot commented Feb 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants