OCPBUGS-25860: Disable managed interrupts for smartpqi

[MarSik]

This driver does not obey the isolcpus=managed_irq hint and is causing interference.

This kernel argument makes sure the interrupt affinity can be managed by userspace services.

The alternative approach using /etc/modprobe.d/smartpqi.conf with option smartpqi ... does not work, because the driver is loaded early at the initrd stage and we would have to rebuild the RHCOS initrd. That is much heavier than a simple kernel arg.

[openshift-ci-robot]

@MarSik: This pull request references Jira Issue OCPBUGS-25860, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.0) matches configured target version for branch (4.16.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira ([email protected]), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This driver does not obey the isolcpus=managed_irq hint and is causing interference.

This kernel argument makes sure the interrupt affinity can be managed by userspace services.

The alternative approach using /etc/modprobe.d/smartpqi.conf with option smartpqi ... does not work, because the driver is loaded early at the initrd stage and we would have to rebuild the RHCOS initrd. That is much heavier than a simple kernel arg.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: MarSik

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [MarSik]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[MarSik]

@jmencak @yarda Could you please check my syntax for the udev regex?

[jmencak]

@jmencak @yarda Could you please check my syntax for the udev regex?

Hmm, I'm getting the following:

2024-01-12 08:30:32,715 ERROR    tuned.units.manager: failed to initialize plugin bootloader_smartpqi
2024-01-12 08:30:32,716 ERROR    tuned.units.manager: No module named 'tuned.plugins.plugin_bootloader_smartpqi'
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tuned/units/manager.py", line 88, in create
    plugin = self._plugins_repository.create(plugin_name)
  File "/usr/lib/python3.9/site-packages/tuned/plugins/repository.py", line 34, in create
    plugin_cls = self.load_plugin(plugin_name)
  File "/usr/lib/python3.9/site-packages/tuned/utils/plugin_loader.py", line 32, in load_plugin
    return self._get_class(module_name)
  File "/usr/lib/python3.9/site-packages/tuned/utils/plugin_loader.py", line 36, in _get_class
    module = __import__(module_name)
ModuleNotFoundError: No module named 'tuned.plugins.plugin_bootloader_smartpqi'
2024-01-12 08:30:32,718 INFO     tuned.daemon.daemon: static tuning from profile 'bz' applied

With:

[main]
summary=PoC

[bootloader_smartpqi]
devices_udev_regex==DRIVER=smartpqi
cmdline_smartpqi=+smartpqi.disable_managed_interrupts=1

Did you test this?

[MarSik]

Not yet, I should have added WIP to the name..
I know what the issue is, will fix it soon.

[jmencak]

This seems to somewhat work now. But I can also see the /etc/tuned/bootcmdline set to:

TUNED_BOOT_CMDLINE="smartpqi.disable_managed_interrupts=1"

even when the smartpqi module is not loaded. Is this intended? I suspect the devices_udev_regex==DRIVER=smartpqi doesn't take any effect. WDYT?

[MarSik]

/hold

The udev regex matching does not work for the bootloader plugin.

[yanirq]

@MarSik lets rebase and check for the right udev rule

[yanirq]

/hold
Depends on #980
to pass ci/prow/e2e-no-cluster tests

[yanirq]

/hold cancel

[MarSik]

/retest

[yanirq]

@MarSik you'll probably need to run render sync

[openshift-ci]

@MarSik: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-no-cluster	580709b	link	true	/test e2e-no-cluster

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.