[RFC] Follow idea of immutable /usr vs. mutable overrides in /etc

[jnpkrn]

There are many practical reasons why we want to copy this growingly
popular scheme, while enabling users to modify the agents per their
needs, for instance:

• having solely static data in /usr allows one to share that as
  read-only (or sparsely utilized copy-on-write) mount point with
  their VMs and containers so as to save space

• no conflict-on-update issue

Hence my expectation is that OCF standard will address this,
presumably in resource-agent-api.md by replacing

The Resource Agents are located in subdirectories under
/usr/ocf/resource.d.

with something like

The Resource Agents are located in subdirectories under
/usr/ocf/resource.d. OCF X.Y compliant RM shall first consult
/etc/ocf/resource.d path for existence of the requested agent,
which, when present, takes a precedence in the agent lookup.
This makes for convenient customization of existing agents without
altering them at the stated standard location, and in turn,
simplifying a revert to stock configuration, coexistence with
package updates, and possibly locked-down use of /usr mount
point. The agent lookup based on the file presence is definite,
any further issue, like file not being executable, notwithstanding.

[krig]

Sounds good to me. 👍

[kgaillot]

I'm uncomfortable with putting executables in /etc, and I strongly think users shouldn't reuse the same provider+agent name when modifying an agent, as it greatly complicates troubleshooting.

The currently recommended approach for modifying resource agents is to create a new, custom provider under /usr/lib/ocf/resource.d. I could see extending the standard to allow providers in an alternate location, such as /usr/local, /opt, or /srv (followed by ocf/resource.d), or even allowing an OCF_RA_PATH environment variable. I'm not convinced it's a good idea though, as custom OCF scripts are not any more mutable than the commonly distributed ones. In production, few users are going to modify custom scripts directly; they are going to have a development environment, and then push changes to all production nodes (comparable to updating the resource-agents package).

[jnpkrn]

On 21/11/17 22:00 +0000, Ken Gaillot wrote: I'm uncomfortable with putting executables in /etc,

There's a bunch of executable glue scripts already (including /etc/rc.d/init.d ones for non-systemd systems), which is exactly what resource agents are meant to be. I see no conflict here.

and I strongly think users shouldn't reuse the same provider+agent name when modifying an agent, as it greatly complicates troubleshooting.

Resource managers would need to identify the particular file clearly, true. There perhaps, if the idea is deemed good enough, should also be a provision in the specification wrt. explicit remapping of agent specification to particular path, like SIGHUP signal sent to the resource manager. Prior to that, it would keep initially figured path.

The currently recommended approach for modifying resource agents is to create a new, custom provider under /usr/lib/ocf/resource.d. I could see extending the standard to allow providers in an alternate location, such as /usr/local, /opt, or /srv (followed by ocf/resource.d), or even allowing an OCF_RA_PATH environment variable. I'm not convinced it's a good idea though, as custom OCF scripts are not any more mutable than the commonly distributed ones. In production, few users are going to modify custom scripts directly; they are going to have a development environment, and then push changes to all production nodes (comparable to updating the resource-agents package).

That's not the use case I had in mind, going that deep as to also change the actual configuration. Rather something like: http://oss.clusterlabs.org/pipermail/users/2017-August/006303.html Anyway, having bulk synchronization of /etc across the nodes can be appealing (also for systemd unit files, that can likewise be employed with a resource manager if there's a support).

…

-- Jan (Poki)

[dmuhamedagic]

I'd rather not. Doesn't the provider concept offer enough flexibility? As Ken said, it would also be quite difficult to figure out which RA is being run if the resource manager is allowed to look at more than one place for the same resource configuration.

[oalbrigt]

And you can already make custom or similarly named directories in /usr/lib/ocf/resource.d/heartbeat to avoid clashing with the agents provided by the distro.

[jnpkrn]

On 22/11/17 14:27 +0000, Dejan Muhamedagic wrote: I'd rather not. Doesn't the provider concept offer enough flexibility? As Ken said, it would also be quite difficult to figure out which RA is being run if the resource manager is allowed to look at more than one place for the same resource configuration.

Additional idea sketched above would make the flip only at defined moments (initial start, being told to rescan the agent mapping). Not at arbitrary points, which would indeed make the situation hard to follow.

On 22/11/17 14:36 +0000, Oyvind Albrigtsen wrote: And you can already make custom or similarly named directories in /usr/lib/ocf/resource.d/heartbeat to avoid clashing with the agents provided by the distro.

Naturally, but I thought it's clear that I anticipated it for a bit different use cases by now.

…

-- Jan (Poki)

[jnpkrn]

On 22/11/17 17:11 +0100, Jan Pokorný wrote: Additional idea sketched above would make the flip only at defined moments (initial start, being told to rescan the agent mapping). Not at arbitrary points, which would indeed make the situation hard to follow.

On the other hand, let's not fall into the fallacy that current situation is a breeze in the "which agent variant was run, exactly" matter, at least with pacemaker in particular: - respective agent files are not locked for the pacemaker's lifespan, so can be edited anytime - ditto agents are not copied to a private temporary location first (or even copied into the memory to be executed from, which would be doable for hashbang/non-binary executables) - checksums of the agents are not logged/remembered+rechecked (or ditto on timestamp comparison basis) Which already makes it rather difficult to tell which variant of the agent was run in any particular moment in the past unless you can testify nothing has intervened (and even then it's not 100%). So I don't see any remarkable regression, pros and cons summed together IMHO yields a positive result here when the mentioned additional idea of explicit rescans is mixed in.

…

-- Jan (Poki)

[krig]

The main benefit as I see it would be enabling the sysadmin to add their own agents on top of a read-only /usr file system delivered by a transactional update mechanism.

[jnpkrn]

On 23/11/17 09:01 +0000, Kristoffer Grönlund wrote: The main benefit as I see it would be enabling the sysadmin to add their own agents on top of a read-only `/usr` file system delivered by a transactional update mechanism.

The other practical value is that administrator would (one wants to say, finally) gain power to defuse OCF-based resources that are not, by any mean, desired in the projected cluster from the set of agents that get installed unconditionally through the common distribution channels, sometimes including ocf:heartbeat:anything, which may be unsettling on its own: http://lists.clusterlabs.org/pipermail/users/2016-January/002178.html This is very similar to and directly inspired with systemd's masking approach. So when the cluster should only ever serve for minimalistic httpd + virtual IP combo, the solution would be to run this upon each install/update of resource-agents in a RPM-based distro: ``` # mkdir -p /etc/lib/ocf/resource.d/heartbeat # rpm -ql resource-agents \ | grep '/usr/lib/ocf/resource.d/heartbeat/[^.].*' \ | grep -vE 'apache|IPaddr2' \ | sed "s|/usr|/etc|" | xargs -I{} echo ln -s /dev/null {} ``` For this to work harmonically, resource managers should further realize zero size of the discovered agents like this and exclude them from "try running" attempts (incl. at the system location, indeed). For pacemaker in particular and putting fence-agents aside (preferrably, there would be a convergence towards OCF in some aspects, plus the agents are separated in discrete subpackages in el7, giving administrator at least some say to what's available), the only way to run an unrestricted command from cluster configuration would then be "lsb:<script>" variety of agent specification...

…

-- Jan (Poki)

[jnpkrn]

Thinking about that, /etc/ocf should indeed be subdir-namespaced per
resource-manager, possibly reserving a chosen name (ANY?) to apply for all.

[krig]

The other practical value is that administrator would (one wants to
say, finally) gain power to defuse OCF-based resources

I'm sorry, but I don't understand this argument at all. Why is the administrator trying to prevent the administrator from configuring resources?

I also don't recall any actual argument for why the anything agent is problematic...

[jnpkrn]

On 27/11/17 11:23 +0000, Kristoffer Grönlund wrote: > The other practical value is that administrator would (one wants to > say, finally) gain power to defuse OCF-based resources I'm sorry, but I don't understand this argument at all. Why is the administrator trying to prevent the administrator from configuring resources?

Why to administratively prevent _selected_ (this needs stressing out) OCF resources (out of all delivered en masse with resource-agents project, for instance)? Mostly to allow (on opt-in basis) one to follow least-privilege principle with the intention to keep any kind of intrusion as limited as possible, especially for static cluster deployments where additional resource agents are irrelevant at any rate. Needless to mention one such intrusion enabler from the recent time: http://oss.clusterlabs.org/pipermail/users/2016-November/004432.html Sure, one might emulate something like that using acls: ``` <acls> <acl_target id="bob"> <role id="admin"/> </acl_target> <acl_role id="admin"> <acl_permission id="admin-deny-1" kind="deny" xpath="//primitive[@Class='ocf' and @Provider='heartbeat' and @type!='apache' and @type!='IPaddr2']"/> <acl_permission id="admin-write-1" kind="write" xpath="/cib/resources"/> <acl_permission id="admin-read-1" kind="read" xpath="/cib"/> </acl_role> </acls> ``` But beside being rather clumsy, it doesn't cover the fully privileged users (e.g. hacluster user) -- it cannot by design. Does it answer your question?

I also don't recall any actual argument for why the `anything` agent is problematic...

This one together with Stateful are perfectly fine for code-less experimenting with how pacemaker works and kicking off custom agents, but quite an antipattern for the production use where you want launcher as tightly fitting as possible, getting the monitoring right, covering the corner cases, etc. O:-) Plus add the above security aspect into the equation. It's not a security measure per se, but onion-like approach to security hardening (just as SELinux is, for instance) makes sense when it doesn't impose new pains. Voluntary constraining of the agents' repertoire is IMHO one of the easy wins on this front.

…

-- Jan (Poki)

[krig]

Yeah, I think I follow what you're saying. Of course the apache agent might not be the best example to allow when trying to avoid privilege escalation, since it can be trivially configured to execute arbitrary executables. Though that might be an argument for fixing apache. ;)

[kgaillot]

There's a bunch of executable glue scripts already (including /etc/rc.d/init.d ones for non-systemd systems), which is exactly what resource agents are meant to be. I see no conflict here.

While there are common existing cases of executables under /etc, they are exceptions, not the rule. System administrators expect /etc to contain configuration, and executables to be located elsewhere, except in unusual cases.

I believe this is recommended in the LSB, with good reason. An example is that resource agents do not necessarily need to be scripts, they can be compiled, but /etc is architecture-independent.

On the other hand, let's not fall into the fallacy that current situation is a breeze in the "which agent variant was run, exactly" matter, at least with pacemaker in particular:

The main goal is whether enterprise support personnel can reasonably determine whether a particular agent is supported, not the exact agent code used. If the user can override an OS-provided agent, extra steps must be taken with every support case to check whether that has happened. The current recommendation of using a different provider name makes it immediately clear.

Also, the provider name is intended to indicate who provided the agent. If a custom script reuses a provider name, it obscures that indicator. The current recommendation of using a different provider name when modifying a script makes it clear where the agent came from.

The main benefit as I see it would be enabling the sysadmin to add their own agents on top of a read-only /usr file system delivered by a transactional update mechanism.

I don't believe this accomplishes that. When users modify or create resource agents, they typically get them working, then rarely or never touch them again. They tend to change less frequently than OS-supplied resource agents. Custom agents don't prevent /usr from being read-only any more than OS-supplied ones do. In either case, there has to be a mechanism to temporarily make /usr writeable during updates.

Even if a non-/usr location is perceived to be desirable, I would argue for using a custom provider name, and have the non-/usr location be where to look for additional providers.

I'm sorry, but I don't understand this argument at all. Why is the administrator trying to prevent the administrator from configuring resources?

I also don't recall any actual argument for why the anything agent is problematic...

I agree. Disabling particular resource agents is no different than disabling particular binaries provided with any other package. If someone wishes to disable unused resource agents, likely they want to disable unused binaries from other packages as well, and already have a generic mechanism for doing so.

Also, this is a security risk, not a mitigation. Being able to write a script into /etc that is automatically run as root without having to touch the pacemaker configuration destroys any security gained by mounting /usr read-only. And I can't imagine any scenario where a security compromise that allows an unused OCF agent as a vector doesn't have an easier vector elsewhere. Pacemaker runs as root and can run arbitrary executables. They don't have to be in the OCF agent directory.

Regarding non-production agents such as Dummy, anything, etc., it is up to each distribution to decide which agents are installed by which packages. For example, RHEL already removes some agents distributed upstream. Any distribution could move such agents to a resource-agents-testing package, for example, or create a separate package for each resource agent, allowing users to install only the ones they need. Similarly users who compile their own can build packages as they like.

Bottom line, I could see some value in having alternate locations for providers, but I think users should be shepherded into using a unique provider name if they modify or create an agent.