GEP-713 enhancements

[guicassolato]

What type of PR is this?
/kind gep

What this PR does / why we need it:
Rewriting of GEP-713 (Memorandum) to clarify concepts and incorporate enhancements discussed at #2927.

Which issue(s) this PR fixes:
Related to #713

Does this PR introduce a user-facing change?:

Enhances GEP-713 (Memorandum) according to top voted suggestions discussed at https://github.com/kubernetes-sigs/gateway-api/discussions/2927, such as:
* merging Direct and Inherited back into a single spec;
* introducing the concept of **merge strategy**

Additionally to:
* targetRef supporting label selectors as an option;
* reduction of targetRef.sectionName to the base case of "it's just another (virtual) resource kind"; and
* algorithm for calculating effective meta resources (effective policies)

And general enhancements to the spec aiming to:
* acknowledge the current known support of the pattern across Gateway API implementations;
* broaden the definitions to potentially welcome known implementations of other meta resource-like concepts into the pattern (or at least acknowledge their similarities with Gateway API)

Moves GEP-713's previous forks GEP-2648 and GEP-2649 to Rejected.

[howardjohn]

can we give some examples of this? I think I understand it but not certain

[guicassolato]

Added a simplified example, plus an extension of it for the case including sectionName.

Please let me know if that works or if you expected to see a full YAML.

[howardjohn]

A full yaml would be nice

[howardjohn]

As long as this is not a MUST then its not a problem, but this seems like it could be quite onerous to compute. For example, imagine I have a global policy and then 1000 namespaces any of which could partially conflict. Its not great to have to 'bubble up' these to the parent.

[howardjohn]

A concrete example of this in existing Gateway API is attachedRoutes, which is similarly complex for implementations to compute (efficiently)

[youngnick]

I think it's important to note that sometimes the merge strategy may be specified in the design of the object (that is, it's a defaults policy or something), rather than in a field?

In fact, I tend to think that, if the merge strategy is listed in a field, it should be in the status, not the spec, since it's relevant info for users of the Policy more than implementers (who will build the merge strategy into code when handling the Policy anyway).

[robscott]

+1, this feels like something that belongs in status.

[candita]

+1 that merge strategy may be defined in the metaresource, e.g. the API contract is either only one is allowed per target, or multiple are allowed.

[guicassolato]

I feel confused about the merge strategy in the status instead of the spec.

Are we talking about the metaresource's status and spec? Or the target's?

The merge strategy, if more than one is supported by the metaresource kind, is a choice of the user that declares an instance of the metaresource. How can it be in the status?

The user literally specify what merge strategy to use when merging that instance of the metaresource. It should be in the spec, no?

[guicassolato]

Added a few lines about merge strategy as a user choice or not, and reflected in the status stanza of the metaresource.

[youngnick]

We need to define "context" again here, I think. (I'd forgotten the definition by the time I got to this part).

Suggested change

	If multiple meta resources target the same context, this is considered to be a conflict.
	If multiple meta resources target the same context (that is, multiple instances of the same or similar policies acting on the same hierarchy have an effective target of the same object), this is considered to be a conflict.

[guicassolato]

"same", yes; "similar", not a good idea IMO. I think the behavior for different kinds of policies should be undefined.

[youngnick]

I'm okay with removing "or similar", but I think that if we're going to leave this as undefined in some cases, we need to be specific in the ones where we do need to have opinions:

• For Gateway API Policy objects included in the specification, in the case of intent conflict with some other Policy on Gateway API objects, the Gateway API Policy must take precedence.
• For implementation specific Policy objects that affect the same properties across multiple implementations, it's up to the implementations to define behavior. If they don't then the behavior is, necessarily, undefined and could produce differing outcomes depending on unknown factors.

In other words, this is a terrible idea and users should try not to use multiple Policy objects that affect the same things.

[youngnick]

Suggested change

	Conflicts must be resolved by applying a defined *merge strategy* (see further definition in the next section), where the meta resource considered higher between two conflicting specs dictates the merge strategy according to which the conflict must be resolved, defaulting to the lower spec (more specific) beating the higher one if not specified otherwise.
	Conflicts must be resolved by applying a defined *merge strategy* (see further definition in the next section).
	When resolving conflicts, the meta resource higher in the relevant hierarchy dictates the merge strategy - that is, merge strategy conflict resolution works on a least-specific-wins basis. After that the merge strategy's conflict resolution rules apply.
	If no merge strategy is specified, then implementations should use more-specific-wins merge strategy by default.

I think this is what you meant here @guicassolato?

[guicassolato]

Yes, but I happen to find the suggested text more confusing than the original.

"least-specific-wins" and "more-specific-wins" have different subjects in the sentences, and therefore I would phrase it differently to avoid confusion.

A merge strategy is a function that takes as input 2 specs and outputs 1.

One thing is determining the merge strategy. When resolving a conflict posed by 2 metaresources, the least specific metaresource among the two dictates the merge strategy that will be used to solve the conflict, i.e. the function that will take both metaresource specs as input. It's always the least specific metaresource that determines it.

The determined merge strategy can be a merge strategy that resolves to "least-specific-wins" or "more-specific-wins" (and occasionally to things more sophisticated than that, like actual merges).

If the least specific metaresource does not specify a merge strategy, then the merge strategy used to resolve the conflict is "more-specific-wins".

[guicassolato]

Rephrased a bit to break down as suggested but trying to avoid overloading terminology.

[shashankram]

The phrasing of more/less specific objects is quite confusing. Perhaps defining a term to convey their position in the config hierarchy would be clearer.

[guicassolato]

Rather than changing the terminology, I've added a sentence elaborating on the idea of being higher/lower in the hierarchy. I hope it helps clarify.

[shashankram]

A bit confusing as the value doesn't convey the meaning. Perhaps BottomUp/TopDown, or a value contextualizing the hierarchy is less ambiguous.

[guicassolato]

The Defaults and Overrides terminology has been employed since the early beginnings of inherited policies in Gateway API. I'm afraid changing this now can lead to even more confusion.

"Bottom-up" and "Top-down" could also clash with the description of the algorithm to calculate effective metaresources, where these terms are employed to describe how one should navigate a path in the hierarchical tree.

[shashankram]

The Defaults and Overrides terminology has been employed since the early beginnings of inherited policies in Gateway API. I'm afraid changing this now can lead to even more confusion.

I understand, but that doesn't make it less confusing. The previous GEP was rejected so we should do our best to avoid ambiguity in terminology this time.

[guicassolato]

We really appreciate the input, @shashankram. And I guess if we can find something better than "defaults" and "overrides" to convey how metaresource/policy rules are applied throughout the hierarchy, I personally think that would be a good addition.

I have to say though that this terminology has so far been well assimilated. This doesn't exclude your criticism of course. I may very well be living in a bubble myself where these words are generally understood, with one exception I will give you: a default value that is specified at a higher, less specific level, in the DAG (i.e. at a node that is closer to the root, including at the root itself) and that is replaced/superseded with a value specified at a lower, more specific level (i.e. at a node that is closer to the leaves, including at a leaf itself) is often described as "overridden at a lower level". I personally find this confusing with otherwise a proper override strategy, i.e., the one where you specify at the less specific level and that yet wins over anything specified at the more specific ones. I made sure to avoid (at least) this confusion in the text.

Other than this, I may need some additional input here to decide how to best move forward. Perhaps @youngnick or @robscott could shine a light.

In time and to be clear, the previous GEP was never rejected. It is actually pretty much alive (as a Memorandum.) We can argue whether "memorandum" is confusing or not for something that is "pretty much alive", but that'd be another conversation 😜

[shashankram]

I have to say though that this terminology has so far been well assimilated

Fair enough

[shashankram]

My overall feedback is that the GEP is too complex as an initial step to land policy merging semantics in Gateway API. The scenarios described are generally applicable to any form of merging, but I wonder if we can layer merging controls gradually versus in the first pass.

[shashankram]

Could you document the names of the implementers that agreed to this pattern?

[guicassolato]

It's a rather tacit agreement. I left a list of all known implementations currently doing Policy Attachment at the end of the GEP. I can't tell for sure which ones participated in the early beginning of the pattern, but I dare to say the ones actively maintaining policy kinds with targetRefs stanza are in agreement.

[shashankram]

Unless Metaresource is an actual API Type, referring to it using a one word canonical name is confusing and the name stutters.

Suggested change

	- _**Metaresource**_: a resource that augments the behavior of another resource without modifying the definition of the resource. Metaresources MUST clearly define a _target_ and an _intent_ as defined in this GEP, and MUST clearly communicate status about whether the augmentation is happening or not.
	- _**Meta-resource**_: a resource that augments the behavior of another resource without modifying the definition of the resource. Metaresources MUST clearly define a _target_ and an _intent_ as defined in this GEP, and MUST clearly communicate status about whether the augmentation is happening or not.

[guicassolato]

We've been back and forth on this one, but attending to requests it was decided to spell it as a single word. Hence 80d8212.

[shashankram]

If this needs to be a single word, I would call this MetaResource not Metaresource for the reason mentioned above. Metaresource reads poor and the name stutters.

[shashankram]

Definitions for defaults/overrides should come before this line.

[shashankram]

What if the targeted Object has distinct sections that need to be referenced using a name? Is a SectionKind/Type necessary to disambiguate (Listener in this case)?

[guicassolato]

I think this will be necessary at some point for sure. I miss the details that first motivated the current, somewhat simpler approach, based only on name and no section kind/type.

Maybe @robscott can refresh my mind.

[shashankram]

Suggested change

	_Virtual types_ are defined as those with a group unkown by the Kubernetes API server. They can be used to apply metaresources to objects that are not actual Kubernetes resources nor Kubernetes custom resources. Rather, virtual types have a meaning for the metaresource controller responsible for implementing the metaresource.
	_Virtual types_ are defined as those with a group unknown by the Kubernetes API server. They can be used to apply metaresources to objects that are not actual Kubernetes resources nor Kubernetes custom resources. Rather, virtual types have a meaning for the metaresource controller responsible for implementing the metaresource.

[shashankram]

Suggested change

	3. Between two metaresources at the same level of the hierarchy and identical creation timestamps, the metaresource appearing first in alphabetical order by `{namespace}/{name}` wins.
	3. Between two metaresources at the same level of the hierarchy and identical creation timestamps, the metaresource appearing first in alphabetical order by `{namespace}/{name}` dictates the merge strategy.

[guicassolato]

Done

[shashankram]

I have to say though that this terminology has so far been well assimilated

Fair enough

[shashankram]

Are Defaults and Overrides flipped here?
Defaults: configured higher in the hierarchy, so less specific beats more specific?
Overrides: lower in the hierarchy so is more specific?

[guicassolato]

Are Defaults and Overrides flipped here?

They are not.

Defaults: configured higher in the hierarchy,

Defaults are configured higher in the hierarchy, yes.

[…] so less specific beats more specific?

However, as stated, it's more specific that beats less specific; not the other way around.

Overrides: lower in the hierarchy so is more specific?

Yes, lower in the hierarchy is more specific.

Think of the Gateway and HTTPRoute kinds, for example. Gateway is higher than HTTPRoute. Gateways specify less specific rules compared to HTTPRoutes, which in turn are more specific.

A default set at the level of a Gateway is initially valid for all HTTPRoutes under the Gateway (less specific). Lower down, it can be replaced for a specific HTTPRoute (more specific.) Therefore the more specific policy rule set at the lower level of HTTPRoute (as in "lower than Gateway") beats the less specific default set at the higher level of Gateway.

[shashankram]

That makes sense. The phrasing is confusing and I would change it to what you explained above.

[guicassolato]

@shashankram, I used your insights to rewrite a bit the whole Conflicting specs, Inheritance, Merge strategies, and Effective metaresources section. Even though I couldn't really run away from "more specific/less specific", I really believe it's more clear now. I hope you agree.

Thanks for the input!

[shashankram]

Instances of these metaresources MUST NOT be allowed to declare more than one merge strategy at a time, but only one of the supported strategies

What does this mean? Are you referring to 2 policy CRs having different merge strategies?

[guicassolato]

There are two possibilities for the design of metaresource CRDs:

1. Single merge strategy – all instances of the CRD (i.e., all CRs) are merged/not merged together following a predefined, static merge strategy that is proper of the CRD.
2. Multiple/user-defined merge strategy: merge strategy specified in a field of each CR. E.g. Kuadrant's AuthPolicy and RateLimitPolicy.

[shashankram]

That makes sense, but the original wording is still ambiguous and needs further clarification. It still doesn't answer my question regarding

Instances of these metaresources MUST NOT be allowed to declare more than one merge strategy at a time, but only one of the supported strategies

[guicassolato]

I meant one CR can only declare one merge strategy. E.g., if the CRD was designed to enable specifying defaults and overrides merge strategies, it's either defaults or overrides in any given CR, but not both simultaneously.

Two CRs of the same CRD can declare different merge strategies between them, if the CRD supports specifying multiple different strategies. E.g., one CR specifies defaults, another CR specifies overrides.

I'll rephrase it to avoid any confusion.

[shashankram]

If the mergeStrategy is an Enum, this would be implicit so you could remove this statement.

[guicassolato]

There is no hard specification for a mergeStrategy field, so to speak. Implementations are free to come up with their own custom strategy names, field names to specify the chosen strategy in a CR, and even schema (should a single field not be enough to specify it). IOW, it is not an enum.

[shashankram]

These are quite ambiguous without examples. Could you first define why "meta-resource classes" are necessary and why Direct and Inherited are relevant?

[guicassolato]

Mainly to ease the transition from the current state of the GEP/s.

GEP-713 was once forked into two GEPs – 2648 and 2649, respectively for Direct and Inherited Policies. We're now merging them back into one single GEP.

The difference between these two classes (Direct and Inherited), if relevant, should emerge from the properties of metaresources specified, including and especially from the concept of merge strategy.

[shashankram]

Those 2 GEPs are marked as declined. So this GEP should reintroduce those concepts before including them in the spec.

[guicassolato]

Sorry. IDK if I follow. There's no spec other than the GEP itself. What do you mean by the "GEP should reintroduce before including in the spec"? This subsection of the guide-level explanation is meant to introduce/define the concepts.

[shashankram]

I meant we should describe the motivation for "meta-resource classes` as it pertains to this GEP instead of assuming it is a well understood concept.

[guicassolato]

Done

[shashankram]

Could you add example policies in YAML for all the scenarios described below

[shashankram]

Would this still be a local namespace selector?

[guicassolato]

It's really unspecified. However, given that cross-namespace refs are only defined/introduced in the next subsection, I understand it's implied that, by default, we mean labels in the same namespace, just like the definition of targeting by name before that makes no reference to namespaces.

Another way to put is: same namespace versus cross-namespace refs are orthogonal to referencing by name or by labels.

[shashankram]

How is it orthogonal? The spec should define whether the label selector is within the local namespace or not.

[shashankram]

Defaults/Overrides doesn't make sense for policies applied to the same level. This should either be None (no merging, oldest wins), or a compositional merge that merges newer policies into older ones in order of priority from old(high)->new(low) without invalidating older policies.

[shashankram]

"merge strategy" seems confusing here as the term should be reserved for how 2 policies are merged, and not to determine their relative prioritization on who decides how they are merged. None/Defaults/Overrides should be categorized as "policy priority".

The atomicity levels defined below reflect the actual merge strategy.

[robscott]

Thanks for all the tremendous work on this @guicassolato!

You've clearly put a lot of time and thought into writing this up, adding some very thorough diagrams and examples.

While I'd like most of the content in this PR to eventually make it into our Policy Attachment Spec, I want to be very careful about what we add at this particular point in time. We're trying to graduate a spec to GA, so making any significant additions essentially means we're taking those new features directly to GA, which I'd like to avoid.

Instead I'd much prefer pulling out anything that is a significant addition to scope (label selectors, merge strategies, etc) to follow up PRs in a way that allows us to separate the stability of those concepts from the core that's already broadly used.

I really would like to get this PR in soon, but the enormous size of it makes it difficult to review and approve with confidence. Anything we can do to split out chunks for smaller more focused follow ups will help get this one in more quickly.

[robscott]

Suggested change

	* Status: Declined
	* Status: Replaced

[guicassolato]

I think Declined was suggested before. OK with Replaced instead, @youngnick?

[robscott]

Suggested change

	status: Declined
	status: Replaced

[robscott]

Suggested change

	* Status: Declined
	* Status: Replaced

[robscott]

Suggested change

	status: Declined
	status: Replaced

[robscott]

Needed for list formatting

Suggested change

	As defined above, a metaresource is a CRD whose purpose is to augment the behavior of some other resource. At its most basic level, the metaresource pattern consists of:
	- A user defines a metaresource describing both the target resource(s) they want to augment, and the intent of the augmentation.
	As defined above, a metaresource is a CRD whose purpose is to augment the behavior of some other resource. At its most basic level, the metaresource pattern consists of:
	- A user defines a metaresource describing both the target resource(s) they want to augment, and the intent of the augmentation.

[robscott]

Adding a selector to policy attachment is a massive increase in scope, I'd much rather focus on stabilizing what we have in this PR and then work on additions like this in a future PR.

[guicassolato]

I can understand that. What does that mean to implementations already doing policies with label selectors though? E.g.: Envoy Gateway, Istio.

[youngnick]

They're not compliant with this version of policy attachment. We can add more detail afterwards.

[robscott]

Suggested change

	_Virtual types_ are defined as those with a group unknown by the Kubernetes API server. They can be used to apply metaresources to objects that are not actual Kubernetes resources nor Kubernetes custom resources. Rather, virtual types have a meaning for the metaresource controller responsible for implementing the metaresource.
	_Virtual types_ are defined as those with a group unknown by the Kubernetes API server. They can be used to apply metaresources to objects that are not actual Kubernetes resources nor Kubernetes custom resources. Rather, virtual types have a meaning for the controller(s) responsible for implementing the metaresource.

[robscott]

Need new line before list for formatting to work

Suggested change

	* a way to augment the behavior of the `Gateway` object itself (e.g. reconcile cloud infrastructure provider settings from the spec declared by the `Gateway` according to the rules specified by the metaresource attached to the `Gateway`) or
	* a way to augment the behavior of the `Gateway` object itself (e.g. reconcile cloud infrastructure provider settings from the spec declared by the `Gateway` according to the rules specified by the metaresource attached to the `Gateway`) or

[robscott]

This feels an awful lot like the previous "Ready" condition on Gateways - near impossible to implement in reality. I'd rather aim for something like "Programmed" here.

[robscott]

I like the idea of merge strategies, but they're a massive addition to the scope of this GEP. I'd rather reserve this for a future addition to policy attachment. That would help keep the scope of what we graduate to GA in this PR very narrow and hopefully simplify the review of this PR. Ideally graduating to GA should be a process of refining existing concepts, not adding new ones.

[guicassolato]

To be completely honest, without the concept of merge strategy I'm not quite sure what we'd be doing.

In my head, defaults, overrides are merge strategies; just like their nuances of JSON patching versus full replacement of specs.

Maybe that was me, but, in #2927 (comment), it felt like we wanted to consolidate Direct and Inherited classes around a well-defined concept of merge strategy.

[shashankram]

How about start with an ImplementationSpecific option?

[youngnick]

Bringing some context here from a Slack thread (https://kubernetes.slack.com/archives/CR0H13KGA/p1745337667779919), @kflynn had a good suggestion for what we need to do in this doc for merge strategies:

1. Define names and mechanisms for possible merge strategies (so both what e.g. “atomic default” means, but also that “atomic default” is the correct name for that strategy)
2. Define a status mechanism by which the strategy SHOULD be reported, and that a conformant implementation MUST use the names defined in 1 to report strategy.
3. Define what merge strategy is preferred for defaults, and define that implementations using the defaults clause SHOULD use that strategy.
4. Define what merge strategy is preferred for overrides, and define that implementations using the overrides clause SHOULD use that strategy.
5. Maybe acknowledge that implementations MAY support other strategies, or selecting strategies at runtime, but that those are implementation-specific behaviors.

@arkodg suggested including a table highlighting the current state of inherited policies in implementations
with suggested columns

1. Implementation
2. How it implements defaults
3. How it implements overrides
4. How it targets resources (refs, selectors)
5. How it merges multiple policies targeting same resource

I'm going to take another pass here today, now that I haven't thought about this for a bit and am a bit more fresh.