Introduce rfc-0005-support-composing-supply-chains

[waciumawanjohi]

Readable

[sclevine]

I wonder if we could combine this with more intelligent (perhaps field-based?) matching between Workloads and SupplyChains to reduce or eliminate the usage of YTT.

[netlify]

✔️ Deploy Preview for elated-stonebraker-105904 canceled.

🔨 Explore the source changes: fab1788

🔍 Inspect the deploy log: https://app.netlify.com/sites/elated-stonebraker-105904/deploys/62030d69710e920008e9d9b5

[waciumawanjohi]

I've mentioned that I expected boolean operators to become necessary at some point for our matchers. This is the first concrete use case I've had with them.

[scothis]

the typically behavior for selectors in k8s is that each facet is ANDed together. If you need OR semantics, define multiple selectors. That won't work if you need nested boolean logic, but at the point the grammar is probably too complex anyway.

[squeedee]

Boolean operators are elidable by the fact that options are already "orable". Yes there's duplication, but its minimal and matches the intent of "when this, do this":

 - name: provide-image
      templateRef:
        kind: ClusterImageTemplate
        options:
          - name: building-image
            selector:
              matchFields:
                - { key: "spec.source.url", operation: exists }
          - name: building-image
            selector:
              matchFields:
                - { key: "spec.source.image", operation: exists }
          - name: image-provider
            selector:
              matchFields:
                - { key: "spec.image", operation: exists }

[waciumawanjohi]

Fair points! @scothis I'm curious if the example Rash gives here is what you had in mind.

[scothis]

@waciumawanjohi that can certainly work, but my suggestion was more like

 - name: provide-image
      templateRef:
        kind: ClusterImageTemplate
        options:
          - name: building-image
            selectors:
            - matchFields:
                - { key: "spec.source.url", operation: exists }
            - matchFields:
                - { key: "spec.source.image", operation: exists }
          - name: image-provider
            selectors:
            - matchFields:
                - { key: "spec.image", operation: exists }

[squeedee]

I wonder if we could combine this with more intelligent (perhaps field-based?) matching between Workloads and SupplyChains to reduce or eliminate the usage of YTT.

@slevine regardless of which solution we end up using for this (There's another proposal on the table) this is a strong desire of mine. We still need looping which we can roll into our standard templates eventually and continue to support with ytt for now, but when we do get there, we have a simple way to deterministically surface exactly which parameters and inputs are used by the template. We can warn supply chain authors when they fail to provide a required input (or provide unused ones), and workload's and deliverables can warn when required params are missed.

[jwntrs]

I've put together a collection of previously discussed alternate syntaxes for this RFC

Typed supply chains (ClusterSourceSupplyChain, ClusterImageSupplyChain, ClusterConfigSupplyChain):
https://gist.github.com/jwntrs/9957c0e99851428b4bf52510d5902285

Allow inlining snippets (which also needs some form of typing):
https://gist.github.com/jwntrs/5cb67f37ed9963cb8381cdcc90bc4e00

Allow our existing templates to enumerate resources:
https://gist.github.com/jwntrs/1df402ccc2202d9e7eb39f97f502d2bd

Allow options to have resources:
https://gist.github.com/jwntrs/bf4f1081e3664615ed640dd90e41a981

[squeedee]

We're currently assessing Snippets as a singular solution to multi-pathing, where a workload can cause a different number of resources to be created or a different set of resources to be created (over the set-of-one supported by options today)

This is not called out in this RFC and should be, if that's part of the decision to implement.

Speaking to that criteria, I feel like snippets do a poor job of solving multipaths because, unless I'm mistaken, the top level supply chain, and most of the sub-supply chains, become supply chains with 2 or 3 resources. The first of which, can feel somewhat nebulous.

Consider a typical flow of resources in a supply chain.
[provider] -> [tester] -> [builder] -> [configurer] -> [pusher]

when you chose to accept a different path for precreated images into configurer you now have a nesting

[[source-provider] -> [source-tester] -> [image-builder] | [image-provider]] -> [configurer] -> [pusher]

which, after adding a choice to test or not:

[[[source-provider] -> [source-tester]] | [source-provider]]-> [image-builder] | [image-provider]] -> [configurer] -> [pusher]

That looks like LISP... I happen to like lisp but I'm not sure it's meaningful if one of the elements is a black box you have to read separately, which also happens to be an encapsulation for conditionals not an encapsulation for task

Also note [source-provider] is enlisted twice, in two different snippets and so should be configured the same way but might not be, complicating debugging and readability of the pure yaml.

See this example from Josh of the shape of the supply chains https://gist.github.com/jwntrs/9957c0e99851428b4bf52510d5902285#file-00-gitops-sc-yml

I've yet to hear why, for this purpose, the multi-path proposal is not a better choice.

Additionally, when coupled with snippets, multi-path will enable encapsulation for task, as per the original motivation for snippets.

[scothis]

s/Kontinue/Cartographer/g

[scothis]

Suggested change

	- key: "spec.source.url"
	- key: "spec.source.git"

[scothis]

Suggested change

	- name: building-image
	selector:
	matchFields:
	- key: "spec.source.url"
	operation: exists
	- name: building-image
	selector:
	matchFields:
	- key: "spec.source.image"
	operation: exists
	- name: building-image
	selector:
	matchFields:
	- key: "spec.source"
	operation: exists

[scothis]

since building-image is a snippet doesn't the kind need to reflect that?

[scothis]

For the same reason that vanilla templates are typed, snippets should also be typed. Making snippets behave differently feels odd. The type creates an unambiguous contract about what the expected outputs are.

We could also achieve the same goal by not introducing any new types, and instead enriching the existing ClusterXTemplate types to support snippets internally, instead of a single stamped resource.

Something like:

apiVersion: carto.run/v1alpha1
kind: ClusterSourceTemplate
metadata:
  name: providing-and-testing-source
spec:
  urlPath: test-source
  revisionPath: test-source
  
  # new snippet
  resources:
   - name: provide-source
     templateRef:
       kind: ClusterSourceTemplate
       options:
       - name: source-from-git-repo
         selector:
           matchFields:
           - { key: "spec.source.url", operation: exists }
       - name: source-from-image-registry
         selector:
           matchFields:
           - { key: "spec.source.image", operation: exists }
  - name: test-source
    templateRef:
      kind: ClusterSourceTemplate
      name: source-tester

I'm not crazy about abusing the urlPath and revisionPath values to point at the resource producing the output, but otherwise it feels like a natural composition as from the supplychains' perspective it's just another component that take inputs and produces source outputs.

[jwntrs]

@scothis Seems like you're in favour of the (option 3) from this list: #72 (comment)

I'm quite opposed to this option because it blurs the line between a supply chain and a template. I'm much more in favour of adding types to supply chains (option 1)

[scothis]

Seems like you're in favour of the (option 3) from this list: #72 (comment)

Mostly, I wouldn't also add support for nested inline templates.

I'm quite opposed to this option because it blurs the line between a supply chain and a template. I'm much more in favour of adding types to supply chains (option 1)

What I like about option 3 is that it's a more pure encapsulation. The outward facing contract for a ClusterXTemplate from the perspective of a ClusterSupplyChain author is identical, the internal implementation decomposes to multiple sub-components. The main difference is that option 1 forces the supply chain author to know more about the implementation detail of the template.

If the encapsulation of option 3 is leaky, then option 1 is better. But if the encapsulation is complete, option 3 is better. The main place where I see the encapsulation being violated is when tracing resources at runtime. For debugability, the snippet shouldn't be a black box. Tools that visualize the supply chain behavior can choose whether to collapse/expand the internals. The purpose of tracing and debug tools is to violate that encapsulation, so I don't think it's enough of a reason on its own for 1 over 3, but it's something to consider.

[scothis]

Do we need to disambiguate components passed into a snippet from components referenced within the same snippet?

[squeedee]

It entirely depends on if the snippet must be easy to validate standalone, or wether we surface these as "required inputs" - a thing that is entirely invalid in a top-level supply chain.

[rawlingsj]

This PR has a lot of comments and discussions which is good but makes it hard to understand what is missing from moving it forward. Ignoring a comparison with the multipath approach can we list out what is needed for this RFC to move forward please?

There's been a lot of chat on office hours, slack, docs and miro but I'd like to understand any blockers for this RFC.

Any specific requests around:
Spec?
Examples?
Fundamental design issues?

[scothis]

Fundamental design issues?

Not so much an issue but an open question around the shape of snippets as started by @jwntrs, and then continued by me.