proposal: PodConfig as a custom resource

[AmitKumarDas]

ProblemStatement: As a DevOps engineer, I want a k8s controller that accepts values from various sources and provides a final set of merged values that can in turn be used to arrive at a desired state of a Pod.

For example:

• PodConfig A provides taints & tolerations for Pod
• PodConfig B provides resource limits (memory & cpu) for Pod
• PodConfig C provides environment variables for containers of the Pod
• PodConfigResult F should merge all the above & present a final state that is a 3 way merge of all the above based on priority.

Now there can be custom controllers that make use of resource F to build the desired Pod state & subsequently apply this Pod state against the kubernetes cluster.

[AmitKumarDas]

kind: PodConfig
metadata:
  name: containers
  namespace: blah
spec:
  podConfigResult:
    name: final
  priority: 100
  state:
    spec:
      containers:
      - name: stress
        image: stress:v1
        resources:
          requests:
            memory: "50Mi"
          limits:
            memory: "100Mi"

kind: PodConfig
metadata:
  name: tolerations
spec:
  podConfigResult:
    name: final
  priority: 100
  state:
    spec:
      tolerations:
      - key: "example-key"
        operator: "Exists"
        effect: "NoSchedule"

kind: PodConfigResult
metadata:
  name: containers
  namespace: blah
status:
  state:
    spec:
      containers:
      - name: stress
        image: stress:v1
        resources:
          requests:
            memory: "50Mi"
          limits:
            memory: "100Mi"
      tolerations:
      - key: "example-key"
        operator: "Exists"
        effect: "NoSchedule"

[sagarkrsd]

Great @AmitKumarDas, this can definitely help us in building the openebs-operator more robust and modular where we handle the OpenEBS pod configurations wrt node selector, affinity, taints and a lot more to a large extent.

[AmitKumarDas]

@sagarkrsd can you explain how exactly this can help? For example, can you list down the pain points that this solution tries to solve & i will try to fine tune the approach.

This approach considers both controller & users making their changes & the change that has higher priority gets accepted as the final state. For example this should help in arriving at the desired states for STS, Deployment, DaemonSet.

Can this reduce the amount of code/boilerplate inn openebs upgrade?

[sagarkrsd]

When we say this(PodConfig) can be edited by users also, we should consider the case where it seems too generic and users would be more interested in providing the config at Deployment, DaemonSet, etc level instead of all the way coming down to pod.
For example, let's say I want my NDM pods to get deployed on a particular set of nodes or I want the NDM pods to use a specified amount of resources such as CPU, memory, etc and the other components to use some different values.

[sagarkrsd]

openebs-upgrade is responsible for deploying/managing OpenEBS components, hence it has the responsibility to form the desired manifest for all the OpenEBS components.

Some of the pain points that it can solve wrt openebs-upgrade are:

• Today, we have got a CR named OpenEBS which stores all the configuration wrt OpenEBS, it contains pod config for a particular component as well if a user wants to change it, the user needs to add that configuration in OpenEBS CR only.
  If somehow the above proposal can help with the above i.e., maybe the user will get an option to form a particular component's podConfig separately, it would be very helpful.

• One of the challenges that we face while forming desired manifests for the OpenEBS components is handling the various pod configuration and that too based on the different container's config i.e., if a pod has multiple containers then forming the manifest(the code that goes into forming the manifest becomes a little cumbersome) becomes a little tedious and complex.

[AmitKumarDas]

@sagarkrsd If PodConfig becomes too generic we can have specific ones e.g. DeploymentConfig, STSConfig & so on. Users can edit these configs or provide new ones with priority & podConfigResult.name. The design is such that n no of PodConfigs get merged to a single PodConfigResult.

Third party operators such as openebs upgrade will be looking out for specific PodConfigResult instance & make use of it to apply desired Pod or Deployment or STS etc.

In other words, PodConfigResult acts as a bridge to third party controllers. User or other controllers deal with PodConfig(s) only.

If we try to think from OpenEBS upgrade as a 3rd party controller, it will now deal with various custom resources like PodConfig, PodConfigResult, DeploymentConfig, DeploymentConfigResult & so on. Amount of boilerplate code in upgrade controller should reduce significantly.

[AmitKumarDas]

  # PodBuilder will lookup various configs & optional existing Pod
  # resource to build a new Pod specification.
  kind: PodBuilder
  spec:
    # prefer Config over Pod or vice-versa
    # Config or Pod are valid values
    prefer:
    
    # selection may include one or more containers
    # they will be added/merged into the generated Pod spec based
    # on priorities assigned
    containerConfigSelector:
      matchLabels:
    
    # selection may include one or more tolerations
    # they will be merged into the generated Pod spec based
    # on priorities assigned
    tolerationsConfigSelector:
      matchLabels:
    
    # selection should merge the matching pods based on
    # priorities. Priority can be set as an annotation. If no
    # priority is found then use the first possible match.
    # Log a warning if other matches are ignored.
    podSelector:
      matchLabels:
    
    # the resource that is applied after pod spec is built
    # PodPreview is the default
    output: # PodPreview, or Pod
  status:

  # This will reflect one kubernetes Container spec
  kind: ContainerConfig
  metadata:
    name: CC-1
    labels:
      cstor: mgmt
  spec:
    priority:
    template:
      name:
      image: cstor-pool-mgmt
      command:
      args:
      env:
      resources:
        cpu:
        mem:

  # This will reflect one kubernetes Container spec
  kind: ContainerConfig
  metadata:
    name: CC-2
    labels:
      cstor: monitor
  spec:
    priority:
    template:
      name:
      image: cstor-pool-monitor
      command:
      args:
      env:
      resources:
        cpu:
        mem:

  # This will reflect one Kubernetes tolerations spec
  kind: TolerationsConfig
  metadata:
    name:
    namespace:
    labels:
  spec:
    priority
    template:
    - key:
      operator:
      effect:
      tolerationSeconds:

  # This will be the output of PodBuilder
  # PodBuilder might apply a Pod as well if set
  kind: PodPreview
  result:

[AmitKumarDas]

How about just limiting to PodBuilder, PodConfig & PodPreview instead of so many listed above?

We might want to include EnvironmentConfig that can be used to apply env across all containers of the Pod.

This might be something to do with this enhancement at metac library.