feat(controller): support volumeClaimTemplates can be referenced by templateRef. Fixes #13977 #7444

[chengjoey]

Fixes #13977 #7444

Motivation

In order for volumeClaimTemplates to be referenced by the templateRef

The purpose is to automatically bring the corresponding volumeClaimTemplates into the current workflow when using templateRef.

Modifications

1. add volumeClaimTemplates field in template
2. workflow operator create the pvcs referred by template

Verification

e2e test

[shuangkun]

looks good. please resolve conflicts!

[Joibel]

Is there any limit on how many times createPVCs() will be called, it looks like it will be called repeatedly.

[Joibel]

execWf.Spec should be considered immutable (constant), this isn't good practice.

[chengjoey]

okay, I might be able to put this part of the logic inside setExecWorkflow(ctx context.Context), and reserve all the PVCs when initializing execWf.

[chengjoey]

@Joibel

argo-workflows/workflow/controller/operator.go

Lines 148 to 150 in 581950f

	// NEVER modify objects from the store. It's a read-only, local cache.
	// You can use DeepCopy() to make a deep copy of original object and modify this copy
	// Or create a copy manually for better performance

execWf is a copy, modifying execWf is not good practice, but is this a feasible solution?

[chengjoey]

Is there any limit on how many times createPVCs() will be called, it looks like it will be called repeatedly.

createPVCsis an idempotent function，len(woc.execWf.Spec.VolumeClaimTemplates) == len(woc.wf.Status.PersistentVolumeClaims) ensured that there will be no duplicate creation

argo-workflows/workflow/controller/operator.go

Lines 1682 to 1692 in 89d75a6

	func (woc *wfOperationCtx) createPVCs(ctx context.Context) error {
	if !(woc.wf.Status.Phase == wfv1.WorkflowPending || woc.wf.Status.Phase == wfv1.WorkflowRunning) {
	// Only attempt to create PVCs if workflow is in Pending or Running state
	// (e.g. passed validation, or didn't already complete)
	return nil
	}
	if len(woc.execWf.Spec.VolumeClaimTemplates) == len(woc.wf.Status.PersistentVolumeClaims) {
	// If we have already created the PVCs, then there is nothing to do.
	// This will also handle the case where workflow has no volumeClaimTemplates.
	return nil
	}

[b0ch3nski]

Hey 🙂

Anything still blocking this feature? We are patiently waiting especially for this with the use case of starting multiple Workflow Templates (each having it's own volume) using withParam based on output of previous steps.

Currently we are forced to declare single big volume at the Workflow level which is quite inconvenient then to share it with multiple concurrent child templates.

Thanks!