argo cli submit needs retries

[tooptoop4]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

using argo cli to submit workflow sometimes get:

level=fatal msg="Failed to submit workflow: rpc error: code = AlreadyExists desc = Operation cannot be fulfilled on resourcequotas \"redact\": the object has been modified; please apply your changes to the latest version and try again"

argo-workflows/cmd/argo/commands/submit.go

Lines 197 to 241 in 89d75a6

	return fmt.Errorf("Failed to submit workflow: %v", err)
	}
	if err = printWorkflow(created, common.GetFlags{Output: cliOpts.Output}); err != nil {
	return err
	}
	return common.WaitWatchOrLog(ctx, serviceClient, namespace, []string{created.Name}, *cliOpts)
	}
	func submitWorkflows(ctx context.Context, serviceClient workflowpkg.WorkflowServiceClient, namespace string, workflows []wfv1.Workflow, submitOpts *wfv1.SubmitOpts, cliOpts *common.CliSubmitOpts) error {
	if err := validateOptions(workflows, submitOpts, cliOpts); err != nil {
	return err
	}
	if len(workflows) == 0 {
	return errors.New("No Workflow found in given files")
	}
	var workflowNames []string
	for _, wf := range workflows {
	if wf.Namespace == "" {
	// This is here to avoid passing an empty namespace when using --server-dry-run
	wf.Namespace = namespace
	}
	err := util.ApplySubmitOpts(&wf, submitOpts)
	if err != nil {
	return err
	}
	if cliOpts.Priority != nil {
	wf.Spec.Priority = cliOpts.Priority
	}
	options := &metav1.CreateOptions{}
	if submitOpts.DryRun {
	options.DryRun = []string{"All"}
	}
	created, err := serviceClient.CreateWorkflow(ctx, &workflowpkg.WorkflowCreateRequest{
	Namespace: wf.Namespace,
	Workflow: &wf,
	ServerDryRun: submitOpts.ServerDryRun,
	CreateOptions: options,
	})
	if err != nil {
	return fmt.Errorf("Failed to submit workflow: %v", err)

Version(s)

3.4.11

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

n/a

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

n/a

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

n/a