artifact build failed to load: failed to checkout "xxx": worktree contains unstaged changes

[DavidGamba]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

When using the following git artifact configuration:

      inputs:
        artifacts:
          - name: build
            path: /build
            git:
              repo: '{{inputs.parameters.repo}}'
              branch: '{{= sprig.trimPrefix(''refs/heads/'', inputs.parameters.branch) }}'
              revision: '{{inputs.parameters.commit}}'
              depth: 0
              usernameSecret:
                name: github-secrets
                key: username
              passwordSecret:
                name: github-secrets
                key: password
              singleBranch: true

I get the following error: artifact build failed to load: failed to checkout "xxx": worktree contains unstaged changes

After following the logic to this line: https://github.com/argoproj/argo-workflows/blob/release-3.6/workflow/artifacts/git/git.go#L167

I found that enabling a force checkout fixes my problem:

import (
        ...
	"github.com/go-git/go-git/v5"
	"github.com/go-git/go-git/v5/plumbing"
	"github.com/go-git/go-git/v5/plumbing/transport/ssh"
)

	auth, err := ssh.NewSSHAgentAuth("git")
	if err != nil {
		return fmt.Errorf("failed auth: %w", err)
	}

	repoName := "[email protected]:DavidGamba/dgtools.git"
	commit := "84cdf63980b49f1cbe73e2e62fac6e2feba9b033"
	branch := "master"

	cloneOptions := &git.CloneOptions{
		URL:             repoName,
		Auth:            auth,
		Depth:           0,
		ReferenceName:   plumbing.NewBranchReferenceName(branch),
		SingleBranch:    true,
		InsecureSkipTLS: true,
		Progress:        os.Stdout,
	}
	repo, err := git.PlainClone("/tmp/gittest", false, cloneOptions)
	if err != nil {
		return fmt.Errorf("failed clone: %w", err)
	}

	// checkout
	wt, err := repo.Worktree()
	if err != nil {
		return fmt.Errorf("failed worktree: %w", err)
	}
	err = wt.Checkout(&git.CheckoutOptions{
		Hash:  plumbing.NewHash(commit),
		Force: true,
	})
	if err != nil {
		return fmt.Errorf("failed checkout: %w", err)
	}

And removing the force checkout reproduces the problem.

I don't see any reason why Force checkout can't be the default and I am willing to submit the PR if you are OK with it.

Logs from init container:

0ef7b6432925e49882cb4294adccf5912ec,GitTag:v3.6.2,GitTreeState:clean,GoVersion:go1.23.3,Compiler:gc,Platform:linux/amd64,}"
time="2025-01-31T19:38:29 UTC" level=info msg="Loading script source to /argo/staging/script"
time="2025-01-31T19:38:29 UTC" level=info msg="Start loading input artifacts..."
time="2025-01-31T19:38:29 UTC" level=info msg="Downloading artifact: build"
time="2025-01-31T19:38:31 UTC" level=info msg="Load artifact" artifactName=build duration=2.396653351s error="failed to checkout \"0ed01a498b9be55d63b769b077003a4e60b5d6f0\": worktree contains unstaged changes" key=
time="2025-01-31T19:38:31 UTC" level=error msg="executor error: artifact build failed to load: failed to checkout \"0ed01a498b9be55d63b769b077003a4e60b5d6f0\": worktree contains unstaged changes"
time="2025-01-31T19:38:31 UTC" level=info msg="Alloc=60052 TotalAlloc=386522 Sys=148987 NumGC=15 Goroutines=5"
Error: artifact build failed to load: failed to checkout "0ed01a498b9be55d63b769b077003a4e60b5d6f0": worktree contains unstaged changes
Usage:
  argoexec init [flags]
Flags:
  -h, --help   help for init
Global Flags:
      --as string                      Username to impersonate for the operation
      --as-group stringArray           Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
      --as-uid string                  UID to impersonate for the operation
      --certificate-authority string   Path to a cert file for the certificate authority
      --client-certificate string      Path to a client certificate file for TLS
      --client-key string              Path to a client key file for TLS
      --cluster string                 The name of the kubeconfig cluster to use
      --context string                 The name of the kubeconfig context to use
      --disable-compression            If true, opt-out of response compression for all requests to the server
      --gloglevel int                  Set the glog logging level
      --insecure-skip-tls-verify       If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure
      --kubeconfig string              Path to a kube config. Only required if out-of-cluster
      --log-format string              The formatter to use for logs. One of: text|json (default "text")
      --loglevel string                Set the logging level. One of: debug|info|warn|error (default "info")
  -n, --namespace string               If present, the namespace scope for this CLI request
      --password string                Password for basic authentication to the API server
      --proxy-url string               If provided, this URL will be used to connect via proxy
      --request-timeout string         The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0")
      --server string                  The address and port of the Kubernetes API server
      --tls-server-name string         If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.
      --token string                   Bearer token for authentication to the API server
      --user string                    The name of the kubeconfig user to use
      --username string                Username for basic authentication to the API server
artifact build failed to load: failed to checkout "0ed01a498b9be55d63b769b077003a4e60b5d6f0": worktree contains unstaged changes

Version(s)

v3.6.2

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

See description

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

[DavidGamba]

reproducer.zip

$ go run .                                
2025/02/03 10:20:30 []
2025/02/03 10:20:30 Running
Enumerating objects: 2687, done.
Counting objects: 100% (432/432), done.
Compressing objects: 100% (189/189), done.
Total 2687 (delta 310), reused 301 (delta 241), pack-reused 2255 (from 2)
ERROR: failed checkout: worktree contains unstaged changes
exit status 1

$ go run . -force
2025/02/03 10:20:42 []
2025/02/03 10:20:42 Running
Enumerating objects: 2687, done.
Counting objects: 100% (432/432), done.
Compressing objects: 100% (189/189), done.
Total 2687 (delta 310), reused 301 (delta 241), pack-reused 2255 (from 2)

[MasonM]

Thanks for the report! I tried running the reproducer, and it works fine for me:

$ go run .                                                                                                                                                                                                                       
2025/02/08 17:39:17 []
2025/02/08 17:39:17 Running
Enumerating objects: 2691, done.
Counting objects: 100% (436/436), done.
Compressing objects: 100% (193/193), done.
Total 2691 (delta 312), reused 302 (delta 241), pack-reused 2255 (from 2)

I'm not sure why this isn't working for you, but I know the worktree contains unstaged changes error happens when there are existing files in the destination directory. Can you provide the full workflow YAML to reproduce this?

[ben-wangz]

go-git/go-git#511 (comment)
is there a symbol link in your repo? @DavidGamba

[DavidGamba]

@ben-wangz I use symlinks extensively. I will be able to generate the workflow reproducer at the end of the week. I had to swap projects for a while. I apologize for the delay in following up.