Can't collect artifacts from a folder containing a "%" or ";" in its name

[asavchkov]

Issue Type

• Bug Report

Summary

I have a test checking that all special characters are correctly handled in my program. The program creates folders with special characters. In GoCD if an artifact wildcard pattern matches one of such folders containing a "%" or ";" in its name then uploading the artifacts fails.

Basic environment details

• Go Version: 23.5.0 (18179-7702b283accd1f90f014f0087aa2e9bd8baf4a97).
• JAVA Version: 17.0.9
• OS: Linux 5.15.0-88-generic

Steps to Reproduce

1. Create such a shell job:
   mkdir %
   touch %/test.log

2. Create an artifact pattern **/*.log

3. Run the pipeline with this job.

Expected Results

%/test.log successfully uploaded to the artifact folder of the GoCD server.

Actual Results

Job log:

[go] Uploading artifacts from /var/lib/go-agent-3/pipelines/bug/%/test.log to /%
[go] Failed to upload /var/lib/go-agent-3/pipelines/bug/%/test.log

The same behaviour is observed with a ";". Interestingly, the error occurs only for folders, artifact files dir/%.log and dir/";".log are uploaded successfully.

[chadlwilson]

Probably URL encoding or URL rewrite handling (server route handling) problems, although there are probably logs on your agent or server (not just job log) to see what's happening if it's not making it there.

I'm not sure how much of this I care to support. At the margins it gets messy due to the way direct server filesystem artifact storage is used, and the way artifacts are archived and redownloaded by agents as archives that may or may not support the various special chars. Server could be windows but agent Linux, or different filesystems used underneath.

I really don't suggest you rely on this. Better to zip or tarball them.

[asavchkov]

After reconsidering this issue I agree with you that this case is so marginal that it should better be avoided than attempted to be handled. I am perfectly fine if you close the issue with a "Nothing to be done" resolution.

Indeed, the error comes from unescaped special characters in the upload URL, e.g. for a percent sign (dir%):

2024-01-26 04:42:59,392 INFO  [scheduler-1] HttpService:64 - Uploading file [/var/lib/go-agent-2/data/cruise-3594e111-3100-458a-ae59-6
cca85efc39a/1581bacf-940f-45c4-8b05-16f035aec923/test.log.zip] to url [http://12.34.56.78:8153/go/remoting/files/bug/25/s1/1/j1/dir%25?a
ttempt=1&buildId=122586]
2024-01-26 04:42:59,406 INFO  [scheduler-1] HttpService:120 - Got back 400 from server
2024-01-26 04:42:59,593 ERROR [scheduler-1] GoArtifactsManipulator:115 - Failed to upload /var/lib/go-agent-2/pipelines/bug/dir%/test.
log
java.lang.RuntimeException: Failed to upload /var/lib/go-agent-2/pipelines/bug/dir%/test.log.  HTTP return code is 400

[chadlwilson]

Does the server log anything? Stack trace etc? Thanks for the additional logs - it's possible it might expose a more general encoding problem that could be of concern, so at some point I'll take a look, although I am aware of some old issues in this area such as #11707 (and fixed some bits recently via #11692) so it might be related.

[asavchkov]

Yes, I shortened the stack trace in the previous message but here is it in full. This is what the agent logs, the server doesn't log any details, only prints the failed job status.

2024-01-26 04:42:59,392 INFO  [scheduler-1] HttpService:64 - Uploading file [/var/lib/go-agent-2/data/cruise-3594e111-3100-458a-ae59-6
cca85efc39a/1581bacf-940f-45c4-8b05-16f035aec923/test.log.zip] to url [http://12.34.56.78:8153/go/remoting/files/bug/25/s1/1/j1/dir%25?a
ttempt=1&buildId=122586]
2024-01-26 04:42:59,406 INFO  [scheduler-1] HttpService:120 - Got back 400 from server
2024-01-26 04:42:59,593 ERROR [scheduler-1] GoArtifactsManipulator:115 - Failed to upload /var/lib/go-agent-2/pipelines/bug/dir%/test.
log
java.lang.RuntimeException: Failed to upload /var/lib/go-agent-2/pipelines/bug/dir%/test.log.  HTTP return code is 400
        at com.thoughtworks.go.util.ExceptionUtils.bomb(ExceptionUtils.java:26)
        at com.thoughtworks.go.publishers.GoArtifactsManipulator.publish(GoArtifactsManipulator.java:110)
        at com.thoughtworks.go.work.DefaultGoPublisher.upload(DefaultGoPublisher.java:69)
        at com.thoughtworks.go.domain.ArtifactPlan.uploadArtifactFile(ArtifactPlan.java:168)
        at com.thoughtworks.go.domain.ArtifactPlan.publishBuildArtifact(ArtifactPlan.java:142)
        at com.thoughtworks.go.domain.ArtifactPlan.publishBuiltInArtifacts(ArtifactPlan.java:130)
        at com.thoughtworks.go.remote.work.artifact.ArtifactsPublisher.publishArtifacts(ArtifactsPublisher.java:75)
        at com.thoughtworks.go.remote.work.BuildWork.completeJob(BuildWork.java:208)
        at com.thoughtworks.go.remote.work.BuildWork.build(BuildWork.java:142)
        at com.thoughtworks.go.remote.work.BuildWork.doWork(BuildWork.java:92)
        at com.thoughtworks.go.agent.JobRunner.run(JobRunner.java:50)
        at com.thoughtworks.go.agent.AgentHTTPClientController.doWork(AgentHTTPClientController.java:136)
        at com.thoughtworks.go.agent.AgentHTTPClientController.tryDoWork(AgentHTTPClientController.java:115)
        at com.thoughtworks.go.agent.AgentController.performWork(AgentController.java:89)
        at com.thoughtworks.go.agent.AgentWorkRetrievalScheduler.run(AgentWorkRetrievalScheduler.java:60)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)