Skip to content

Conversation

@jorgee
Copy link
Contributor

@jorgee jorgee commented Nov 12, 2025

close #5888

.command.stage file is created when stage script is larger than 1MB and stage file is in the default filesystem. In the case of Google Batch executor, the stage file is /mnt/xxx/ because of the gcsfuse. Later, it tries to write the file that doesn't exist in the head node, so it fails. It is mainly due to the targetStage file returns the mount path instead of the remote path as for .command.run, etc.

In this PR, I have fixed the error by returning he remote version of the stage file instead of the localone when requesting the targetStageFile.

@netlify
Copy link

netlify bot commented Nov 12, 2025

Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit 9d31376
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/6914a52e97fc410008ee8671
😎 Deploy Preview https://deploy-preview-6558--nextflow-docs-staging.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@bentsherman
Copy link
Member

Ideally it should not write the stage script at all for google batch. That was the intention of this check:

// enable only when the stage uses the default file system, i.e. it's not a remote object storage file
// see https://github.com/nextflow-io/nextflow/issues/4279
if( stageFile.fileSystem == FileSystems.default && stagingScript.size() >= stageFileThreshold.bytes ) {
stageScript = stagingScript
return header + "bash ${stageFile}"
}
else
return header + stagingScript

But I didn't consider that the google batch script launcher overwrites the task workDir to be the local container-mounted path. So I'm not sure if there is a clean way to fix this check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Staging script not found when running on Google Batch

3 participants