Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set repository_meta = "local" by default? #1427

Open
wlandau opened this issue Feb 7, 2025 · 0 comments
Open

Set repository_meta = "local" by default? #1427

wlandau opened this issue Feb 7, 2025 · 0 comments

Comments

@wlandau
Copy link
Member

wlandau commented Feb 7, 2025

repository_meta is an option in tar_option_set(). targets pipelines with repository_meta = "aws" (or "gcp") periodically upload _targets/meta/meta and other metadata (even debugging workspaces in targets >= 1.10.1) to a bucket in the cloud while the pipeline is running. The idea is to allow targets pipelines to run non-interactively, i.e. in an AWS Batch job, and allow the user's interactive session to download and view the data and metadata.

The current default for repository_meta is the same as repository, which means that repository = "aws" will upload both data and metadata to the cloud if the user does not manually set tar_option_set(repository_meta = "local").

In other words, pipelines opt into uploading metadata to buckets by default. On reflection, I am wondering if repository_meta = "local" would be a better default. targets pipelines are most convenient when you have an interactive R session (e.g. in Posit Workbench), the main process runs in the R console or an RStudio Background Job, and parallel workers run on Batch as needed. In other words, the pipeline is part of the interactive Workbench session, and users can stay in the comfort of their own familiar environment. And from a security standpoint, this seems feasible because Workbench can take care of AWS credentials behind the scenes.

If you use targets with AWS or GCP, what do you think? Should the default repository_meta be "local"?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant