You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
repository_meta is an option in tar_option_set(). targets pipelines with repository_meta = "aws" (or "gcp") periodically upload _targets/meta/meta and other metadata (even debugging workspaces in targets >= 1.10.1) to a bucket in the cloud while the pipeline is running. The idea is to allow targets pipelines to run non-interactively, i.e. in an AWS Batch job, and allow the user's interactive session to download and view the data and metadata.
The current default for repository_meta is the same as repository, which means that repository = "aws" will upload both data and metadata to the cloud if the user does not manually set tar_option_set(repository_meta = "local").
In other words, pipelines opt into uploading metadata to buckets by default. On reflection, I am wondering if repository_meta = "local" would be a better default. targets pipelines are most convenient when you have an interactive R session (e.g. in Posit Workbench), the main process runs in the R console or an RStudio Background Job, and parallel workers run on Batch as needed. In other words, the pipeline is part of the interactive Workbench session, and users can stay in the comfort of their own familiar environment. And from a security standpoint, this seems feasible because Workbench can take care of AWS credentials behind the scenes.
If you use targets with AWS or GCP, what do you think? Should the default repository_meta be "local"?
The text was updated successfully, but these errors were encountered:
repository_meta
is an option intar_option_set()
.targets
pipelines withrepository_meta = "aws"
(or"gcp"
) periodically upload_targets/meta/meta
and other metadata (even debugging workspaces intargets
>= 1.10.1) to a bucket in the cloud while the pipeline is running. The idea is to allowtargets
pipelines to run non-interactively, i.e. in an AWS Batch job, and allow the user's interactive session to download and view the data and metadata.The current default for
repository_meta
is the same asrepository
, which means thatrepository = "aws"
will upload both data and metadata to the cloud if the user does not manually settar_option_set(repository_meta = "local")
.In other words, pipelines opt into uploading metadata to buckets by default. On reflection, I am wondering if
repository_meta = "local"
would be a better default.targets
pipelines are most convenient when you have an interactive R session (e.g. in Posit Workbench), the main process runs in the R console or an RStudio Background Job, and parallel workers run on Batch as needed. In other words, the pipeline is part of the interactive Workbench session, and users can stay in the comfort of their own familiar environment. And from a security standpoint, this seems feasible because Workbench can take care of AWS credentials behind the scenes.If you use
targets
with AWS or GCP, what do you think? Should the defaultrepository_meta
be"local"
?The text was updated successfully, but these errors were encountered: