Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support dynamic variables for S3 path configs #22

Open
5 tasks
zstumgoren opened this issue Sep 26, 2019 · 0 comments
Open
5 tasks

Support dynamic variables for S3 path configs #22

zstumgoren opened this issue Sep 26, 2019 · 0 comments

Comments

@zstumgoren
Copy link
Contributor

zstumgoren commented Sep 26, 2019

We should add the ability to dynamically set the S3 path prefix and suffix based on common variables.

This would simplify user workflow especially when using date-based directories to separate projects in an S3 bucket. For example, a variable for YEAR would free users from having to remember to update the plugin configuration file when January 1st arrives.

A running list of variables to support:

  • YEAR
  • MONTH
  • DAY
  • PROJECTNAME
  • USERNAME

Background/Implementation

The plugin currently supports the ability to configure prefix and suffix path components for S3 in a plugin-level configuration file, which is used to generate a project-level config.

The plugin-level config should support dynamic configurations such as below:

# ~/.datakit/plugins/datakit-data/config.json
{
  "aws_user_profile": "stanford",
  "s3_bucket": "stanford-data-projects",
  "s3_path_prefix": "$YEAR/"
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant