S3 to S3 backup and sync

[jbygdell]

Describe the pull request:

• Bug fix
• Functional change
• New feature
• Code cleanup
• Build system change
• Documentation change
• Language translation

Pull request long description:

This PR add functionality to backup or sync S3 buckets in the same or different endpoints.

• In backup mode the files transferred to the destination bucket will be encrypted using the configured c4gh public key.
• For restoring the backed up objects the corresponding private key and passphrase are needed.
• When syncing two buckets the objects are not modified.
• A prefix can be set in the config to allow only a subset of objects to be selected, object path needs to begin with the prefix, that can be a string or a path. (test or foo/subset/bar)

Changes made:

1. Adds an option to backup one S3 bucket to a bucket in another S3 endpoint, data will be stored encrypted
2. Adds an option to restore encrypted data from one bucket to a bucket in another endpoint
3. Adds the option to sync two bucket in different endpoints
4. Both integration and code tests are added for these options.

Related issues:

Additional information:

Release note:

Documentation change:

Mentions:

[aaperis]

@jbygdell where are the go tests run in the workflow? are they buried in some container?

[jbygdell]

@jbygdell where are the go tests run in the workflow? are they buried in some container?

This repository never had any workflow for code tests, since there hasn't existed any test files.

[aaperis]

@jbygdell where are the go tests run in the workflow? are they buried in some container?

This repository never had any workflow for code tests, since there hasn't existed any test files.

Right, I see. I tried to run them locally and I got the following:

$ go test .

WORK/b001/_testmain.go:16:8: could not import main (cannot import "main")
FAIL	main [build failed]
FAIL

and

$ go test s3_test.go 

# command-line-arguments [command-line-arguments.test]
./s3_test.go:21:17: undefined: S3Config
./s3_test.go:36:15: undefined: S3Config
./s3_test.go:38:3: undefined: S3Port
./s3_test.go:91:13: undefined: newS3Backend
./s3_test.go:131:18: undefined: newS3Backend
./s3_test.go:137:11: undefined: newS3Backend
./s3_test.go:143:14: undefined: newS3Backend
./s3_test.go:157:14: undefined: newS3Backend
./s3_test.go:163:28: undefined: BackupS3BuckeEncrypted
./s3_test.go:189:18: undefined: newS3Backend
./s3_test.go:189:18: too many errors
FAIL    command-line-arguments [build failed]
FAIL

What am I missing?

[jbygdell]

What am I missing?

A bunch of other files (non go) that interferes.

run:

go test -v *.go

And you'll see that it works.

[aaperis]

Looks good, some small things to change.