Allow the user to specify a KMS key in the event of bucket encryption

[jdeprin]

Currently, there is no way to transfer out of a bucket encrypted with KMS. If you try the transfer job fails. Ideally, the DTH task would allow a user to enter the KMS key ARN or ID and include appropriate IAM permissions in the auto-generated policies.

[YikaiHu]

Hi @jdeprin , currently you can refer to this guide:
https://github.com/awslabs/data-transfer-hub/blob/main/docs/S3-SSE-KMS-Policy.md

[os-henryngo]

The instructions in the guide doesn't work if you try to add it to the source S3 bucket policy. The principal is missing and you really can't add the KMS resource in the bucket policy. Can you elaborate how to properly implement it?

[YikaiHu]

Hi @openspace-henryngo , sorry this is a documentation mistake, we will update this soon.

This policy is not the S3 bucket policy, it is the policy for the IAM user which Credential stored in the Secrets Manager.

For example, if you deployed the Data Transfer Hub in AWS Account A China Region, and want to pull S3 data from AWS Account B US-west-2 Region, and the source bucket in us-west-2 is encrypted this KMS.

• You need to create an IAM user in AWS Account B, with the policy in the guide. And write down the AccessKeyID and SecretsAccessKey.
• Then, you should go to AWS Account A China Region's Secrets Manager to create a secrets, using AK and SK from the previous step. Input the credentials as text in Plaintext, the credentials format should follow:

{
  "access_key_id": "<Your Access Key ID>",
  "secret_access_key": "<Your Access Key Secret>"
}

[os-henryngo]

Hi @YikaiHu. I deployed the DTH in the US region (Account A), the origin bucket is in the same region and the destination bucket in China (Account B). I ended up creating 2 credentials in Account A Secrets Manager. The 1st one will access the Account A S3 bucket and has KMS decrypt permissions and the other is the IAM user in Account B which has the write access to the Account B/destination S3 bucket.

[YikaiHu]

I deployed the DTH in the US region (Account A), the origin bucket is in the same region and the destination bucket in China (Account B). I ended up creating 2 credentials in Account A Secrets Manager. The 1st one will access the Account A S3 bucket and has KMS decrypt permissions and the other is the IAM user in Account B which has the write access to the Account B/destination S3 bucket.

Hi @openspace-henryngo ,This scenario will required some additional steps.

1. When you creating the transfer job in DTH, you choose the Account B's secret key as the credential for Destination.
2. After the task is created, go to the AWS CloudFormation in your Region, you can see a sub CloudFormation Stack named DTH-S3EC2-XXXXX.
3. Select this sub stack, and go to the its Resources tab.
4. Search for EC2WorkerStackWorkerAsgRole, and attach the required KMS policy to this role, you can refer to this.
5. Search for ECSStackJobFinderTaskDefTaskRole, and attach the required KMS policy to this role, you can refer to this.
6. Then you just have to wait until the next Finder execution cycle and the data will be transferred, the default execution interval is one hour.
7. Or you can trigger the finder job manually if you don't want to wait for the execution interval.
   1. Go to the AWS ECS Cluster in your region.
   2. Choose the cluster named DataTransferHub-TaskCluster-xxxxxxxxxxxxx
   3. Click Tasks tab, and click Run New Task, refer to the attachment picture1.
   4. Config the Launch type as Fargate, Operating system family as Linux , Task Definition as your specific transfer task, Cluster VPC as DTH Created VPC, Subnets. Then, click Run Task. Refer to the attachment
      picture2.

[os-henryngo]

I think creating an IAM user in the source account with KMS decrypt permission to the source bucket is a faster option at this time.

[YikaiHu]

I think creating an IAM user in the source account with KMS decrypt permission to the source bucket is a faster option at this time.

But the EC2 launched by Data Transfer Hub will not automatically use the IAM user you created.

[os-henryngo]

But the EC2 launched by Data Transfer Hub will not automatically use the IAM user you created.

It's not clear to me what you mean by this but it's been working for me so far. For clarification, in the Source settings, I have to select 'No' for 'Is bucket in this account section then I select the appropriate Secrets under Credentials Store. There are no errors from the EC2s launch and the transfer completes. The task status remains 'In Progress' indefinitely but after I confirm both buckets have the same size I just go ahead and stop the task.

[kaplanan]

The solution with creating credentials for the KMS encrypted source bucket only works if the Source setting for "Is Bucket in this account" is set to "No". This, however, removes the option to "Enable S3 Event" which is very relevant for Fixed Rate Transfer Tasks on Changes within the S3 Bucket. I think that this is a very common use case across all customers, which should be enabled, even if the source bucket is "not in the current account"