[Bug]: Getting empty KMS key for DynamoDB replica

[dobriSu]

Terraform Core Version

v1.5.7,v1.8.1

AWS Provider Version

5.48.0,5.12.0

Affected Resource(s)

aws_dynamodb_table_replica

Expected Behavior

terraform import aws_dynamodb_table_replica.table_name_region TableName:<main_replica_region>
should import dynamoDB replica for region with KMS key.

Actual Behavior

terraform import aws_dynamodb_table_replica.table_name_region TableName:<main_replica_region>
sometime import dynamoDB replica without empty KMS key.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

N/A

Steps to Reproduce

1. Creating DynamoDB replica that encrypted with Customer Managed KMS key in AWS
2. import that replica to aws_dynamodb_table_replica resource

Debug Output

2024-05-07T18:21:45.027Z [WARN]  Provider "registry.terraform.io/hashicorp/aws" produced an unexpected new value for aws_dynamodb_table_replica.tunnel_config_replica_ap-south-1 during refresh.
      - .arn: was null, but now cty.StringVal("arn:aws:dynamodb:ap-south-1:171364143024:table/TunnelConfigs.v1")
      - .global_table_arn: was null, but now cty.StringVal("arn:aws:dynamodb:us-west-2:171364143024:table/TunnelConfigs.v1")
      - .kms_key_arn: was null, but now cty.StringVal("")
      - .point_in_time_recovery: was null, but now cty.False
      - .table_class_override: was null, but now cty.StringVal("")
      - .tags: was null, but now cty.MapValEmpty(cty.String)
      - .tags_all: was null, but now cty.MapValEmpty(cty.String)

2024-05-07T18:21:44.452Z [DEBUG] provider.terraform-provider-aws_v5.48.0_x5: HTTP Response Received: http.status_code=200 tf_rpc=ReadResource http.response.header.connection=keep-alive http.response.header.date="Tue, 07 May 2024 18:21:44 GMT" http.response.header.x_amz_crc32=3872555691 rpc.method=DescribeTable rpc.system=aws-api tf_mux_provider=*schema.GRPCProviderServer @caller=github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/[email protected]/logger.go:157 aws.region=ap-south-1 http.response.header.content_type=application/x-amz-json-1.0 http.response.header.server=Server rpc.service=DynamoDB tf_provider_addr=registry.terraform.io/hashicorp/aws tf_req_id=4018123c-35f5-322a-9fee-15360fd384df @module=aws http.duration=1730 http.response.header.x_amzn_requestid=O9FKLCUAC5MVRJBBUDD3K2QER3VV4KQNSO5AEMVJF66Q9ASUAAJG http.response_content_length=3520 tf_resource_type=aws_dynamodb_table_replica http.response.body="{"Table":{"AttributeDefinitions":[{"AttributeName":"HeadendID","AttributeType":"S"},{"AttributeName":"Version","AttributeType":"S"}],"BillingModeSummary":{"BillingMode":"PAY_PER_REQUEST","LastUpdateToPayPerRequestDateTime":1.714771244405E9},"CreationDateTime":1.714770410745E9,"DeletionProtectionEnabled":false,"GlobalTableVersion":"2019.11.21","ItemCount":1034,"KeySchema":[{"AttributeName":"HeadendID","KeyType":"HASH"},{"AttributeName":"Version","KeyType":"RANGE"}],"LatestStreamArn":"arn:aws:dynamodb:ap-south-1:171364143024:table/TunnelConfigs.v1/stream/2024-05-03T21:21:04.620","LatestStreamLabel":"2024-05-03T21:21:04.620","ProvisionedThroughput":{"NumberOfDecreasesToday":0,"ReadCapacityUnits":0,"WriteCapacityUnits":0},"Replicas":[{"RegionName":"ap-northeast-1","ReplicaStatus":"ACTIVE","ReplicaStatusDescription":"Failed to describe settings for the replica in region: ‘ap-northeast-1’."},{"RegionName":"us-east-1","ReplicaStatus":"ACTIVE","ReplicaStatusDescription":"Failed to describe settings for the replica in region: ‘us-east-1’."},{"RegionName":"ap-southeast-1","ReplicaStatus":"ACTIVE","ReplicaStatusDescription":"Failed to describe settings for the replica in region: ‘ap-southeast-1’."},{"RegionName":"eu-central-2","ReplicaStatus":"ACTIVE","ReplicaStatusDescription":"Failed to describe settings for the replica in region: ‘eu-central-2’."},{"RegionName":"ca-central-1","ReplicaStatus":"ACTIVE","ReplicaStatusDescription":"Failed to describe settings for the replica in region: ‘ca-central-1’."},{"RegionName":"eu-west-2","ReplicaStatus":"ACTIVE","ReplicaStatusDescription":"Failed to describe settings for the replica in region: ‘eu-west-2’."},{"RegionName":"ap-southeast-2","ReplicaStatus":"ACTIVE","ReplicaStatusDescription":"Failed to describe settings for the replica in region: ‘ap-southeast-2’."},{"RegionName":"il-central-1","ReplicaStatus":"ACTIVE","ReplicaStatusDescription":"Failed to describe settings for the replica in region: ‘il-central-1’."},{"RegionName":"eu-central-1","ReplicaStatus":"ACTIVE","ReplicaStatusDescription":"Failed to describe settings for the replica in region: ‘eu-central-1’."},{"RegionName":"ap-southeast-3","ReplicaStatus":"ACTIVE","ReplicaStatusDescription":"Failed to describe settings for the replica in region: ‘ap-southeast-3’."},{"RegionName":"ap-southeast-4","ReplicaStatus":"ACTIVE","ReplicaStatusDescription":"Failed to describe settings for the replica in region: ‘ap-southeast-4’."},{"RegionName":"us-west-2","ReplicaStatus":"ACTIVE","ReplicaStatusDescription":"Failed to describe settings for the replica in region: ‘us-west-2’."},{"RegionName":"eu-west-1","ReplicaStatus":"ACTIVE","ReplicaStatusDescription":"Failed to describe settings for the replica in region: ‘eu-west-1’."},{"RegionName":"us-west-1","ReplicaStatus":"ACTIVE","ReplicaStatusDescription":"Failed to describe settings for the replica in region: ‘us-west-1’."}],"SSEDescription":{"KMSMasterKeyArn":"arn:aws:kms:ap-south-1:171364143024:key/800814ac-d0b1-4098-b881-fffa30d74a9e","SSEType":"KMS","Status":"ENABLED"},"StreamSpecification":{"StreamEnabled":true,"StreamViewType":"NEW_AND_OLD_IMAGES"},"TableArn":"arn:aws:dynamodb:ap-south-1:171364143024:table/TunnelConfigs.v1","TableId":"9940f755-0993-41f2-a850-3d13b8a64bc9","TableName":"TunnelConfigs.v1","TableSizeBytes":515873,"TableStatus":"ACTIVE","TableThroughputModeSummary":{"LastUpdateToPayPerRequestDateTime":1.714771244405E9,"TableThroughputMode":"PAY_PER_REQUEST"}}}
" tf_aws.sdk=aws-sdk-go timestamp=2024-05-07T18:21:44.451Z

Panic Output

No response

Important Factoids

Probably it was caused by not guaranteed respond from AWS on DynamoDB tables DescribeTable output.
It is not documented but this is what I get from AWS Support:

I have investigated the issue, please find my response below.

Please note that replica settings populated for DynamoDB global table replicas in DescribeTable output is populated in a best effort manner on AWS console. Hence, to receive the full view of settings (including KMS key ARNs of all regions) for all replicas of a global table, we recommend calling DescribeTable API on each replica region's DynamoDB endpoint. Please also ensure the caller/IAM user has "dynamodb:DescribeTable" permission on each global table replica in every region. In other words "Manage Encryption Settings" page does the DescribeTable call and shows the details of all regions in a best effort manner and its recommended to make DescribeTable call on each region's DynamoDB endpoint to get the details.

The error message: "Failed to describe settings for the replica in regions:" is being shown in the console to better reflect that the observed behavior is due to failure to describe settings for the replica region as KMS key information for replicas is shown on best effort basis.

I hope the provided information is helpful. Please do let me know if you have any questions.

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[chrised]

To follow up @dobriSu's information, I think what's going on here is that when AttrKMSKeyARN is set for a aws_dynamodb_table_replica, it's based off the information from the primary/master instance (resourceTableReplicaRead). Sometimes (according to AWS Support) the information provided in the Replicas attribute isn't always fully formed (perhaps this is an issue with tables that have more replicas?).

The kicker for this, is that when the replica's description field isn't populated, the provider assumes that the key is nil. It follows up with a request to determine more information about the replica (resourceTableReplicaReadReplica) which has a valid (guaranteed!?) SSEDescription (and .KMSMasterKeyArn) for the replica. So the fix here, would be to set names.AttrKMSKeyARN's value in the resourceTableReplicaReadReplica function, guaranteeing that the value is set.

[chrised]

diff --git a/internal/service/dynamodb/table_replica.go b/internal/service/dynamodb/table_replica.go
index 92e8fa3b90..838818d9bc 100644
--- a/internal/service/dynamodb/table_replica.go
+++ b/internal/service/dynamodb/table_replica.go
@@ -297,6 +297,12 @@ func resourceTableReplicaReadReplica(ctx context.Context, d *schema.ResourceData
 		return create.AppendDiagError(diags, names.DynamoDB, create.ErrActionReading, resNameTableReplica, d.Id(), fmt.Errorf("continuous backups: %w", err))
 	}

+	if d.Get(names.AttrKMSKeyARN) == nil {
+		if table.SSEDescription.KMSMasterKeyArn != nil {
+			d.Set(names.AttrKMSKeyARN, table.SSEDescription.KMSMasterKeyArn)
+		}
+	}
+
 	if pitrOut != nil && pitrOut.ContinuousBackupsDescription != nil && pitrOut.ContinuousBackupsDescription.PointInTimeRecoveryDescription != nil {
 		d.Set("point_in_time_recovery", pitrOut.ContinuousBackupsDescription.PointInTimeRecoveryDescription.PointInTimeRecoveryStatus == awstypes.PointInTimeRecoveryStatusEnabled)
 	} else {

This might work, but there aren't any existing import tests I could utilise for inspiration to test it.

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[github-actions]

This functionality has been released in v5.51.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!