DNSEntry cannot be created because of duplicates?

[rfranzke]

How to categorize this issue?

/area usability
/kind bug
/priority normal

What happened:
The creation of new DNSEntrys is sometimes erroneous with the following message:

task "Deploying external domain DNS record" failed: Failed to create "external" DNS record: "DNS record \"external\" is not ready (status=Error, message=InvalidChangeBatch: [Tried to create resource record set [name='comment-api.cluster.example.com.', type='TXT'] but it already exists, Tried to create resource record set [name='api.cluster.example.com.', type='A'] but it already exists]\n\tstatus code: 400, request id: dac27459-c70c-46f5-83b3-165d430de657)" (status=Error, message=InvalidChangeBatch: [Tried to create resource record set [name='comment-api.cluster.example.com.', type='TXT'] but it already exists, Tried to create resource record set [name='api.cluster.example.com.', type='A'] but it already exists]
    status code: 400, request id: dac27459-c70c-46f5-83b3-165d430de657)

After some time this error self-heals again and the DNSEntry comes up.

What you expected to happen:
DNSEntry can be created successfully without intermediate error

How to reproduce it (as minimally and precisely as possible):
n/a, sorry

[MartinWeindel]

This seems to be a concurrency problem. Here are two worker goroutines passing the same records in a batch:

time="2021-03-19T07:17:39Z" level=info msg="controllers: compound: dns: 38: cmd:hostedzone:ZXXXXXX: aws-route53: CREATE TXT record set *.comment-ingress.hxtp.294ece.xxxxxx.[ZXXXXXX]: [\"owner=shoot--hc-us21--294ece-hxtp-ba5e863e-02e9-42ca-8e0e-4d74cc480a30-sap-landscape-live-ingress\", \"prefix=comment-\"](600)"
time="2021-03-19T07:17:39Z" level=info msg="controllers: compound: dns: 38: cmd:hostedzone:ZXXXXXX: aws-route53: CREATE A record set *.ingress.hxtp.294ece.xxxxxx.[ZXXXXXX]: [52.186.163.108](120)"
time="2021-03-19T07:17:39Z" level=info msg="controllers: compound: dns: 38: cmd:hostedzone:ZXXXXXX: aws-route53: desired change: CREATE *.comment-ingress.hxtp.294ece.xxxxxx. TXT"
time="2021-03-19T07:17:39Z" level=info msg="controllers: compound: dns: 38: cmd:hostedzone:ZXXXXXX: aws-route53: desired change: CREATE *.ingress.hxtp.294ece.xxxxxx. A"
time="2021-03-19T07:34:01Z" level=info msg="controllers: compound: dns: 70: cmd:hostedzone:ZXXXXXX: aws-route53: CREATE TXT record set *.comment-ingress.hxtp.294ece.xxxxxx.[ZXXXXXX]: [\"owner=shoot--hc-us21--294ece-hxtp-ba5e863e-02e9-42ca-8e0e-4d74cc480a30-sap-landscape-live-ingress\", \"prefix=comment-\"](600)"
time="2021-03-19T07:34:01Z" level=info msg="controllers: compound: dns: 70: cmd:hostedzone:ZXXXXXX: aws-route53: CREATE A record set *.ingress.hxtp.294ece.xxxxxx.[ZXXXXXX]: [52.186.163.108](120)"
time="2021-03-19T07:34:01Z" level=info msg="controllers: compound: dns: 70: cmd:hostedzone:ZXXXXXX: aws-route53: desired change: CREATE *.ingress.hxtp.294ece.xxxxxx. A"
time="2021-03-19T07:34:01Z" level=info msg="controllers: compound: dns: 70: cmd:hostedzone:ZXXXXXX: aws-route53: desired change: CREATE *.comment-ingress.hxtp.294ece.xxxxxx. TXT"
time="2021-03-19T07:34:02Z" level=error msg="controllers: compound: dns: 70: cmd:hostedzone:ZXXXXXX: aws-route53: 2 records in zone ZXXXXXX fail: InvalidChangeBatch: [Tried to create resource record set [name='\\052.ingress.hxtp.294ece.xxxxxx.', type='A'] but it already exists, Tried to create resource record set [name='\\052.comment-ingress.hxtp.294ece.xxxxxx.', type='TXT'] but it already exists]\n\tstatus code: 400, request id: ee105efd-93fe-4d65-95e4-e9aa2dfea0df"