Drop support for metadata records

README.md

@@ -30,6 +30,29 @@ For a detailed explanation of the model, see section [The Model](#the-model).
 For extending or adapting this project with your own source or provisioning controllers, see section
 [Extensions](#extensions)
 
+## Important Note: Support for owner identifiers is discontinued
+
+Starting with release `v0.23`, the support for owner identifiers is discontinued.
+
+The creation and management of meta data DNS records holding the owner identifier for each `DNSEntry` has been removed.
+These meta data DNS records will be removed automatically. To avoid running into rate limits, this removals will happen
+only during updates or with low batch size during the periodic reconciliation.
+Depending on size of the hosted zone these cleanup can take multiple days.
+To ensure the correct work of the cleanup of these special `TXT` DNS records, the owner identifier provided via 
+the `--identifier` command line option and the `DNSOwner` custom resources need still be provided as before.
+In a future release, the `DNSOwner` resources will be removed completely.
+
+These identifiers are now only used for cleanup, but not for any other purposes.
+
+The ownership information was used to several purposes:
+- detect conflicts (i.e. same DNS record written by multiple dns-controller-manager instances)
+- handing over responsibility of DNS records from one to another controller instance
+- detection of orphan DNS records and their cleanup
+
+Please note that these edge cases are not supported anymore.
+For handing over responsibility of DNS record, please use the `dns.gardener.cloud/ignore=true` annotation 
+on `DNSEntries` or the annotated source objects (like `Ingress`, `Service`, etc.)
+
 ## Quick start
 
 To install the <b>DNS controller manager</b> in your Kubernetes cluster, follow these steps.
@@ -406,31 +429,15 @@ and/or DNS zone identifiers to override the scanning results of the account.
 
 ### Owner Identifiers
 
-Every DNS Provisioning Controller is responsible for a set of _Owner Identifiers_.
-DNS records in an external DNS environment are attached to such an identifier.
-This is used to identify the records in the DNS environment managed by a dedicated
-controller (manager). Every controller manager hosting DNS Provisioning Controllers
-offers an option to specify a default identifier. Additionally there might
-be dedicated `DNSOwner` objects that enable or disable additional owner ids.
-
-Every `DNSEntry` object may specify a dedicated owner that is used to tag
-the records in the DNS environment. A DNS provisioning controller only acts
-of DNS entries it is responsible for. Other resources in the external DNS
-environment are not touched at all.
-
-This way it is possbible to
-- identify records in the external DNS management environment that are managed
-  by the actual controller instance
-- distinguish different DNS source environments sharing the same hosted zones
-  in the external management environment
-- cleanup unused entries, even if the whole resource set is already
-  gone
-- move the responsibility for dedicated sets of DNS entries among different
-  kubernetes clusters or DNS source environments running different
-  DNS Provisioning Controller without loosing the entries during the
-  migration process.
-
-**If multiple DNS controller instances have access to the same DNS zones, it is very important, that every instance uses a unique owner identifier! Otherwise the cleanup of stale DNS record will delete entries created by another instance if they use the same identifier.**
+Starting with release `v0.23`, owner identifier are no longer supported.
+Formerly, every DNS Provisioning Controller was responsible for a set of _Owner Identifiers_.
+For every DNS record, there was an additional `TXT` DNS record ("metadata record") referencing the owner identifier.
+It was decided to remove this feature, as it doubles the number of DNS records without adding
+enough value.
+
+In the release `v0.23`, it is still important to specify the `--identifier` option for the compound DNS
+Provisioning Controller and also to keep the `DNSOwner` resources as they are used to clean up the "metadata records".
+In the future, the `DNSOwner` resources will be removed completely.
 
 ### DNS Classes
 
@@ -531,12 +538,13 @@ The following provider types can be selected (comma separated):
 - `remote`: Remote DNS provider (a dns-controller-manager with enabled remote access service)
 - `powerdns`: PowerDNS provider
 
-If the compound DNS Provisioning Controller is enabled it is important to specify a
-unique controller identity using the `--identifier` option.
-This identifier is stored in the DNS system to identify the DNS entries
-managed by a dedicated controller. There should never be two
-DNS controllers with the same identifier running at the same time for the
-same DNS domains/accounts.
+If the compound DNS Provisioning Controller is enabled, a unique controller identity was specified using the
+`--identifier` option in former release.
+This identifier was used to tag the DNS entries managed by a dedicated controller by creating additional
+"metadata `TXT` records" in the DNS system.
+Starting with release `v0.23`, this feature has been dropped as it doubles the number of DNS records.
+It is still important and required to specify the `--identifier` option to enable the cleanup of "metadata records" 
+created by former releases of the DNS Provisioning Controller.
 
 Here is the complete list of options provided:
 
@@ -640,6 +648,7 @@ Flags:
       --compound.infoblox-dns.ratelimiter.enabled                     enables rate limiter for DNS provider requests of controller compound
       --compound.infoblox-dns.ratelimiter.qps int                     maximum requests/queries per second of controller compound
       --compound.lock-status-check-period duration                    interval for dns lock status checks of controller compound
+      --compound.max-metadata-record-deletions-per-reconciliation int   maximum number of metadata owner records that can be deleted per zone reconciliation of controller compound
       --compound.netlify-dns.advanced.batch-size int                  batch size for change requests (currently only used for aws-route53) of controller compound
       --compound.netlify-dns.advanced.max-retries int                 maximum number of retries to avoid paging stops on throttling (currently only used for aws-route53) of controller compound
       --compound.netlify-dns.blocked-zone zone-id                     Blocks a zone given in the format zone-id from a provider as if the zone is not existing. of controller compound
@@ -680,7 +689,6 @@ Flags:
       --compound.rfc2136.ratelimiter.qps int                          maximum requests/queries per second of controller compound
       --compound.secrets.pool.size int                                Worker pool size for pool secrets of controller compound
       --compound.setup int                                            number of processors for controller setup of controller compound
-      --compound.statistic.pool.size int                              Worker pool size for pool statistic of controller compound
       --compound.ttl int                                              Default time-to-live for DNS entries. Defines how long the record is kept in cache by DNS servers or resolvers. of controller compound
       --compound.zonepolicies.pool.size int                           Worker pool size for pool zonepolicies of controller compound
       --config string                                                 config file
@@ -812,6 +820,7 @@ Flags:
       --lock-status-check-period duration                             interval for dns lock status checks
   -D, --log-level string                                              logrus log level
       --maintainer string                                             maintainer key for crds (default "dns-controller-manager")
+      --max-metadata-record-deletions-per-reconciliation int          maximum number of metadata owner records that can be deleted per zone reconciliation      
       --name string                                                   name used for controller manager (default "dns-controller-manager")
       --namespace string                                              namespace for lease (default "kube-system")
   -n, --namespace-local-access-only                                   enable access restriction for namespace local access only (deprecated)
@@ -884,7 +893,6 @@ Flags:
       --service-dns.target-set-ignore-owners                          mark generated DNS entries to omit owner based access control of controller service-dns
       --service-dns.targets.pool.size int                             Worker pool size for pool targets of controller service-dns
       --setup int                                                     number of processors for controller setup
-      --statistic.pool.size int                                       Worker pool size for pool statistic
       --target string                                                 target cluster for dns requests
       --target-creator-label-name string                              label name to store the creator for replicated DNS providers, label name to store the creator for generated DNS entries
       --target-creator-label-value string                             label value for creator label
@@ -1178,8 +1186,6 @@ The Gardener DNS controller uses a custom resource DNSProvider to dynamically ma
 
 A DNS provider can also restrict its actions on subset of the DNS domains (includes and excludes) for which the credentials are capable to edit.
 
-Each provider can define a separate “owner” identifier, to differentiate DNS entries in the same DNS zone from different providers.
-
 3. Multi cluster support
 
 The Gardener DNS controller distinguish three different logical Kubernetes clusters: Source cluster, target cluster and runtime cluster. The source cluster is monitored by the DNS source controllers for annotations on ingress and service resources. These controllers then create DNS entries in the target cluster. DNS entries in the target cluster are then reconciliated/synchronized with the corresponding DNS backend service by the provider controller. The runtime cluster is the cluster the DNS controller runs on. For example, this enables needed flexibility in the Gardener deployment. The DNS controller runs on the seed cluster. This is also the target cluster. DNS providers and entries resources are created in the corresponding namespace of the shoot control plane, while the source cluster is the shoot cluster itself.

charts/external-dns-management/templates/deployment.yaml

@@ -324,6 +324,9 @@ spec:
         {{- if .Values.configuration.compoundLockStatusCheckPeriod }}
         - --compound.lock-status-check-period={{ .Values.configuration.compoundLockStatusCheckPeriod }}
         {{- end }}
+        {{- if .Values.configuration.compoundMaxMetadataRecordDeletionsPerReconciliation }}
+        - --compound.max-metadata-record-deletions-per-reconciliation={{ .Values.configuration.compoundMaxMetadataRecordDeletionsPerReconciliation }}
+        {{- end }}
         {{- if .Values.configuration.compoundNetlifyDnsAdvancedBatchSize }}
         - --compound.netlify-dns.advanced.batch-size={{ .Values.configuration.compoundNetlifyDnsAdvancedBatchSize }}
         {{- end }}
@@ -420,9 +423,6 @@ spec:
         {{- if .Values.configuration.compoundSetup }}
         - --compound.setup={{ .Values.configuration.compoundSetup }}
         {{- end }}
-        {{- if .Values.configuration.compoundStatisticPoolSize }}
-        - --compound.statistic.pool.size={{ .Values.configuration.compoundStatisticPoolSize }}
-        {{- end }}
         {{- if .Values.configuration.compoundTtl }}
         - --compound.ttl={{ .Values.configuration.compoundTtl }}
         {{- end }}
@@ -801,6 +801,9 @@ spec:
         {{- if .Values.configuration.maintainer }}
         - --maintainer={{ .Values.configuration.maintainer }}
         {{- end }}
+        {{- if .Values.configuration.maxMetadataRecordDeletionsPerReconciliation }}
+        - --max-metadata-record-deletions-per-reconciliation={{ .Values.configuration.maxMetadataRecordDeletionsPerReconciliation }}
+        {{- end }}
         {{- if .Values.configuration.namespace }}
         - --namespace={{ .Values.configuration.namespace }}
         {{- end }}
@@ -975,9 +978,6 @@ spec:
         {{- if .Values.configuration.setup }}
         - --setup={{ .Values.configuration.setup }}
         {{- end }}
-        {{- if .Values.configuration.statisticPoolSize }}
-        - --statistic.pool.size={{ .Values.configuration.statisticPoolSize }}
-        {{- end }}
         {{- if .Values.configuration.target }}
         - --target={{ .Values.configuration.target }}
         {{- end }}

charts/external-dns-management/values.yaml

@@ -132,6 +132,7 @@ configuration:
   # compoundInfobloxDnsRatelimiterEnabled:
   # compoundInfobloxDnsRatelimiterQps:
   # compoundLockStatusCheckPeriod:
+  # compoundMaxMetadataRecordDeletionsPerReconciliation:
   # compoundNetlifyDnsAdvancedBatchSize:
   # compoundNetlifyDnsAdvancedMaxRetries:
   # compoundNetlifyDnsRatelimiterBurst:
@@ -164,7 +165,6 @@ configuration:
   # compoundRfc2136RatelimiterQps:
   # compoundSecretsPoolSize: 2
   # compoundSetup: 10
-  # compoundStatisticPoolSize:
   # compoundTtl: 120
   # compoundZonepoliciesPoolSize:
   # config:
@@ -291,6 +291,7 @@ configuration:
   # lockStatusCheckPeriod:
   # logLevel: info
   # maintainer:
+  # maxMetadataRecordDeletionsPerReconciliation:
   # namespace: default
   # namespaceLocalAccessOnly: false
   # netlifyDnsAdvancedBatchSize:
@@ -350,7 +351,6 @@ configuration:
   # serviceDNSTargetsPoolSize: 2
   # servicesPoolSize:
   # setup: 10
-  # statisticPoolSize:
   # target: ""
   # targetCreatorLabelName: ""
   # targetCreatorLabelValue: ""

examples/60-owner.yaml

@@ -1,3 +1,5 @@
+# Please note that starting with release `v0.23` DNSOwner resources are only used to clean up metadata DNS records.
+# The owner identifiers are not used for any other purpose anymore.
 apiVersion: dns.gardener.cloud/v1alpha1
 kind: DNSOwner
 metadata:

pkg/controller/provider/aws/execution.go

@@ -72,8 +72,8 @@ func buildResourceRecordSet(ctx context.Context, name dns.DNSSetName, policy *dn
 }
 
 func (this *Execution) addChange(ctx context.Context, action route53types.ChangeAction, req *provider.ChangeRequest, dnsset *dns.DNSSet) error {
-	name, rset := dns.MapToProvider(req.Type, dnsset, this.zone.Domain())
-	name = name.Align()
+	name := dnsset.Name.Align()
+	rset := dnsset.Sets[req.Type]
 	if len(rset.Records) == 0 {
 		return nil
 	}

pkg/controller/provider/azure-private/execution.go

@@ -56,12 +56,12 @@ func (exec *Execution) buildRecordSet(req *provider.ChangeRequest) (buildStatus,
 		return bs_invalidRoutingPolicy, "", nil
 	}
 
-	setName, rset := dns.MapToProvider(req.Type, dnsset, exec.zoneName)
-	name, ok := utils.DropZoneName(setName.DNSName, exec.zoneName)
+	name, ok := utils.DropZoneName(dnsset.Name.DNSName, exec.zoneName)
 	if !ok {
 		return bs_invalidName, "", &armprivatedns.RecordSet{Name: &name}
 	}
 
+	rset := dnsset.Sets[req.Type]
 	if len(rset.Records) == 0 {
 		return bs_empty, "", nil
 	}

pkg/controller/provider/azure/execution.go

@@ -56,12 +56,12 @@ func (exec *Execution) buildRecordSet(req *provider.ChangeRequest) (buildStatus,
 		return bs_invalidRoutingPolicy, "", nil
 	}
 
-	setName, rset := dns.MapToProvider(req.Type, dnsset, exec.zoneName)
-	name, ok := utils.DropZoneName(setName.DNSName, exec.zoneName)
+	name, ok := utils.DropZoneName(dnsset.Name.DNSName, exec.zoneName)
 	if !ok {
 		return bs_invalidName, "", &armdns.RecordSet{Name: &name}
 	}
 
+	rset := dnsset.Sets[req.Type]
 	if len(rset.Records) == 0 {
 		return bs_empty, "", nil
 	}

pkg/controller/provider/google/execution.go

@@ -51,11 +51,13 @@ func (this *Execution) addChange(req *provider.ChangeRequest) {
 	var err error
 
 	if req.Addition != nil {
-		setName, newset = dns.MapToProvider(req.Type, req.Addition, this.zone.Domain())
+		setName = req.Addition.Name
+		newset = req.Addition.Sets[req.Type]
 		policy, err = extractRoutingPolicy(req.Addition)
 	}
 	if req.Deletion != nil {
-		setName, oldset = dns.MapToProvider(req.Type, req.Deletion, this.zone.Domain())
+		setName = req.Deletion.Name
+		oldset = req.Deletion.Sets[req.Type]
 		if req.Addition == nil {
 			policy, err = extractRoutingPolicy(req.Deletion)
 		}

pkg/controller/provider/openstack/execution.go

@@ -52,8 +52,8 @@ func (exec *Execution) buildRecordSet(req *provider.ChangeRequest) (buildStatus,
 		return bsInvalidRoutingPolicy, nil
 	}
 
-	name, rset := dns.MapToProvider(req.Type, dnsset, exec.zone.Domain())
-
+	name := dnsset.Name
+	rset := dnsset.Sets[req.Type]
 	if len(rset.Records) == 0 {
 		return bsEmpty, nil
 	}

pkg/controller/provider/openstack/handler_test.go

@@ -268,24 +268,12 @@ func TestGetZoneStateAndExecuteRequests(t *testing.T) {
 			Type:    "A",
 			Records: []string{"1.2.3.4", "5.6.7.8"},
 		},
-		{
-			Name:    "comment-sub1.z1.test.",
-			TTL:     600,
-			Type:    "TXT",
-			Records: []string{"\"owner=test\"", "\"prefix=comment-\""},
-		},
 		{
 			Name:    "sub2.z1.test.",
 			TTL:     302,
 			Type:    "CNAME",
 			Records: []string{"cname.target.test."},
 		},
-		{
-			Name:    "comment-sub2.z1.test.",
-			TTL:     600,
-			Type:    "TXT",
-			Records: []string{"\"owner=test\"", "\"prefix=comment-\""},
-		},
 		{
 			Name:    "sub3.z1.test.",
 			TTL:     303,
@@ -298,23 +286,20 @@ func TestGetZoneStateAndExecuteRequests(t *testing.T) {
 		Ω(err).ShouldNot(HaveOccurred(), fmt.Sprintf("CreateRecordSet failed for %s %s", opts.Name, opts.Type))
 	}
 
-	stdMeta := buildRecordSet("META", 600, "\"owner=test\"", "\"prefix=comment-\"")
 	sub1 := dns.DNSSetName{DNSName: "sub1.z1.test"}
 	sub2 := dns.DNSSetName{DNSName: "sub2.z1.test"}
 	sub3 := dns.DNSSetName{DNSName: "sub3.z1.test"}
 	expectedDnssets := dns.DNSSets{
 		sub1: &dns.DNSSet{
 			Name: sub1,
 			Sets: dns.RecordSets{
-				"A":    buildRecordSet("A", 301, "1.2.3.4", "5.6.7.8"),
-				"META": stdMeta,
+				"A": buildRecordSet("A", 301, "1.2.3.4", "5.6.7.8"),
 			},
 		},
 		sub2: &dns.DNSSet{
 			Name: sub2,
 			Sets: dns.RecordSets{
 				"CNAME": buildRecordSet("CNAME", 302, "cname.target.test"),
-				"META":  stdMeta,
 			},
 		},
 		dns.DNSSetName{DNSName: "sub3.z1.test"}: &dns.DNSSet{
@@ -343,16 +328,6 @@ func TestGetZoneStateAndExecuteRequests(t *testing.T) {
 				},
 			},
 		},
-		{
-			Action: provider.R_CREATE,
-			Type:   "META",
-			Addition: &dns.DNSSet{
-				Name: sub4,
-				Sets: dns.RecordSets{
-					"META": stdMeta,
-				},
-			},
-		},
 		{
 			Action: provider.R_UPDATE,
 			Type:   "A",
@@ -368,11 +343,6 @@ func TestGetZoneStateAndExecuteRequests(t *testing.T) {
 			Type:     "CNAME",
 			Deletion: expectedDnssets[sub2],
 		},
-		{
-			Action:   provider.R_DELETE,
-			Type:     "META",
-			Deletion: expectedDnssets[sub2],
-		},
 		{
 			Action:   provider.R_DELETE,
 			Type:     "TXT",
@@ -386,15 +356,13 @@ func TestGetZoneStateAndExecuteRequests(t *testing.T) {
 		sub1: &dns.DNSSet{
 			Name: sub1,
 			Sets: dns.RecordSets{
-				"A":    buildRecordSet("A", 305, "1.2.3.55", "5.6.7.8"),
-				"META": stdMeta,
+				"A": buildRecordSet("A", 305, "1.2.3.55", "5.6.7.8"),
 			},
 		},
 		sub4: &dns.DNSSet{
 			Name: sub4,
 			Sets: dns.RecordSets{
-				"A":    buildRecordSet("A", 304, "11.22.33.44"),
-				"META": stdMeta,
+				"A": buildRecordSet("A", 304, "11.22.33.44"),
 			},
 		},
 	}

pkg/controller/provider/powerdns/execution.go

@@ -42,7 +42,8 @@ func (exec *Execution) buildRecordSet(req *provider.ChangeRequest) (*RecordSet,
 		dnsset = req.Deletion
 	}
 
-	name, rset := dns.MapToProvider(req.Type, dnsset, exec.zone.Domain())
+	name := dnsset.Name
+	rset := dnsset.Sets[req.Type]
 
 	if name.SetIdentifier != "" || dnsset.RoutingPolicy != nil {
 		return nil, fmt.Errorf("routing policies not supported for " + TYPE_CODE)

pkg/controller/provider/rfc2136/execution.go

@@ -39,10 +39,12 @@ func (exec *Execution) buildRecordSet(req *provider.ChangeRequest) ([]miekgdns.R
 
 	domain := dns.NormalizeHostname(exec.handler.zone)
 	if req.Addition != nil {
-		setName, addset = dns.MapToProvider(req.Type, req.Addition, domain)
+		setName = req.Addition.Name
+		addset = req.Addition.Sets[req.Type]
 	}
 	if req.Deletion != nil {
-		setName, delset = dns.MapToProvider(req.Type, req.Deletion, domain)
+		setName = req.Deletion.Name
+		delset = req.Deletion.Sets[req.Type]
 	}
 	if setName.DNSName == "" || (addset.Length() == 0 && delset.Length() == 0) {
 		return nil, nil, nil