Cert-manager pod is restarting and end up in crashloop back off error

[Snehil03]

Describe the bug:

Hello,
I am facing issue with restart of cert-manager pods ,
I am not see any issue with logs, kindly guide me what all things can be checked with respect to it,
Cert-Manager : 1.1.0
Kubernetes : 1.17
Deployment completed via Helm Chart.
Any pointers toward debug or fixing issue would be a really great help.
Below are the complete logs for pod :

resource "helm_release" "cert_manager" {
  name       = "cert-manager"
  namespace  = kubernetes_namespace.ingress.id
  #repository = data.helm_repository.jetstack.name
  chart      = "jetstack/cert-manager"
  version    = "v1.1.0"
  values = [file("${path.module}/cert-manager.yaml")]

  set {
    name  = "podAnnotations.iam\\.amazonaws\\.com/role"
    value = aws_iam_role.route_53_management_role.arn
  }
}

module "letsencrypt_issuers" {
  source = "../k8s_manifest"
  data = templatefile("${path.module}/certificate-issuers.yaml", {
    region = local.common.region
  })
  kubeconfig = var.kubeconfig
}


--- 
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ####
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
    - selector: {}
      http01:
        ingress:
          class: nginx
    - selector:
        matchLabels:
          use-dns-solver: "true"
      dns01:
        route53:
          region: ${region}
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: ###
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - selector: {}
      http01:
        ingress:
          class: nginx
    - selector:
        matchLabels:
          use-dns-solver: "true"
      dns01:
        route53:
          region: ${region}


----

cert-manager.yaml

---
installCRDs: true

ingressShim:
  enabled: true
  defaultIssuerName: letsencrypt-production
  defaultIssuerKind: ClusterIssuer
  defaultIssuerGroup: cert-manager.io

resources:
  requests:
    cpu: 10m
    memory: 120Mi
  limits:
    cpu: 20m
    memory: 250Mi

kubectl describe deploy/cert-manager -n ingress
Name: cert-manager
Namespace: ingress
CreationTimestamp: Fri, 14 May 2021 16:23:09 +0200
Labels: app=cert-manager
app.kubernetes.io/component=controller
app.kubernetes.io/instance=cert-manager
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=cert-manager
helm.sh/chart=cert-manager-v1.1.0
Annotations: deployment.kubernetes.io/revision: 2
meta.helm.sh/release-name: cert-manager
meta.helm.sh/release-namespace: ingress
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=cert-manager,app.kubernetes.io/name=cert-manager
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=cert-manager
app.kubernetes.io/component=controller
app.kubernetes.io/instance=cert-manager
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=cert-manager
helm.sh/chart=cert-manager-v1.1.0
Annotations: iam.amazonaws.com/role: arn:aws:iam::YYYY
prometheus.io/path: /metrics
prometheus.io/port: 9402
prometheus.io/scrape: true
Service Account: cert-manager
Containers:
cert-manager:
Image: quay.io/jetstack/cert-manager-controller:v1.1.0
Port: 9402/TCP
Host Port: 0/TCP
Args:
--v=2
--cluster-resource-namespace=$(POD_NAMESPACE)
--leader-election-namespace=kube-system
--default-issuer-name=letsencrypt-production
--default-issuer-kind=ClusterIssuer
--default-issuer-group=cert-manager.io
Limits:
cpu: 20m
memory: 250Mi
Requests:
cpu: 10m
memory: 120Mi
Environment:
POD_NAMESPACE: (v1:metadata.namespace)
Mounts:
Volumes:
Conditions:
Type Status Reason

---

Progressing True NewReplicaSetAvailable
Available True MinimumReplicasAvailable
OldReplicaSets:
NewReplicaSet: cert-manager-586d46d8b4 (1/1 replicas created)
Events:

---

kubectl describe po cert-manager-586d46d8b4-x75sx -n ingress
Name: cert-manager-586d46d8b4-x75sx
Namespace: ingress
Priority: 0
Node: ip-10-0-162-160.eu-west-1.compute.internal/10.0.162.160
Start Time: Mon, 17 May 2021 12:54:29 +0200
Labels: app=cert-manager
app.kubernetes.io/component=controller
app.kubernetes.io/instance=cert-manager
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=cert-manager
helm.sh/chart=cert-manager-v1.1.0
pod-template-hash=586d46d8b4
Annotations: iam.amazonaws.com/role: arn:aws:iam::XXXXX
kubernetes.io/psp: eks.privileged
prometheus.io/path: /metrics
prometheus.io/port: 9402
prometheus.io/scrape: true
Status: Running
IP: 10.0.129.80
IPs:
IP: 10.0.129.80
Controlled By: ReplicaSet/cert-manager-586d46d8b4
Containers:
cert-manager:
Container ID: docker://4085131dbb770d004f899fb43970520d7d60d6d46d71ef9f369c0e90e761aa6a
Image: quay.io/jetstack/cert-manager-controller:v1.1.0
Image ID: docker-pullable://quay.io/jetstack/cert-manager-controller@sha256:153d99b48570b053e599a03b918dbc80406ba1cf2137f14d5a80a9c7043ac06b
Port: 9402/TCP
Host Port: 0/TCP
Args:
--v=2
--cluster-resource-namespace=$(POD_NAMESPACE)
--leader-election-namespace=kube-system
--default-issuer-name=letsencrypt-production
--default-issuer-kind=ClusterIssuer
--default-issuer-group=cert-manager.io
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Tue, 18 May 2021 09:04:34 +0200
Finished: Tue, 18 May 2021 09:08:39 +0200
Ready: False
Restart Count: 154
Limits:
cpu: 20m
memory: 250Mi
Requests:
cpu: 10m
memory: 120Mi
Environment:
POD_NAMESPACE: ingress (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from cert-manager-token-nvbz9 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
cert-manager-token-nvbz9:
Type: Secret (a volume populated by a Secret)
SecretName: cert-manager-token-nvbz9
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Warning BackOff 9m45s (x3481 over 20h) kubelet Back-off restarting failed container
Normal Pulled 4m45s (x154 over 20h) kubelet Container image "quay.io/jetstack/cert-manager-controller:v1.1.0" already present on machine

---

Expected behaviour:
cluster issuer should generated all the api with acme.cert-manager.io/v1 not with acme.cert-manager.io/v1alpha2

Steps to reproduce the bug:
Re-deployed helm chart of cert-manager with 1.1.0
and checked generated apiversions of the CRDs like clusterissuer , it's coming as acme.cert-manager.io/v1alpha2

Also, certificate launched for generation is pending .

Cert-manager pods keep on restarting.

Anything else we need to know?:

Environment details::

• Kubernetes version: 1.17
• Cloud-provider/provisioner: AWS/ TERRAFORM (HELM)
• cert-manager version: 1.1.0
• Install method: e.g. helm/static manifests : HELM

/kind bug

[Snehil03]

Additional info : I have tried to create clusterissuer manually with kubectl apply on yaml file provided above as part of code description, It get created succcessfully but apiversions coming as acme.cert-manager.io/v1alpha2

kubectl apply -f  ClusterIssuer.yaml -n ingress
clusterissuer.cert-manager.io/letsencrypt-production created
clusterissuer.cert-manager.io/letsencrypt-staging created

[Snehil03]

I were able to fix these issue yesterday, it's because earlier certificate generated with cert-manager.io/v1 and after the cert-manager updated or reinstall of helm chart it moved to different apisversion cert-manager.io/v1alpha2 ,

I did not found these information as part of upgrade document .

https://cert-manager.io/v1.2-docs/installation/upgrading/upgrading-1.1-1.2/

Also in document found out that API is currently at v1 and stable
https://cert-manager.io/v1.2-docs/concepts/project-maturity/#api

What are the best steps to do be taken care during update to avoid this issue ?

[maelvls]

From what I understand, Helm returned an error when upgrading cert-manager from 1.1 to 1.2. Your hypothesis is that a previous version of cert-manager (1.0) had old v1alpha2 objects.

I would have expected the old v1alpha2 to still work after upgrading from 1.1 to 1.2 (as detailed in the the 1.0 release notes).

I still do not understand what caused your Pods to restart. That said, I noticed that the Pod had restarted once due to being OOM. From the above Pod status:

State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: OOMKilled

I don't have enough information to diagnose the issue 😞