argocd-notifications-controller: unmarshalling error invalid character '}' when updating argocd-notifications-cm.yaml

[mjtrangoni]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

After upgrading ArgoCD from 2.13.5 to 2.14.4, and upgrading argocd-notifications-cm.yaml to the 2.13.5 to the 2.14.4 status, no more MS Teams "IncomingWebhook" notification was sent anymore. See the argocd-notifications-controller logs.

To Reproduce

On a running ArgoCD, and an Application installation via helm, like this,

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kyverno-policy-reporter
  namespace: argocd
  annotations:
    # See https://kyverno.io/docs/writing-policies/mutate/#argocd-v210
    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true
    notifications.argoproj.io/subscribe.on-deployed.teams: argocd-deployments
    notifications.argoproj.io/subscribe.on-health-degraded.teams: argocd-deployments
    notifications.argoproj.io/subscribe.on-sync-failed.teams: argocd-deployments
    notifications.argoproj.io/subscribe.on-sync-running.teams: argocd-deployments
    notifications.argoproj.io/subscribe.on-sync-succeeded.teams: argocd-deployments
spec:
  destination:
    namespace: infra-admission-controller
    server: 'https://kubernetes.default.svc'
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=false
      - Replace=true
  source:
    repoURL: https://kyverno.github.io/policy-reporter
    targetRevision: 3.0.5
    chart: policy-reporter
    helm:
      values: |
        # See https://github.com/kyverno/policy-reporter/blob/main/charts/policy-reporter/values.yaml
        ui:
          enabled: true
          ingress:
            enabled: true
            className: internal
            annotations:
              cert-manager.io/cluster-issuer: letsencrypt
            hosts:
              - host: policy-reporter.xxx.com
                paths:
                  - path: /
                    pathType: Prefix
            tls:
              - secretName: kyverno-policy-reporter-tls
                hosts:
                  - policy-reporter.xxx.com

        plugin:
          kyverno:
            # -- (bool) Enable Kyverno Plugin
            enabled: true

        monitoring:
          enabled: true
          serviceMonitor:
            labels:
              team: infra

Update argocd-notifications-cm.yaml from 2.13.5 to 2.14.4

curl -Lv https://raw.githubusercontent.com/argoproj/argo-cd/refs/tags/v2.13.5/notifications_catalog/install.yaml > argocd-notifications-cm.yaml
curl -Lv https://raw.githubusercontent.com/argoproj/argo-cd/refs/tags/v2.14.4/notifications_catalog/install.yaml > argocd-notifications-cm.yaml

where our diff for MS Teams there from the vanilla one is,

 apiVersion: v1
 data:
+  context: |
+    argocdUrl: "https://gitops.xxx.com"
+  defaultTriggers: |
+    - on-sync-status-unknown
+  service.teams: |
+    recipientUrls:
+      argocd-deployments: $channel-teams-url

with the 2.13.5 version it works without problems, and with the 2.14.4 is not rendered.

Expected behavior

ArgoCD sending the same notifications as before.

Screenshots

Version

argocd version --grpc-web
argocd: v2.14.4+3d901f2
  BuildDate: 2025-03-04T22:56:08Z
  GitCommit: 3d901f2037888af302a85f518bea70b33ee8e1c7
  GitTreeState: clean
  GoVersion: go1.24.0
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.14.4+3d901f2
  BuildDate: 2025-03-04T20:39:46Z
  GitCommit: 3d901f2037888af302a85f518bea70b33ee8e1c7
  GitTreeState: clean
  GoVersion: go1.23.3
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.4.3 2024-07-19T16:40:33Z
  Helm Version: v3.16.3+gcfd0749
  Kubectl Version: v0.31.0
  Jsonnet Version: v0.20.0

Logs

argocd-notifications-controller:

time="2025-03-06T10:44:29Z" level=error msg="Failed to notify recipient {teams argocd-deployments} defined in resource argocd/kyverno-policy-reporter: teams facts unmarshalling error invalid character '}' looking for beginning of object key string using the configuration in namespace argocd" resource=argocd/kyverno-policy-reporter

[andrii-korotkov-verkada]

argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true

Can you remove IncludeMutationWebhook=true, please? Not sure it's causing the issue, but you generally don't want this.

[mjtrangoni]

@andrii-korotkov-verkada thank you for your quick response! This happens also with other apps without the IncludeMutationWebhook=true flag set.

[sivchari]

Hi @mjtrangoni
I open the PR to resolve it.
#22237

This bug suddenly happen whe you upgrade since it is only included in this v2.14.4

[mjtrangoni]

@sivchari Thank you! I can confirm it works now on my instance. Could we add a small test here for loading the configuration and checking if it is a valid JSON?

Another thing I found now is that ":arrow_heading_up:" is not rendered on MS Teams. The support for Markdown shortcuts is limited (very!). The only one alternative is using Unicode Emojis like ⬆️🔼💡🔥.

[sivchari]

@mjtrangoni
Great! Thank you for checking it.

Could we add a small test here for loading the configuration and checking if it is a valid JSON?

Sounds good to me. What do you think ? @andrii-korotkov-verkada

Another thing I found now is that ":arrow_heading_up:" is not rendered on MS Teams. The support for Markdown shortcuts is limited (very!). The only one alternative is using Unicode Emojis like ⬆️🔼💡🔥.

Interesting. Using unicode emojis is good way to visibility.

[andrii-korotkov-verkada]

Sounds good to me. What do you think ?

Sure, we can try adding a test

[sivchari]

Another thing I found now is that ":arrow_heading_up:" is not rendered on MS Teams. The support for Markdown shortcuts is limited (very!). The only one alternative is using Unicode Emojis like ⬆️🔼💡🔥.

It seems to be other problem with this one. So let's split issue and open it as a other one.