App.status.health.LastTransitionTime missing from Application Status

[liranZiv27]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug
Missing field in Application status, causing the suggested trigger for on-deployed template example to fail to send.

Noticed this log warn message by Application-controller

W0305 13:12:24.248939       7 warnings.go:70] unknown field "status.health.lastTransitionTime"

To Reproduce

1. Running ArgoCD instance with notifications-controller enabled.
2. on-deployed template + on-deployed trigger - from here
3. Application installed and set with on-deployed annotation notifications.argoproj.io/subscribe.on-deployed.slack: "slack-channel"
4. notifications-controller logs next message -

level=error msg="failed to execute when condition: reflect: Call using interface {} as type string (1:139)\n | app.status.operationState != nil and app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy' and !time.Parse(app.status.health.lastTransitionTime).Add(1 * time.Minute).Before(time.Parse(app.status.operationState.finishedAt))

Expected behavior
Notification sent for an application deployment successfully.

Screenshots

Version

Paste the output from `argocd version` here.
argocd: v2.14.4+3d901f2
  BuildDate: 2025-03-04T22:56:08Z
  GitCommit: 3d901f2037888af302a85f518bea70b33ee8e1c7
  GitTreeState: clean
  GoVersion: go1.24.0
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.14.2+ad27246

[liranZiv27]

Issue was created per @andrii-korotkov-verkada request.

[andrii-korotkov-verkada]

Did you recently upgrade from an earlier version? The last transition time was added in 2.14, so older apps may not have it. But once you deploy an app, it should get it.

[liranZiv27]

Did you recently upgrade from an earlier version? The last transition time was added in 2.14, so older apps may not have it. But once you deploy an app, it should get it. @andrii-korotkov-verkada

I have tried deploying after upgrading the version, no lastTransitionTime field in the Application status spec.

What's making me suspicious that the application-controller doesn't update this field, is this log message from the application-controller pod -
W0306 09:14:36.913198 6 warnings.go:70] unknown field "status.health.lastTransitionTime"

Can you help with that as well?
I even tried deleting an application and re-deploying it, but still no lastTransitionTime

[andrii-korotkov-verkada]

Ah, when upgrading the app, did you upgrade the install manifest as well including the CRD with types definitions?

[liranZiv27]

Using a helm release - 7.8.8
https://artifacthub.io/packages/helm/argo/argo-cd
Checked the Application CRD, LastTransitionTime is present.

[svghadi]

Seems like Argo CD CRDs are not updated to latest during upgrade. Can you confirm if CRDs are updated on cluster?

$ kubectl get crd applications.argoproj.io -o yaml|grep LastTransitionTime
.......
                      description: LastTransitionTime is the time the condition was
                    description: LastTransitionTime is the time the HealthStatus was
                          description: LastTransitionTime is the time the HealthStatus

I created a Argo CD Application with 2.14.4 and could see the LastTransitionTime field in status.

status:
....
  health:
    lastTransitionTime: "2025-03-06T11:37:53Z"
    status: Healthy

[liranZiv27]

It is present in the CRD

          status:
            description: ApplicationStatus contains status information for the application
            properties:
              conditions:
                description: Conditions is a list of currently observed application
                  conditions
                items:
                  description: ApplicationCondition contains details about an application
                    condition, which is usually an error or warning
                  properties:
                    lastTransitionTime:
                      description: LastTransitionTime is the time the condition was
                        last observed
                      format: date-time
                      type: string

[svghadi]

There are multiple lastTransitionTime fields in the Application CRD. The one you shared is under conditions. There should be another under health. Can you confirm?

status:
            description: ApplicationStatus contains status information for the application
            properties:
              conditions:
                description: Conditions is a list of currently observed application
                  conditions
                items:
                  description: ApplicationCondition contains details about an application
                    condition, which is usually an error or warning
                  properties:
                    lastTransitionTime:
                      description: LastTransitionTime is the time the condition was
                        last observed
                      format: date-time
                      type: string
              .......
              .......
              health:
                description: Health contains information about the application's current
                  health status
                properties:
                  lastTransitionTime:
                    description: LastTransitionTime is the time the HealthStatus was
                      set or updated
                    format: date-time
                    type: string
               ......

[liranZiv27]

It is not under Health:

  health:
    description: Health contains information about the application's current
      health status
    properties:
      message:
        description: Message is a human-readable informational message
          describing the health status
        type: string
      status:
        description: Status holds the status code of the application or
          resource
        type: string
    type: object

[svghadi]

Okay, then the CRDs are not updated during upgrade. Are CRD upgrades disabled in your helm chart install?
https://artifacthub.io/packages/helm/argo/argo-cd#custom-resource-definitions

[liranZiv27]

I will need to upgrade manually in that case.

[andrii-korotkov-verkada]

I recommend discussing at the contributors meeting about why upgrade doesn't happen https://docs.google.com/document/d/1xkoFkVviB70YBzSEa4bDnu-rUZ1sIFtwKKG1Uw8XsY8/edit?tab=t.0

[liranZiv27]

I figured out why the crd upgrade didn't happen.
The application is now updated with the lastTransitionTime field.
My issue now is that we prematurely send the template, before the update actually happens, even after using the trigger here -
when: app.status.operationState != nil and app.status.operationState.phase in ['Succeeded'] and app.status.health.status == 'Healthy' and !time.Parse(app.status.health.lastTransitionTime).Add(1 * time.Minute).Before(time.Parse(app.status.operationState.finishedAt))

[liranZiv27]

Example of timing - difference of ~2 min between deployment notification to actual deployment finish

[andrii-korotkov-verkada]

Maybe CRD needs a sync wave to be defined before the application controller? You can configure in your overlays if deploying argo with argo