Service X for template 'Y': service X already exists and is not owned by experiment template Y

[meeech]

Checklist:

• I've included steps to reproduce the bug.
• I've included the version of argo rollouts.
  1.6.6 but observed in earlier versions

Describe the bug

RolloutAborted: Rollout aborted update to revision 5: Failed to create
    Service experiment for template 'exp': service experiment already exists and
    is not owned by experiment template exp

When doing an experiment step, under certain conditions(see repro), a new rollout which changes config (eg: experiment.templates.[n].metadata.labels) uses old information to set up experiment service. then it tries to make correct service and you end up with this error.

To Reproduce

Using a rollout with a WORKLOAD REF. This does not seem to affect rollouts with the template defined in the rollout.

• deploy v1
• deploy v2 w/experiment step
• abort rollout while in experiment step (not strictly needed - looks like it happens even when superceeding in progress release)
• add a new label to experiment.templates.[n].metadata.labels
• deploy v3 w/experiment step

time="2024-03-17T15:49:57Z" level=error msg="Notifications failed to send for eventReason RolloutAborted with error: [trigger 'on-rollout-aborted' is not configured]" event_reason=RolloutAborted namespace=default rollout=scratch-meeech-old-timey-service
time="2024-03-17T15:49:57Z" level=warning msg="Rollout aborted update to revision 3: Failed to create Service experiment for template 'exp': service experiment already exists and is not owned by experiment template exp" event_reason=RolloutAborted namespace=default rollout=scratch-meeech-old-timey-service

Expected behaviour

no error

Observed behaviour

The v3 attempt with the new label makes 2 replicasets:

the deploy-v2 - which DOESN'T have the new label
the service gets created, and its selectors match those pods
then the pod from v2 gets torn down
then the new rs is created - v3
then it tries to make new service, which is in conflict with the old one

Screenshots

rollouts-experiment-bug.mp4

Version

1.6.6

Logs

from 3rd release that errors

time="2024-03-17T15:49:55Z" level=error msg="Error: updating rollout revision" error="Operation cannot be fulfilled on rollouts.argoproj.io \"scratch-meeech-old-timey-service\": the object has been modified; please apply your changes to the latest version and try again" namespace=default rollout=scratch-meeech-old-timey-service
time="2024-03-17T15:49:55Z" level=error msg="roCtx.reconcile err failed to getAllReplicaSetsAndSyncRevision in rolloutCanary create true: Operation cannot be fulfilled on rollouts.argoproj.io \"scratch-meeech-old-timey-service\": the object has been modified; please apply your changes to the latest version and try again" generation=3 namespace=default resourceVersion=689668 rollout=scratch-meeech-old-timey-service
time="2024-03-17T15:49:55Z" level=error msg="rollout syncHandler error: failed to getAllReplicaSetsAndSyncRevision in rolloutCanary create true: Operation cannot be fulfilled on rollouts.argoproj.io \"scratch-meeech-old-timey-service\": the object has been modified; please apply your changes to the latest version and try again" namespace=default rollout=scratch-meeech-old-timey-service
time="2024-03-17T15:49:55Z" level=error msg="failed to getAllReplicaSetsAndSyncRevision in rolloutCanary create true: Operation cannot be fulfilled on rollouts.argoproj.io \"scratch-meeech-old-timey-service\": the object has been modified; please apply your changes to the latest version and try again\n" error="<nil>"
time="2024-03-17T15:49:56Z" level=error msg="Failed to run trigger, trigger: on-rollout-updated, destination: {slack davey-jones-locker}, namespace config:  : trigger 'on-rollout-updated' is not configured"
time="2024-03-17T15:49:56Z" level=error msg="Notifications failed to send for eventReason RolloutUpdated with error: [trigger 'on-rollout-updated' is not configured]" event_reason=RolloutUpdated namespace=default rollout=scratch-meeech-old-timey-service
time="2024-03-17T15:49:56Z" level=info msg="unknown field \"spec.templates[0].template.metadata.creationTimestamp\"\n"
time="2024-03-17T15:49:56Z" level=info msg="unknown field \"spec.templates[0].template.metadata.creationTimestamp\"\n"
time="2024-03-17T15:49:56Z" level=info msg="unknown field \"spec.templates[0].template.metadata.creationTimestamp\"\n"
time="2024-03-17T15:49:56Z" level=warning msg="Template 'exp' transitioned from Progressing -> Error: Failed to create Service experiment for template 'exp': service experiment already exists and is not owned by experiment template exp" event_reason=TemplateError experiment=scratch-meeech-old-timey-service-564f597567-3-0-1 namespace=default
time="2024-03-17T15:49:56Z" level=warning msg="Experiment transitioned from Pending -> Error" event_reason=ExperimentError experiment=scratch-meeech-old-timey-service-564f597567-3-0-1 namespace=default
time="2024-03-17T15:49:57Z" level=error msg="Failed to run trigger, trigger: on-rollout-aborted, destination: {slack davey-jones-locker}, namespace config:  : trigger 'on-rollout-aborted' is not configured"
time="2024-03-17T15:49:57Z" level=error msg="Notifications failed to send for eventReason RolloutAborted with error: [trigger 'on-rollout-aborted' is not configured]" event_reason=RolloutAborted namespace=default rollout=scratch-meeech-old-timey-service
time="2024-03-17T15:49:57Z" level=warning msg="Rollout aborted update to revision 3: Failed to create Service experiment for template 'exp': service experiment already exists and is not owned by experiment template exp" event_reason=RolloutAborted namespace=default rollout=scratch-meeech-old-timey-service

Some similarity with issue #2699. but can't be sure its identical since that one is a bit light on details.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[meeech]

workaround: If you use rollout with workload ref, and use experiment step, and must change the experiment.templates.[n].metadata.labels between deployments, then know the first time you change it, it will fail. Run it again, and it should be fine going forward.

[meeech]

Q: re: Expected behaviour: Rev 2 w/ experiment step is running. Deploy rev 3 - I am expecting that to superceded rev2. Get the same error re: service creation. Captured some log:

[INFO[1816] invalidated cache for resource in namespace: argo-rollouts with the name: argo-rollouts-notification-configmap
INFO[1816] invalidated cache for resource in namespace: argo-rollouts with the name: argo-rollouts-notification-secret
INFO[1816] Started syncing Experiment at (2024-03-19 21:29:37.7729 -0400 EDT m=+1816.225634168)  experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1816] Claimed ReplicaSet 'scratch-meeech-old-timey-service-99c96c6cc-2-0-exp' for template 'exp'  experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1816] Claimed Service 'svc-for-experiment' for template 'exp'  experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1816] Reconciling template                          experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default template=exp
INFO[1816] Start processing                              resource=default/scratch-meeech-old-timey-service
INFO[1816] Started syncing rollout                       generation=2 namespace=default resourceVersion=919243 rollout=scratch-meeech-old-timey-service
INFO[1816] delaying service switch from 76669c69fb to 99c96c6cc: ReplicaSet has zero availability  namespace=default rollout=scratch-meeech-old-timey-service service=scratch-meeech-old-timey-service-canary
INFO[1816] No TrafficRouting Reconcilers found           namespace=default rollout=scratch-meeech-old-timey-service
INFO[1816] Reconciling experiment step (stepIndex: 0)    namespace=default rollout=scratch-meeech-old-timey-service
INFO[1816] No status changes. Skipping patch             generation=2 namespace=default resourceVersion=919243 rollout=scratch-meeech-old-timey-service
INFO[1816] Reconciliation completed                      generation=2 namespace=default resourceVersion=919243 rollout=scratch-meeech-old-timey-service time_ms=1.568167
INFO[1816] Processing completed                          resource=default/scratch-meeech-old-timey-service
INFO[1816] syncing service                               namespace=default rollout=scratch-meeech-old-timey-service service=scratch-meeech-old-timey-service-canary
INFO[1816] syncing service                               namespace=default service=svc-for-experiment
INFO[1816] cleaned service                               namespace=default service=svc-for-experiment
INFO[1816] Started syncing rollout                       generation=2 namespace=default resourceVersion=919243 rollout=scratch-meeech-old-timey-service
INFO[1816] delaying service switch from 76669c69fb to 99c96c6cc: ReplicaSet has zero availability  namespace=default rollout=scratch-meeech-old-timey-service service=scratch-meeech-old-timey-service-canary
INFO[1816] No TrafficRouting Reconcilers found           namespace=default rollout=scratch-meeech-old-timey-service
INFO[1816] Reconciling experiment step (stepIndex: 0)    namespace=default rollout=scratch-meeech-old-timey-service
INFO[1816] No status changes. Skipping patch             generation=2 namespace=default resourceVersion=919243 rollout=scratch-meeech-old-timey-service
INFO[1816] Reconciliation completed                      generation=2 namespace=default resourceVersion=919243 rollout=scratch-meeech-old-timey-service time_ms=1.131958
INFO[1816] No status changes. Skipping patch             experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1816] Reconciliation completed                      experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default time_ms=29.826542
INFO[1818] Start processing                              resource=default/scratch-meeech-old-timey-service
INFO[1818] Processing completed                          resource=default/scratch-meeech-old-timey-service
INFO[1818] Started syncing rollout                       generation=2 namespace=default resourceVersion=919311 rollout=scratch-meeech-old-timey-service
INFO[1818] Pod template change detected (new: 67b6786dfb, old: 99c96c6cc)  namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Assuming 67b6786dfb for new replicaset pod hash  namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Pod template change detected (new: 67b6786dfb, old: 99c96c6cc)  namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Patched: {"status":{"canary":{"currentExperiment":null},"currentPodHash":"67b6786dfb","workloadObservedGeneration":"3"}}  generation=2 namespace=default resourceVersion=919311 rollout=scratch-meeech-old-timey-service
INFO[1818] persisted to informer                         generation=2 namespace=default resourceVersion=919315 rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciliation completed                      generation=2 namespace=default resourceVersion=919311 rollout=scratch-meeech-old-timey-service time_ms=8.609416999999999
INFO[1818] Start processing                              resource=default/scratch-meeech-old-timey-service
INFO[1818] Processing completed                          resource=default/scratch-meeech-old-timey-service
INFO[1818] Started syncing rollout                       generation=2 namespace=default resourceVersion=919315 rollout=scratch-meeech-old-timey-service
INFO[1818] Updating replica set 'scratch-meeech-old-timey-service-67b6786dfb' revision from 0 to 3  namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Created ReplicaSet scratch-meeech-old-timey-service-67b6786dfb  namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Enqueueing parent of default/scratch-meeech-old-timey-service-67b6786dfb: Rollout default/scratch-meeech-old-timey-service
INFO[1818] Enqueueing parent of default/scratch-meeech-old-timey-service-67b6786dfb: Rollout default/scratch-meeech-old-timey-service
INFO[1818] unknown field "spec.strategy.canary.steps[0].experiment.analysisRunMetadata"
INFO[1818] Rollout updated to revision 3                 event_reason=RolloutUpdated namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Created ReplicaSet scratch-meeech-old-timey-service-67b6786dfb (revision 3)  event_reason=NewReplicaSetCreated namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"default", Name:"scratch-meeech-old-timey-service", UID:"a13bf665-ee3b-4d08-a9f4-eb30d7fb7137", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"919318", FieldPath:""}): type: 'Normal' reason: 'RolloutUpdated' Rollout updated to revision 3
INFO[1818] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"default", Name:"scratch-meeech-old-timey-service", UID:"a13bf665-ee3b-4d08-a9f4-eb30d7fb7137", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"919318", FieldPath:""}): type: 'Normal' reason: 'NewReplicaSetCreated' Created ReplicaSet scratch-meeech-old-timey-service-67b6786dfb (revision 3)
INFO[1818] Start processing                              resource=default/scratch-meeech-old-timey-service
INFO[1818] Processing completed                          resource=default/scratch-meeech-old-timey-service
INFO[1818] unknown field "spec.strategy.canary.steps[0].experiment.analysisRunMetadata"
INFO[1818] Set rollout condition: &RolloutCondition{Type:Progressing,Status:True,LastUpdateTime:2024-03-19 21:29:40.432367 -0400 EDT m=+1818.885074168,LastTransitionTime:2024-03-19 21:29:40.432367 -0400 EDT m=+1818.885074210,Reason:NewReplicaSetCreated,Message:Created new replica set "scratch-meeech-old-timey-service-67b6786dfb",}  namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] synced ephemeral metadata nil to ReplicaSet scratch-meeech-old-timey-service-99c96c6cc  namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] delaying service switch from 76669c69fb to 67b6786dfb: ReplicaSet has zero availability  namespace=default rollout=scratch-meeech-old-timey-service service=scratch-meeech-old-timey-service-canary
INFO[1818] No TrafficRouting Reconcilers found           namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciling experiment step (stepIndex: 0)    namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Enqueueing parent of default/scratch-meeech-old-timey-service-99c96c6cc: Rollout default/scratch-meeech-old-timey-service
INFO[1818] unknown field "spec.analysisRunMetadata"
INFO[1818] unknown field "spec.templates[0].template.metadata.creationTimestamp"
INFO[1818] Enqueueing parent of default/scratch-meeech-old-timey-service-67b6786dfb-3-0: Rollout default/scratch-meeech-old-timey-service
INFO[1818] Started syncing Experiment at (2024-03-19 21:29:40.451662 -0400 EDT m=+1818.904368751)  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Created Experiment 'scratch-meeech-old-timey-service-67b6786dfb-3-0'  event_reason=ExperimentCreated namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Canceling other running experiment 'scratch-meeech-old-timey-service-99c96c6cc-2-0' owned by rollout  namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"default", Name:"scratch-meeech-old-timey-service", UID:"a13bf665-ee3b-4d08-a9f4-eb30d7fb7137", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"919318", FieldPath:""}): type: 'Normal' reason: 'ExperimentCreated' Created Experiment 'scratch-meeech-old-timey-service-67b6786dfb-3-0'
INFO[1818] Reconciling template                          experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1818] Created ReplicaSet scratch-meeech-old-timey-service-67b6786dfb-3-0-exp  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Event(v1.ObjectReference{Kind:"Experiment", Namespace:"default", Name:"scratch-meeech-old-timey-service-67b6786dfb-3-0", UID:"2422778d-5ed0-44d4-9adf-baa16e3b0f4f", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"919323", FieldPath:""}): type: 'Normal' reason: 'TemplateProgressing' Template 'exp' transitioned from  -> Progressing
INFO[1818] Template 'exp' transitioned from  -> Progressing  event_reason=TemplateProgressing experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Enqueueing parent of default/scratch-meeech-old-timey-service-67b6786dfb-3-0-exp: Experiment default/scratch-meeech-old-timey-service-67b6786dfb-3-0
INFO[1818] Event(v1.ObjectReference{Kind:"Experiment", Namespace:"default", Name:"scratch-meeech-old-timey-service-67b6786dfb-3-0", UID:"2422778d-5ed0-44d4-9adf-baa16e3b0f4f", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"919323", FieldPath:""}): type: 'Normal' reason: 'ExperimentPending' Experiment transitioned from  -> Pending
INFO[1818] Experiment transitioned from  -> Pending      event_reason=ExperimentPending experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Enqueueing Experiment in 9m59.996127292s seconds  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Enqueueing parent of default/scratch-meeech-old-timey-service-99c96c6cc-2-0: Rollout default/scratch-meeech-old-timey-service
INFO[1818] Started syncing Experiment at (2024-03-19 21:29:40.457369 -0400 EDT m=+1818.910075376)  experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1818] Claimed ReplicaSet 'scratch-meeech-old-timey-service-99c96c6cc-2-0-exp' for template 'exp'  experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1818] Claimed Service 'svc-for-experiment' for template 'exp'  experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1818] Reconciling template                          experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default template=exp
INFO[1818] Patch status successfully                     experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Reconciliation completed                      experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default time_ms=9.442874999999999
INFO[1818] Started syncing Experiment at (2024-03-19 21:29:40.461125 -0400 EDT m=+1818.913831501)  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Enqueueing parent of default/scratch-meeech-old-timey-service-67b6786dfb-3-0: Rollout default/scratch-meeech-old-timey-service
INFO[1818] Claimed ReplicaSet 'scratch-meeech-old-timey-service-67b6786dfb-3-0-exp' for template 'exp'  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Reconciling template                          experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1818] Set 'scale-down-deadline' annotation on 'scratch-meeech-old-timey-service-99c96c6cc-2-0-exp' to 2024-03-20T01:30:10Z (30ns)  experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1818] Start processing                              resource=default/scratch-meeech-old-timey-service
INFO[1818] Processing completed                          resource=default/scratch-meeech-old-timey-service
INFO[1818] Patched: {"status":{"canary":{"currentExperiment":"scratch-meeech-old-timey-service-67b6786dfb-3-0"}}}  generation=2 namespace=default resourceVersion=919318 rollout=scratch-meeech-old-timey-service
INFO[1818] persisted to informer                         generation=2 namespace=default resourceVersion=919331 rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciliation completed                      generation=2 namespace=default resourceVersion=919315 rollout=scratch-meeech-old-timey-service time_ms=46.558542
INFO[1818] Started syncing rollout                       generation=2 namespace=default resourceVersion=919331 rollout=scratch-meeech-old-timey-service
INFO[1818] delaying service switch from 76669c69fb to 67b6786dfb: ReplicaSet has zero availability  namespace=default rollout=scratch-meeech-old-timey-service service=scratch-meeech-old-timey-service-canary
INFO[1818] No TrafficRouting Reconcilers found           namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciling experiment step (stepIndex: 0)    namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] RS 'scratch-meeech-old-timey-service-99c96c6cc-2-0-exp' has not reached the scaleDownTime  experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1818] No status changes. Skipping patch             generation=2 namespace=default resourceVersion=919331 rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciliation completed                      generation=2 namespace=default resourceVersion=919331 rollout=scratch-meeech-old-timey-service time_ms=0.9355
INFO[1818] Template 'exp' transitioned from Running -> Successful  event_reason=TemplateSuccessful experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1818] Experiment transitioned from Running -> Successful  event_reason=ExperimentSuccessful experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1818] Event(v1.ObjectReference{Kind:"Experiment", Namespace:"default", Name:"scratch-meeech-old-timey-service-99c96c6cc-2-0", UID:"8298cd45-9c19-4fa0-9610-d5dc91247856", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"919326", FieldPath:""}): type: 'Normal' reason: 'TemplateSuccessful' Template 'exp' transitioned from Running -> Successful
INFO[1818] Event(v1.ObjectReference{Kind:"Experiment", Namespace:"default", Name:"scratch-meeech-old-timey-service-99c96c6cc-2-0", UID:"8298cd45-9c19-4fa0-9610-d5dc91247856", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"919326", FieldPath:""}): type: 'Normal' reason: 'ExperimentSuccessful' Experiment transitioned from Running -> Successful
INFO[1818] Enqueueing parent of default/scratch-meeech-old-timey-service-99c96c6cc-2-0: Rollout default/scratch-meeech-old-timey-service
INFO[1818] Patch status successfully                     experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1818] Reconciliation completed                      experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default time_ms=16.069917
INFO[1818] Started syncing rollout                       generation=2 namespace=default resourceVersion=919331 rollout=scratch-meeech-old-timey-service
INFO[1818] Started syncing Experiment at (2024-03-19 21:29:40.473678 -0400 EDT m=+1818.926384460)  experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1818] Claimed ReplicaSet 'scratch-meeech-old-timey-service-99c96c6cc-2-0-exp' for template 'exp'  experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1818] Claimed Service 'svc-for-experiment' for template 'exp'  experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1818] Reconciling template                          experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default template=exp
INFO[1818] RS 'scratch-meeech-old-timey-service-99c96c6cc-2-0-exp' has not reached the scaleDownTime  experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1818] No status changes. Skipping patch             experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default
INFO[1818] Reconciliation completed                      experiment=scratch-meeech-old-timey-service-99c96c6cc-2-0 namespace=default time_ms=0.193583
INFO[1818] delaying service switch from 76669c69fb to 67b6786dfb: ReplicaSet has zero availability  namespace=default rollout=scratch-meeech-old-timey-service service=scratch-meeech-old-timey-service-canary
INFO[1818] No TrafficRouting Reconcilers found           namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciling experiment step (stepIndex: 0)    namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] No status changes. Skipping patch             generation=2 namespace=default resourceVersion=919331 rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciliation completed                      generation=2 namespace=default resourceVersion=919331 rollout=scratch-meeech-old-timey-service time_ms=1.1241249999999998
INFO[1818] Started syncing rollout                       generation=2 namespace=default resourceVersion=919331 rollout=scratch-meeech-old-timey-service
INFO[1818] Scaled up ReplicaSet scratch-meeech-old-timey-service-67b6786dfb-3-0-exp from 0 to 1  event_reason=ScalingReplicaSet experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Scaled up ReplicaSet scratch-meeech-old-timey-service-67b6786dfb-3-0-exp from 0 to 1  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
WARN[1818] Template 'exp' transitioned from  -> Error: Failed to create Service svc-for-experiment for template 'exp': service svc-for-experiment already exists and is not owned by experiment template exp  event_reason=TemplateError experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Experiment transitioned from  -> Pending      event_reason=ExperimentPending experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] delaying service switch from 76669c69fb to 67b6786dfb: ReplicaSet has zero availability  namespace=default rollout=scratch-meeech-old-timey-service service=scratch-meeech-old-timey-service-canary
INFO[1818] No TrafficRouting Reconcilers found           namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciling experiment step (stepIndex: 0)    namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Enqueueing parent of default/scratch-meeech-old-timey-service-67b6786dfb-3-0-exp: Experiment default/scratch-meeech-old-timey-service-67b6786dfb-3-0
INFO[1818] Event(v1.ObjectReference{Kind:"Experiment", Namespace:"default", Name:"scratch-meeech-old-timey-service-67b6786dfb-3-0", UID:"2422778d-5ed0-44d4-9adf-baa16e3b0f4f", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"919323", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up ReplicaSet scratch-meeech-old-timey-service-67b6786dfb-3-0-exp from 0 to 1
INFO[1818] Event(v1.ObjectReference{Kind:"Experiment", Namespace:"default", Name:"scratch-meeech-old-timey-service-67b6786dfb-3-0", UID:"2422778d-5ed0-44d4-9adf-baa16e3b0f4f", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"919323", FieldPath:""}): type: 'Warning' reason: 'TemplateError' Template 'exp' transitioned from  -> Error: Failed to create Service svc-for-experiment for template 'exp': service svc-for-experiment already exists and is not owned by experiment template exp
INFO[1818] Event(v1.ObjectReference{Kind:"Experiment", Namespace:"default", Name:"scratch-meeech-old-timey-service-67b6786dfb-3-0", UID:"2422778d-5ed0-44d4-9adf-baa16e3b0f4f", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"919323", FieldPath:""}): type: 'Normal' reason: 'ExperimentPending' Experiment transitioned from  -> Pending
INFO[1818] No status changes. Skipping patch             generation=2 namespace=default resourceVersion=919331 rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciliation completed                      generation=2 namespace=default resourceVersion=919331 rollout=scratch-meeech-old-timey-service time_ms=1.3247499999999999
INFO[1818] Patch status successfully                     experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Reconciliation completed                      experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default time_ms=39.683167
INFO[1818] Enqueueing parent of default/scratch-meeech-old-timey-service-67b6786dfb-3-0: Rollout default/scratch-meeech-old-timey-service
INFO[1818] Started syncing Experiment at (2024-03-19 21:29:40.500825 -0400 EDT m=+1818.953530626)  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Claimed ReplicaSet 'scratch-meeech-old-timey-service-67b6786dfb-3-0-exp' for template 'exp'  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Reconciling template                          experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1818] Started syncing rollout                       generation=2 namespace=default resourceVersion=919331 rollout=scratch-meeech-old-timey-service
INFO[1818] delaying service switch from 76669c69fb to 67b6786dfb: ReplicaSet has zero availability  namespace=default rollout=scratch-meeech-old-timey-service service=scratch-meeech-old-timey-service-canary
INFO[1818] No TrafficRouting Reconcilers found           namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciling experiment step (stepIndex: 0)    namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] No status changes. Skipping patch             generation=2 namespace=default resourceVersion=919331 rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciliation completed                      generation=2 namespace=default resourceVersion=919331 rollout=scratch-meeech-old-timey-service time_ms=1.209333
INFO[1818] Set 'scale-down-deadline' annotation on 'scratch-meeech-old-timey-service-67b6786dfb-3-0-exp' to 2024-03-20T01:30:10Z (30ns)  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] RS 'scratch-meeech-old-timey-service-67b6786dfb-3-0-exp' has not reached the scaleDownTime  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
WARN[1818] Experiment transitioned from Pending -> Error  event_reason=ExperimentError experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Event(v1.ObjectReference{Kind:"Experiment", Namespace:"default", Name:"scratch-meeech-old-timey-service-67b6786dfb-3-0", UID:"2422778d-5ed0-44d4-9adf-baa16e3b0f4f", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"919341", FieldPath:""}): type: 'Warning' reason: 'ExperimentError' Experiment transitioned from Pending -> Error
INFO[1818] Start processing                              resource=default/scratch-meeech-old-timey-service
INFO[1818] Processing completed                          resource=default/scratch-meeech-old-timey-service
INFO[1818] Started syncing rollout                       generation=2 namespace=default resourceVersion=919350 rollout=scratch-meeech-old-timey-service
INFO[1818] delaying service switch from 76669c69fb to 67b6786dfb: ReplicaSet has zero availability  namespace=default rollout=scratch-meeech-old-timey-service service=scratch-meeech-old-timey-service-canary
INFO[1818] No TrafficRouting Reconcilers found           namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciling experiment step (stepIndex: 0)    namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] No status changes. Skipping patch             generation=2 namespace=default resourceVersion=919350 rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciliation completed                      generation=2 namespace=default resourceVersion=919350 rollout=scratch-meeech-old-timey-service time_ms=1.077541
INFO[1818] Enqueueing parent of default/scratch-meeech-old-timey-service-67b6786dfb-3-0-exp: Experiment default/scratch-meeech-old-timey-service-67b6786dfb-3-0
INFO[1818] Patch status successfully                     experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Reconciliation completed                      experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default time_ms=21.406000000000002
INFO[1818] Started syncing Experiment at (2024-03-19 21:29:40.522249 -0400 EDT m=+1818.974954835)  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Enqueueing parent of default/scratch-meeech-old-timey-service-67b6786dfb-3-0: Rollout default/scratch-meeech-old-timey-service
INFO[1818] Claimed ReplicaSet 'scratch-meeech-old-timey-service-67b6786dfb-3-0-exp' for template 'exp'  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Reconciling template                          experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1818] RS 'scratch-meeech-old-timey-service-67b6786dfb-3-0-exp' has not reached the scaleDownTime  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Template progressed                           experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1818] Prev status: Current: 0, Updated: 0, Ready: 0, Available: 0  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1818] New status: Current: 1, Updated: 1, Ready: 0, Available: 0  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1818] Started syncing rollout                       generation=2 namespace=default resourceVersion=919350 rollout=scratch-meeech-old-timey-service
INFO[1818] Event(v1.ObjectReference{Kind:"Experiment", Namespace:"default", Name:"scratch-meeech-old-timey-service-67b6786dfb-3-0", UID:"2422778d-5ed0-44d4-9adf-baa16e3b0f4f", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"919341", FieldPath:""}): type: 'Warning' reason: 'ExperimentError' Experiment transitioned from Pending -> Error
WARN[1818] Experiment transitioned from Pending -> Error  event_reason=ExperimentError experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] delaying service switch from 76669c69fb to 67b6786dfb: ReplicaSet has zero availability  namespace=default rollout=scratch-meeech-old-timey-service service=scratch-meeech-old-timey-service-canary
INFO[1818] No TrafficRouting Reconcilers found           namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciling experiment step (stepIndex: 0)    namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Skipping analysis: isAborted: true, promoteFull: false, rollbackToScaleDownDelay: false, initialDeploy: false  namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Event(v1.ObjectReference{Kind:"Rollout", Namespace:"default", Name:"scratch-meeech-old-timey-service", UID:"a13bf665-ee3b-4d08-a9f4-eb30d7fb7137", APIVersion:"argoproj.io/v1alpha1", ResourceVersion:"919350", FieldPath:""}): type: 'Warning' reason: 'RolloutAborted' Rollout aborted update to revision 3: Failed to create Service svc-for-experiment for template 'exp': service svc-for-experiment already exists and is not owned by experiment template exp
WARN[1818] Rollout aborted update to revision 3: Failed to create Service svc-for-experiment for template 'exp': service svc-for-experiment already exists and is not owned by experiment template exp  event_reason=RolloutAborted namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Patch status successfully                     experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Reconciliation completed                      experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default time_ms=9.087541
INFO[1818] Started syncing Experiment at (2024-03-19 21:29:40.531372 -0400 EDT m=+1818.984077751)  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Claimed ReplicaSet 'scratch-meeech-old-timey-service-67b6786dfb-3-0-exp' for template 'exp'  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Reconciling template                          experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1818] RS 'scratch-meeech-old-timey-service-67b6786dfb-3-0-exp' has not reached the scaleDownTime  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Template progressed                           experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1818] Prev status: Current: 0, Updated: 0, Ready: 0, Available: 0  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1818] New status: Current: 1, Updated: 1, Ready: 0, Available: 0  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1818] Enqueueing parent of default/scratch-meeech-old-timey-service-67b6786dfb-3-0: Rollout default/scratch-meeech-old-timey-service
INFO[1818] Patched: {"status":{"abort":true,"abortedAt":"2024-03-20T01:29:40Z","canary":{"currentExperiment":null},"conditions":[{"lastTransitionTime":"2024-03-20T01:25:51Z","lastUpdateTime":"2024-03-20T01:25:51Z","message":"Rollout has minimum availability","reason":"AvailableReason","status":"True","type":"Available"},{"lastTransitionTime":"2024-03-20T01:29:17Z","lastUpdateTime":"2024-03-20T01:29:17Z","message":"Rollout is not healthy","reason":"RolloutHealthy","status":"False","type":"Healthy"},{"lastTransitionTime":"2024-03-20T01:29:17Z","lastUpdateTime":"2024-03-20T01:29:17Z","message":"RolloutCompleted","reason":"RolloutCompleted","status":"False","type":"Completed"},{"lastTransitionTime":"2024-03-20T01:29:40Z","lastUpdateTime":"2024-03-20T01:29:40Z","message":"Rollout aborted update to revision 3: Failed to create Service svc-for-experiment for template 'exp': service svc-for-experiment already exists and is not owned by experiment template exp","reason":"RolloutAborted","status":"False","type":"Progressing"}],"message":"RolloutAborted: Rollout aborted update to revision 3: Failed to create Service svc-for-experiment for template 'exp': service svc-for-experiment already exists and is not owned by experiment template exp","phase":"Degraded"}}  generation=2 namespace=default resourceVersion=919350 rollout=scratch-meeech-old-timey-service
INFO[1818] persisted to informer                         generation=2 namespace=default resourceVersion=919357 rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciliation completed                      generation=2 namespace=default resourceVersion=919350 rollout=scratch-meeech-old-timey-service time_ms=12.322208
INFO[1818] Start processing                              resource=default/scratch-meeech-old-timey-service
INFO[1818] Processing completed                          resource=default/scratch-meeech-old-timey-service
INFO[1818] Started syncing rollout                       generation=2 namespace=default resourceVersion=919357 rollout=scratch-meeech-old-timey-service
INFO[1818] No TrafficRouting Reconcilers found           namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Skipping analysis: isAborted: true, promoteFull: false, rollbackToScaleDownDelay: false, initialDeploy: false  namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] No status changes. Skipping patch             generation=2 namespace=default resourceVersion=919357 rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciliation completed                      generation=2 namespace=default resourceVersion=919357 rollout=scratch-meeech-old-timey-service time_ms=1.8453339999999998
INFO[1818] Started syncing rollout                       generation=2 namespace=default resourceVersion=919357 rollout=scratch-meeech-old-timey-service
INFO[1818] Patch status successfully                     experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Reconciliation completed                      experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default time_ms=5.507167
INFO[1818] Started syncing Experiment at (2024-03-19 21:29:40.536897 -0400 EDT m=+1818.989602751)  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Claimed ReplicaSet 'scratch-meeech-old-timey-service-67b6786dfb-3-0-exp' for template 'exp'  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Reconciling template                          experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1818] RS 'scratch-meeech-old-timey-service-67b6786dfb-3-0-exp' has not reached the scaleDownTime  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] No status changes. Skipping patch             experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1818] Reconciliation completed                      experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default time_ms=0.15612499999999999
INFO[1818] No TrafficRouting Reconcilers found           namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] Skipping analysis: isAborted: true, promoteFull: false, rollbackToScaleDownDelay: false, initialDeploy: false  namespace=default rollout=scratch-meeech-old-timey-service
INFO[1818] No status changes. Skipping patch             generation=2 namespace=default resourceVersion=919357 rollout=scratch-meeech-old-timey-service
INFO[1818] Reconciliation completed                      generation=2 namespace=default resourceVersion=919357 rollout=scratch-meeech-old-timey-service time_ms=1.012583
INFO[1819] Enqueueing parent of default/scratch-meeech-old-timey-service-67b6786dfb-3-0-exp: Experiment default/scratch-meeech-old-timey-service-67b6786dfb-3-0
INFO[1819] Started syncing Experiment at (2024-03-19 21:29:41.378614 -0400 EDT m=+1819.831310793)  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1819] Claimed ReplicaSet 'scratch-meeech-old-timey-service-67b6786dfb-3-0-exp' for template 'exp'  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1819] Reconciling template                          experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1819] RS 'scratch-meeech-old-timey-service-67b6786dfb-3-0-exp' has not reached the scaleDownTime  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1819] Template progressed                           experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1819] Prev status: Current: 1, Updated: 1, Ready: 0, Available: 0  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1819] New status: Current: 1, Updated: 1, Ready: 1, Available: 1  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1819] Enqueueing parent of default/scratch-meeech-old-timey-service-67b6786dfb-3-0: Rollout default/scratch-meeech-old-timey-service
INFO[1819] Patch status successfully                     experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1819] Reconciliation completed                      experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default time_ms=7.099375
INFO[1819] Started syncing Experiment at (2024-03-19 21:29:41.385747 -0400 EDT m=+1819.838444001)  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1819] Claimed ReplicaSet 'scratch-meeech-old-timey-service-67b6786dfb-3-0-exp' for template 'exp'  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1819] Reconciling template                          experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default template=exp
INFO[1819] RS 'scratch-meeech-old-timey-service-67b6786dfb-3-0-exp' has not reached the scaleDownTime  experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1819] Started syncing rollout                       generation=2 namespace=default resourceVersion=919357 rollout=scratch-meeech-old-timey-service
INFO[1819] No status changes. Skipping patch             experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default
INFO[1819] Reconciliation completed                      experiment=scratch-meeech-old-timey-service-67b6786dfb-3-0 namespace=default time_ms=1.158917
INFO[1819] No TrafficRouting Reconcilers found           namespace=default rollout=scratch-meeech-old-timey-service
INFO[1819] Skipping analysis: isAborted: true, promoteFull: false, rollbackToScaleDownDelay: false, initialDeploy: false  namespace=default rollout=scratch-meeech-old-timey-service
INFO[1819] No status changes. Skipping patch             generation=2 namespace=default resourceVersion=919357 rollout=scratch-meeech-old-timey-service
INFO[1819] Reconciliation completed                      generation=2 namespace=default resourceVersion=919357 rollout=scratch-meeech-old-timey-service time_ms=3.154375