How to update Calico without outage

[mcantinqc]

Hi everyone,

We recently upgraded Calico from 3.28 to 3.29 on an EKS cluster (running in iptables mode), and we encountered a significant issue: the upgrade completed successfully, but we still experienced downtime. The only way to restore connectivity was to recreate the node groups.

From our understanding, Calico updates itself independently of the node groups, meaning the existing nodes might retain outdated iptables rules that are incompatible with the new version. This forces a node recreation, which leads to unavoidable downtime.

Has anyone found a way to:

• Update Calico without requiring node group recreation? It seems impossible to orchestrate a Calico update and node recreation at exactly the same time—there will always be a window of downtime.
• Perform a zero-downtime rolling update for Calico on EKS? Running two separate EKS clusters and switching traffic is an option, but it introduces significant overhead.

Our installation is very basic:

kind: Installation
apiVersion: operator.tigera.io/v1
metadata:
  name: default
spec:
  kubernetesProvider: EKS
  cni:
    type: Calico
  calicoNetwork:
    bgp: Disabled

Any insights or best practices would be greatly appreciated!

[caseydavenport]

@mcantinqc that sounds like unexpected behavior to me - Calico is meant to perform a rolling update without any outages. Could you explain more specifically what happened when you upgraded, and what the symptoms of the outage were?

Do you have any log files from the occurrence?

meaning the existing nodes might retain outdated iptables rules that are incompatible with the new version

This shouldn't happen - Calico is capable of flushing and reinstalling its new ruleset even after upgrade.

[mcantinqc]

We have minimal configuration, with only two files—one from the original disucssion and the following Felix configuration:

apiVersion: projectcalico.org/v3
kind: FelixConfiguration
metadata:
  name: default
spec:
  awsSrcDstCheck: Disable
  ipv6Support: false

Pod-to-pod communication has stopped working, and NodePort is also non-functional. Restarting all nodes resolves the issue, and I was able to reproduce it this morning.

The logs don't show anything significant, except for a single warning from one node:

 2025-02-19 10:08:11.707 | 2025-02-19 15:08:11.707 [WARNING][9431] felix/table.go 680: Chain had unexpected inserts, marking for resync actualRuleIDs=[]string{"", "0i8pjzKKPyA34aQD"} chainName="POSTROUTING" ipVersion=0x4 table="nat" |  
-- | -- | --
  |   | 2025-02-19 10:08:06.703 | 2025-02-19 15:08:06.702 [INFO][1] typha/sync_server.go 1081: Status update to send. client=10.7.39.180:58190 connID=0x4 newStatus=in-sync thread="kv-sender" type="node-status" |  
  |   | 2025-02-19 10:08:06.702 | 2025-02-19 15:08:06.702 [INFO][1] typha/sync_server.go 1075: Calculated end of client's grace period. graceEnd=2025-02-19 15:13:06.702233612 +0000 UTC m=+345.428587709 |  
  |   | 2025-02-19 10:08:06.702 | 2025-02-19 15:08:06.702 [INFO][1] typha/sync_server.go 806: Waiting for messages from client client=10.7.39.180:58190 connID=0x4 type="node-status" |  
  |   | 2025-02-19 10:08:06.702 | 2025-02-19 15:08:06.702 [INFO][1] typha/sync_server.go 774: Sent compressed binary snapshot and received ACK from client. client=10.7.39.180:58190 connID=0x4 type="node-status" |  
  |   | 2025-02-19 10:08:06.702 | 2025-02-19 15:08:06.701 [INFO][1] typha/sync_server.go 989: Received ACK message from client. client=10.7.39.180:58190 connID=0x4 msg=syncproto.MsgACK{} type="node-status" |  
  |   | 2025-02-19 10:08:06.701 | 2025-02-19 15:08:06.701 [INFO][1] typha/sync_server.go 989: Received ACK message from client. client=10.7.39.180:58190 connID=0x4 msg=syncproto.MsgACK{} type="node-status" |  
  |   | 2025-02-19 10:08:06.700 | 2025-02-19 15:08:06.700 [INFO][1] typha/sync_server.go 748: Restarting encoding. client=10.7.39.180:58190 connID=0x4 reasons=[]string{"enable compression: snappy", "send binary snapshot"} type="node-status" |  
  |   | 2025-02-19 10:08:06.700 | 2025-02-19 15:08:06.700 [INFO][1] typha/sync_server.go 908: Received Hello message from client. client=10.7.39.180:58190 connID=0x4 msg=syncproto.MsgClientHello{Hostname:"ip-10-7-39-180.ca-central-1.compute.internal", Info:"node-status v3.29.2", Version:"v3.29.2", SyncerType:"node-status", SupportsDecoderRestart:true, SupportedCompressionAlgorithms:[]syncproto.CompressionAlgorithm{"snappy"}, ClientConnID:0x1} |  
  |   | 2025-02-19 10:08:06.692 | 2025-02-19 15:08:06.692 [INFO][1] typha/sync_server.go 721: Per-connection goroutine started client=10.7.39.180:58190 connID=0x4 |  
  |   | 2025-02-19 10:08:06.692 | 2025-02-19 15:08:06.692 [INFO][1] typha/sync_server.go 433: New connection connID=0x4 port=5473 |  
  |   | 2025-02-19 10:08:06.692 | 2025-02-19 15:08:06.692 [INFO][1] typha/sync_server.go 424: Accepted from 10.7.39.180:58190 port=5473 |  
  |   | 2025-02-19 10:07:53.266 | 2025-02-19 15:07:53.266 [INFO][1] typha/sync_server.go 532: New target number of connections currentNum=3 newMax=400 oldMax=10000 thread="numConnsGov" |  
  |   | 2025-02-19 10:07:53.266 | 2025-02-19 15:07:53.266 [INFO][1] typha/rebalance.go 77: Calculated new connection limit. newLimit=400 numNodes=5 numSyncerTypes=4 numTyphas=3 reason="configured lower limit" thread="k8s-poll" |  
  |   | 2025-02-19 10:07:53.259 | 2025-02-19 15:07:53.259 [WARNING][1] typha/winutils.go 150: Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. |  
  |   | 2025-02-19 10:07:51.712 | 2025-02-19 15:07:51.711 [WARNING][9431] felix/table.go 680: Chain had unexpected inserts, marking for resync actualRuleIDs=[]string{"", "0i8pjzKKPyA34aQD"} chainName="POSTROUTING" ipVersion=0x4 table="nat" |  
  |   | 2025-02-19 10:07:44.629 | I0219 15:07:44.629258 1 main.go:108] "Received NotifyRegistrationStatus call" status="&RegistrationStatus{PluginRegistered:true,Error:,}" |  
  |   | 2025-02-19 10:07:44.592 | I0219 15:07:44.592398 1 main.go:96] "Received GetInfo call" request="&InfoRequest{}" |  
  |   | 2025-02-19 10:07:43.841 | I0219 15:07:43.841438 1 node_register.go:96] "Skipping HTTP server" |  
  |   | 2025-02-19 10:07:43.841 | I0219 15:07:43.841313 1 node_register.go:66] "Registration Server started" socketPath="/registration/csi.tigera.io-reg.sock" |  
  |   | 2025-02-19 10:07:43.841 | I0219 15:07:43.841109 1 node_register.go:56] "Starting Registration Server" socketPath="/registration/csi.tigera.io-reg.sock" |  
  |   | 2025-02-19 10:07:43.841 | I0219 15:07:43.841091 1 main.go:189] "CSI driver name" csiDriverName="csi.tigera.io" |  
  |   | 2025-02-19 10:07:43.841 | I0219 15:07:43.841076 1 connection.go:270] "GRPC response" response="{"name":"csi.tigera.io","vendor_version":"0.1"}" err=null |  
  |   | 2025-02-19 10:07:43.839 | I0219 15:07:43.839255 1 connection.go:264] "GRPC call" method="/csi.v1.Identity/GetPluginInfo" request="{}" |  
  |   | 2025-02-19 10:07:43.839 | I0219 15:07:43.839241 1 main.go:180] "Calling CSI driver to discover driver name" |  
  |   | 2025-02-19 10:07:43.838 | I0219 15:07:43.838599 1 connection.go:234] "Connecting" address="unix:///csi/csi.sock" |  
  |   | 2025-02-19 10:07:43.838 | I0219 15:07:43.838578 1 main.go:172] "Attempting to open a gRPC connection" csiAddress="/csi/csi.sock" |  
  |   | 2025-02-19 10:07:43.838 | I0219 15:07:43.838559 1 main.go:151] "Running node-driver-registrar" mode="" |  
  |   | 2025-02-19 10:07:43.838 | I0219 15:07:43.838493 1 main.go:150] "Version" version="unknown"

Other logs appear normal, showing status updates and connection handling without any obvious errors.