Pods stuck in creating: "No valid options provided. Usage:\n": invalid character 'N' looking for beginning of value

[jnorman-atwork]

Attempting to use Calico with Kubernetes. I'm using "Calico the Hard Way" as the cluster is entirely offline. The calico-node pods are up, Typha is up, BGP is looking good, but certain pods (coredns, metrics-server, calico-kube-controllers) are stuck in ContainerCreating with an error that does not google:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "HUGEUUIDHERE": plugin type="calico" failed (add): netplugin failed but error parsing its diagnostic message "No valid options provided. Usage:\n": invalid character 'N' looking for beginning of value

That error message "No valid options provided" is what happens when you try to run calico without any options. Which is fine, but what in the chain of events is calling "calico" with no options, and how do I change / affect / troubleshoot / debug that?

Expected Behavior

Pods should start

Current Behavior

Pods hang in ConatinerCreating

Possible Solution

Steps to Reproduce (for bugs)

1. Install kubernetes 1.28.2 on rhel8.8
2. Run through the steps for "Calico the hard way" using zero modifications
3. check pod status
4. troubleshoot for multiple weeks

Context

Your Environment

• Calico version v3.24.5, cluster version 3.27.0
• Orchestrator version (e.g. kubernetes, mesos, rkt): kubernetes version 1.28.2, containerd.io 1.6.31-3.1
• Operating System and version: redhat 8.8
• Link to your project (optional):

[caseydavenport]

I suspect this is a problem with your CNI config file on disk - could you share it here?

[jnorman-atwork]

I suspect this is a problem with your CNI config file on disk - could you share it here?

Apologies; I forgot to update this when I found the root cause. This all happened due to an improper calico binary. The version of calico binary wasn't calico; something in our chef deployment caused the wrong binary to be placed on the node and named 'calico' but the binary was actually part of calico-node.

Using the correct calico binary for v3.27.0 fixed all the issues.

[jnorman-atwork]

(closing for completness)