You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have a Java application (Java 17, Spring Boot 2.7.18), running on Kubernetes and scaled to 4 pods. It's running embedded Infinispan 13.0.20. We do have network policies in place, so port 7800 TCP is only reachable between the pods of this application.
Configuration is very basic - using jgroups config default-configs/default-jgroups-kubernetes.xml and a headless service for dns.
A week ago or so, we started having problems when rolling out new deployments. New pods didn't not come up, because of some Infinispan problems.
After some investigation, we noticed that in the Infinispan cluster view (according to logs), there always appeared two pods, that don't exist (lets call them foobar-5ccd5468ff-q2krt-35808 and foobar-5554dd5d78-hlkfw-2948).
So we scaled down the whole deployment to 0, waited a minute, scaled it up again. We don't have any persistence (no PVC), Infinispan is supposed to be just in memory. When we scale it down, everything is supposed to be gone.
After scaling up again, the two nodes wanted to join the cluster again: "ISPN100000: Node foobar-5ccd5468ff-q2krt-35808 joined the cluster".
They keep joining and leaving. Cluster view is "poisoned" with these names on non-existing and not reachable node names. I have no idea, where these messages are coming from.
How is this even possible? Even if there would be some bug in infinispan - after scaling down, there are no pods, no memory. These are supposed to be forgotten at that point.
Some bug in the network cards or something like that? But we placed the pods on other racks, even in another data center.
Has anyone experienced something similar? Or has the slightest idea, what's going on here?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
We have a Java application (Java 17, Spring Boot 2.7.18), running on Kubernetes and scaled to 4 pods. It's running embedded Infinispan 13.0.20. We do have network policies in place, so port 7800 TCP is only reachable between the pods of this application.
Configuration is very basic - using jgroups config
default-configs/default-jgroups-kubernetes.xml
and a headless service for dns.A week ago or so, we started having problems when rolling out new deployments. New pods didn't not come up, because of some Infinispan problems.
After some investigation, we noticed that in the Infinispan cluster view (according to logs), there always appeared two pods, that don't exist (lets call them
foobar-5ccd5468ff-q2krt-35808
andfoobar-5554dd5d78-hlkfw-2948
).So we scaled down the whole deployment to 0, waited a minute, scaled it up again. We don't have any persistence (no PVC), Infinispan is supposed to be just in memory. When we scale it down, everything is supposed to be gone.
After scaling up again, the two nodes wanted to join the cluster again:
"ISPN100000: Node foobar-5ccd5468ff-q2krt-35808 joined the cluster"
.They keep joining and leaving. Cluster view is "poisoned" with these names on non-existing and not reachable node names. I have no idea, where these messages are coming from.
How is this even possible? Even if there would be some bug in infinispan - after scaling down, there are no pods, no memory. These are supposed to be forgotten at that point.
Some bug in the network cards or something like that? But we placed the pods on other racks, even in another data center.
Has anyone experienced something similar? Or has the slightest idea, what's going on here?
Beta Was this translation helpful? Give feedback.
All reactions