Service watcher keeps editing managed exit nodes #155
Labels
bug
Something isn't working
help wanted
Extra attention is needed
question
Further information is requested
Comments
|
That shouldn't really happen, feedback loop? I need to investigate this. |
|
After completing the cluster wipe & full redeployment, I can confirm this is still happening. |
|
What is your setup? Are you using multiple |
korewaChino
added
bug
korewaChino
changed the title
|
I am using Envoy Gateway Operator, MetalLB operator, 3 nodes running Talos OS in a fairly standard environment. Gateway manifest: apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: gw
namespace: default
annotations:
metallb.universe.tf/loadBalancerIPs: 10.100.101.15 # I've edited this in, its from a variable
spec:
gatewayClassName: eg
infrastructure:
annotations:
chisel-operator.io/exit-node-name: "digitalocean"GatewayClass manifest: apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
namespace: envoy
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: config
namespace: envoyand EnvoyProxy manifest: apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: config
namespace: envoy
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
replicas: 3I think thats all the relevant information. Adding an ExitNode manifest: apiVersion: chisel-operator.io/v1
kind: ExitNode
metadata:
name: digitalocean
namespace: chisel
annotations:
chisel-operator.io/exit-node-provisioner: "digitalocean"
spec:
host: ""
port: 9090
auth: digitalocean-authis provisioning a DO droplet, and the tunnel is connecting (although I havent actually tested it for routing, something I'll do tomorrow), and kicking off the log spam. I'll try piping the logs through less/more or whatever, see if I can catch the start of the log spam to see what kicks it off. |
|
I don't really know what Envoy is doing, but maybe it's conflicting with Chisel Operator in some weird way. Anyway I may have accidentally fixed this bug in #156, so you might wanna try the docker build in that. |
|
awesome, I'll give it a shot tomorrow. |
|
Cool, with the config I had yesterday (with an If I delete the Creating the relevant auth secret kicks off the operator, and I see an exit node pod and droplet being provisioned. Unfortunately, the log spam is still happening (I've included the entire logs with sensitive data snipped - I think): |
|
Here are the logs from after killing the operator pod then applying an exitnode config: |
|
Still having excessive logs after a cluster wipe. |
|
This might be some kind of feedback loop caused by MetalLB or Envoy wrestling control over changes against Chisel Operator in the service. So I assume this is some kind of conflict caused by it. Does either service try to modify the service state or annotations or anything? Seeing that there are 2 issues related to using Chisel Operator under Envoy I would suggest not using Envoy with it for now, unless we know what's Envoy doing with the services that cause Chisel Operator to freak out. |
|
I'll try and have a look, but that is 2 services that seem to fairly well accepted that may be causing an issue. Is there any way for Chisel Operator to filter superfluous events? Can it check against already provisioned CRDs and rate limit events that don't impact it's own operation? I can delete and add an ExitNode and watch the log entries go from every 5s to every 50ms (respectively, seems like as fast as possible). As soon as the droplet gets an IP, logs go mental. If chisel isn't fundamentally compatible with MetalLB or Envoy Gateway, then I'll explore other options. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello,
I am using chisel operator with Envoy Gateway and MetalLB.
Chisel is correctly picking up that it has to create an exit node, and is provisioning it on Digital Ocean (aside form the issues I am having described in #152 ). It is correctly not targeting any other services. So it seems to be working correctly.
However, the operator is spamming the following logs every 100ms or so (I've cut them down a bit, let me know if you need more info):
This repeats and repeats.
Is this normal?
It seems like it is updating an object, which is triggering an object updated event, which is causing it to update the object....
The text was updated successfully, but these errors were encountered: