Add support for graceful shutdown

[mlallaouret]

Hello,

When doing some load testing of kafka bridge on kubernetes and doing some rolling restart, some messages are not successful (http code 502) and so some data are lost..

Steps to reproduce:

• Deploy 2 replicas of kafka-bridge
• Use tool like fortio to generate multiple http calls over some time.
• While generating http calls, launch a rolling restart of the kafka bridge deployment

I tested to patch the kafka-bridge deployment with a preStop hook like the one below and didn't see any dropped messages anymore.

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 10"]

The issue is we cannot specify pre stop hook in the kafkabridge custom resource for now.
However, a cleaner solution seems to be to implement vertx graceful shutdown but I didn't tested this one as it requires code changes.

What do you think ?

[ppatierno]

Hi, it's not clear to me how you are deploying the bridge.
You are mentioning Deployment but then referring the KafkaBridge custom resource, so it's not clear if you are using the bridge within a container image and then using your own Deployment or using the Strimzi operator with the KafkaBridge custom resource.
Another thing is that the KafkaBridge can't have more than one replicas because how it works, mostly from a consumer perspective. Each consumer is tight to one instance and if you put a Service in front of it which goes in round robin you would not reach the right HTTP bridge instance where your consumer "lives".
Said that, a graceful shutdown would be useful ... are you willing to contribute to it, by making the changes, testing and open a PR? Thanks!

[mlallaouret]

Thanks for your quick answer !

I deploy the bridge with the strimzi operator.
For information, here is the custom resource I am using:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaBridge
metadata:
  name: main
  namespace: kafka
spec:
  replicas: 2
  bootstrapServers: main-kafka-bootstrap:9094
  authentication:
    type: oauth
    accessTokenLocation: /var/run/secrets/kubernetes.io/serviceaccount/token
    enableMetrics: true
  consumer:
    enabled: false
  enableMetrics: true
  logging:
    type: inline
    loggers:
      rootLogger.level: DEBUG
  template:
    serviceAccount:
      metadata:
        annotations:
          iam.gke.io/gcp-service-account: <service_account_name>
    pod:
      topologySpreadConstraints:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/name: kafka-bridge
              app.kubernetes.io/instance: main
          maxSkew: 1
          minDomains: 2
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          matchLabelKeys:
            - pod-template-hash
  http:
    port: 8080

In order to do my test with the lifecycle hook, I scaled down to 0 the strimzi operator to be able to patch manually the kafka-bridge deployment.
For now, I use it only to produce messages so I am not impacted with the instance stickyness issue (unless I am missing something ?)

[ppatierno]

Ok now it's clearer thanks! :-)

For now, I use it only to produce messages so I am not impacted with the instance stickyness issue (unless I am missing something ?)

Yes the producer side is not impacted by the instance stickyness so you are "safe" from this pov.

Anyway, I think having a graceful shutdown would be good.
Not sure when someone will come to fix it (I can't right now but I could in the next weeks) unless you are willing to contribute to Strimzi. It would be really welcome! :-)

[mlallaouret]

What would you recommend between adding the ability to configure a lifecycle hook in the KafkaBridge custom resource (at the path .spec.template.bridgeContainer.lifecycle ?) or adding the graceful shutdown in the kafka bridge source code ?

I can try to make a contribution but I have not a lot of knowledge on the strimzi operator code base and even less on VertX framework

More I will be off until the end of the month, so I will not be able to take a look before september.

[ppatierno]

What would you recommend between adding the ability to configure a lifecycle hook in the KafkaBridge custom resource (at the path .spec.template.bridgeContainer.lifecycle ?) or adding the graceful shutdown in the kafka bridge source code ?

I would recommend to work on the Vert.x side, because the HTTP bridge can even run on bare metal or virtual machine so a graceful shutdown even when working outside of Kubernetes is better imho.

More I will be off until the end of the month, so I will not be able to take a look before september.

Well, that would be the same for me. Busy now and then holiday :-)

[scholzj]

I agree with @ppatierno that this should be solved in the bridge itself and not by hacking around it in the Strimzi Cluster Operator.