Triaged 03.10.2024: agreed on implementing this.

@scholzj I’m interested in contributing more to Strimzi and would love to be involved in development. Is this something I can pick up as the next candidate?

I guess you can if you want. But this is a fairly advanced issue.

@varada-sunanda-ibm if you are not working on this, I would like to. Please let me know if you are.

@scholzj @ppatierno Can I please clarify things on the Mirrormaker cluster configurations setup with you? The PR, I raised currently, removes kafka_connect_config_generator.sh and creates a configmap instead. The configmap would be used for both Connect cluster and MirrorMaker cluster (built on top Connect).

Is the intention of this issue also to remove kafka_mirror_maker_2_connector_config_generator.sh and use configmap instead? Because the configurations generated by kafka_mirror_maker_2_connector_config_generator.sh is used different to kafka_connect_config_generator.sh. For example, this script generates strimzi-mirrormaker2-connector.properties based on the exported environment variables, and then it is used to provide MM2 configuration values for source and target clusters. For example, ssl truststore for a source cluster is configured as:
source.cluster.ssl.truststore.password = ${file:/tmp/strimzi-mirrormaker2-connector.properties:ssl.truststore.password}

@tinaselenge The intention of this issue was about the Connect properties file generated by kafka_connect_config_generator.sh. To be honest, the connector configurations did not really come to my mind when I opened it. So, having this not done in the initial PR is definitely fine as long as it can work separately.

I'm not sure how much we can simplify the kafka_mirror_maker_2_connector_config_generator.sh script. It essentially just generates the files with secret information that is loaded from the connectors through config providers. I guess we could generate it in the operator and pass it to the Pod through our own Secret. Or can we pass the secrets directly into the REST API? That might be insecure I guess.

I guess we should try that to get rid of that ugly script as well. But it might need some more thinking maybe what exactly we can do.

@tinaselenge let's leave the current PR focused on kafka_connect_config_generator.sh replacing with a ConfigMap.

@scholzj @ppatierno will Proposal: Support for Custom Authentication Type in Kafka Clients strimzi/proposals#148 be supported after the replacement of kafka_connect_config_generator.sh with ConfigMap? If not, is their a plan for supporting custom authentication for Kafka Connect, MirrorMaker 2, and Kafka Bridge?

@scholzj @ppatierno will Proposal: Support for Custom Authentication Type in Kafka Clients strimzi/proposals#148 be supported after the replacement of kafka_connect_config_generator.sh with ConfigMap? If not, is their a plan for supporting custom authentication for Kafka Connect, MirrorMaker 2, and Kafka Bridge?

No, it won't be supported. But its implementation might be significantly easier than it would be today.

No, it won't be supported. But its implementation might be significantly easier than it would be today.

Thanks for the update! Do you have an estimate for when the PR#11062 might be merged?

I'm not sure anyone can give you an real estimate when something happens in an open source project. You can follow that PR and there might be one more afterwards for some of the MM2 stuff possibly.

I'm still working on my Proposal: Support for Custom Authentication Type in Kafka Clients #148. However, I noticed there are conflicts with that PR#11062. Could you please clarify the relationship between my proposal and that PR? How do we plan to move forward? Should I pause my work and wait for that PR to be completed before continuing development based on it?

@YuanShisong I'm hoping PR #11062 will be merged soon so you can wait and rebase with it. From the proposal alone, I can't tell what conflicts you are encountering but I imagine it is to do with how Kafka client authentication configured for Connect/Mirrormaker clusters. With this PR, based on the CR's authentication field, a configmap containing all the required fields (e.g. sasl.mechanism, sasl.jaas.config etc.) is created which would be then volume mounted in connect pods. Since you are introducing a new authentication type, it needs to be handled in the configuration setup too so that config map contains relevant fields for the custom authentication.