Why Kafka? And how would that work on OpenShift?

[dudash]

A question I've gotten informally when discussing this with colleague was "Why use Kafka and how does that run on OpenShift/Kubernetes?"

Specifically, the "Why Kafka" was targeted at other options for microservices to communicate with each other (like REST, gRPC, etc.). This is probably the more important question of the two for this repo.

The "How does it work in K8s/OpenShift?" was looking to understand how a multi-broker with zookeeper would work in a managed container platform like OpenShift.

So I'm starting that discussion here.

[dudash]

How would it work on OpenShift is straightforward to answer. And I'll punt to someone else's documentation. See the following articles:

1. https://developers.redhat.com/blog/2018/10/25/welcome-apache-kafka-to-the-kubernetes-era/
2. https://developers.redhat.com/blog/2018/10/29/how-to-run-kafka-on-openshift-the-enterprise-kubernetes-with-amq-streams/

[dudash]

For reference the standard Kafka use cases: https://kafka.apache.org/uses

• Messaging
• Website Activity Tracking
• Metrics
• Log Aggregation
• Stream Processing
• Event Sourcing
• Commit Log

In this case Kafka is only being used for a subset of the communications of the services. So REST/gRPC aren't off the table. We are streaming data from multiple decoupled services into Kafka for the purposes of a search service. It's similar to the discussion on this stackexchange:

• https://softwareengineering.stackexchange.com/questions/352194/search-across-data-cross-several-microservices

Because this is just a demo repo with no actual users or requirements to meet, this seemed like an interesting example to showcase as part of the architecture.

[dudash]

an alternative (not chosen) for the search service to get access to multiple service's data would be to use a composition pattern like this:

• https://microservices.io/patterns/data/api-composition.html

[dudash]

This might need to be reconsidered now that istio/jaeger should be able to leverage Kafka (envoyproxy/envoy#8188) and we are considering making istio first-class in this demo (#43 )