Skip to content

mkuthan/example-spark-kafka

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

62 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Apache Spark and Apache Kafka integration example

Build Status Coverage Status

This example shows how to send processing results from Spark Streaming to Apache Kafka in reliable way. The example follows Spark convention for integration with external data sinks:

// import implicit conversions
import org.mkuthan.spark.KafkaDStreamSink._

// send dstream to Kafka
dstream.sendToKafka(kafkaProducerConfig, topic)

Features

  • KafkaDStreamSink for sending streaming results to Apache Kafka in reliable way.
  • Stream processing fail fast, if the results could not be sent to Apache Kafka.
  • Stream processing is blocked (back pressure), if the Kafka producer is too slow.
  • Stream processing results are flushed explicitly from Kafka producer internal buffer.
  • Kafka producer is shared by all tasks on single JVM (see KafkaProducerFactory).
  • Kafka producer is properly closed when Spark executor is shutdown (see KafkaProducerFactory).
  • Twitter Bijection is used for encoding/decoding KafkaPayload from/into String or Avro.

Quickstart guide

Download latest Apache Kafka distribution and un-tar it.

Start ZooKeeper server:

./bin/zookeeper-server-start.sh config/zookeeper.properties

Start Kafka server:

./bin/kafka-server-start.sh config/server.properties

Create input topic:

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic input

Create output topic:

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 3 --topic output

Start Kafka producer:

./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic input

Start Kafka consumer:

./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic output

Run example application:

sbt "runMain example.WordCountJob"

Publish a few words on input topic using Kafka console producer and check the processing result on output topic using Kafka console producer.

References

Releases

No releases published

Packages

No packages published

Languages