Real-Time Data Streaming Pipeline using

Kafka and Python

Overview

A real-time streaming pipeline can stream live tweets from Twitter and ingest the data to the Apache Kafka clusters as a topic, and consumers consume the hashtag tweets as a message.

Create a developer account here to get the Twitter API key. Then change the Twitter API key configuration at the virtual environment level by the newly generated API key.

    export CONSUMER_KEY="XXXXXXXXXXXXXXXXXXXXXXXXX"
    export CONSUMER_SECRET="XXXXXXXXXXXXXXXXXXXXXX"
    export ACCESS_KEY="XXXXXXXXXXXXXXXXXXXXXXXXXXX"
    export ACCESS_SECRET="XXXXXXXXXXXXXXXXXXXXXXXX"

Install

    git clone https://github.com/saboye/Real-Time-Data-Streaming-Pipeline.

    pip install -r requirements.txt
 

Kafka configuration

    advertised.listeners=PLAINTEXT://localhost:9092
    zookeeper.connect=localhost:2181 

To run locally you need to installed Apache Kafka

1. run the zookeeper the next command:

    bin/zookeeper-server-start.sh config/zookeeper.properties 

2. To run the Kafka use the following command:

    JMX_PORT=8004 bin/kafka-server-start.sh config/server.properties

3. run the producer Notebook.

4. run the Consumer Notebook.

Author

👤 SABOYE

• Twitter: @saboye1
• Github: @saboye

🤝 Contributing

Contributions, issues and feature requests are welcome!
Feel free to check issues page.

Show your support

Give a ⭐️ if this project helped you!

📝 License

This project is MIT licensed.

---