Distributed web-crawler based on Docker

Introdution

This is a small web-crawler project used distributed architecture and based on Docker. It also used cool OpenSource projects :

• RethinkDB
• Gearman Job Server
• Redis
• Bloom Filter Server

It's mainly built on Python Language, but you can use many others Languages (C, C++, Java, PHP, NodeJS, ...) to implement.

Distributed Architecture

Requirements

• Docker
• Docker Compose

Installation

Step 1 : Build docker image

git clone https://github.com/khanhicetea/distributed-webcrawler.git
cd distributed-webcrawler
docker-compose build

Step 2 : Run init containers

docker-compose up -d

Step 3 : Run crawlers

docker exec -it distributedwebcrawler_worker1_1 /entrypoint.sh crawler [number_of_crawlers]

ENJOY !!!

• Go to : http://YOUR_IP_ADDRESS:8080/ to manage your RethinkDB database
• Go to : http://YOUR_IP_ADDRESS:4731/ to view Gearmand server status

Credits

• All contributors

License

Licensed under the MIT License - see the LICENSE file for details.