Awwwwoooo, It's my personal project to scraping themef****t.net via tor. We're watching all items they have, the old and the new. We collects items' data such as today sales, tags, etc.
- MongoDB (www.mongodb.org)
- Redis (redis.io)
- Tor (Run tor with config .torrc, geoip and geoip6 from
dist
directory)
Run build.sh
and you'll got 3 binaries:
api
will run API server on port 8080 (admin and visualization), with endpoints url/api/v1
worker
will always run and retrieve task from redis viaRPUSH
scheduler
will crawling category/subcategory page from API, it's like cronjob