This is a Webcrawler based on Scrapy and Selenium frameworks
This spider crawls thru the directferries.com website in order to generate a json file with all tickets available for one of these directions Dublin-Liverpool / Liverpool-Dublin with departure tomorrow and returning in 3 days.
HOW TO EXECUTE:
Is required an environment with the following requirements installed: -Python 2.7 -Scrapy 0.18 -Selenium web-drivers
To execute the crawler the following command should be executed from the project path
scrapy crawl crawlermate_selenium -a category=[dublin or liverpool] -o [filename] -t json
for an example, to generate tickets for dublin to liverpool on items.json file you should execute scrapy crawl crawlermate_selenium -a category=dublin -o items.json -t json
References: