Indexing: Index PDF files

[m-i-l]

There are a couple of sites that people have mentioned which consist primarily of PDF files:

• https://icann-hamster.nl/
• http://radar.oreilly.com/r2/release1-0 (main index appears to be gone now, but individual PDF files still seem to be available e.g. via http://cdn.oreillystatic.com/radar/r1/01-83.pdf)

Unfortunately searchmysite.net doesn't currently index PDF files. In fact, parse_item in search_my_site_script.py does a isinstance(response, TextResponse) to ensure it only indexes text content to exclude all binary files (i.e. exclude images etc. not just PDF files). This would have to be updated, and of course there would need to be a way to extract text from PDF files for indexing.

Might want to have a look at the Python bindings in https://cwiki.apache.org/confluence/display/TIKA/API+Bindings+for+Tika or something like that.