Historical Archive

[jqnatividad]

For certain popular sites, the admin should have the option to schedule crawls, and for those sites that have a crawl history, be able to see the results.

Also, for crawl history, have to option to fetch the data itself and archive it as a lot of these files are updated in place, and some of the historical data is lost.

[jqnatividad]

The historical archive would double as a cache too (#26)

[waldoj]

This sounds like a fine feature. It's way outside of the scope of what we're prepared to do with this, but it would be lovely to see somebody else add it!