Some HTML-format reports are split into multiple web pages

[divergentdave]

Some reports are split up over multiple web pages, and we're only fetching the table of contents thus far. For example, http://oig.federalreserve.gov/reports/board-full-report-20140312a.htm points to an executive summary and five sub-pages. This will require multiple URLs and files per report, or maybe crawling the pages we need and stuffing them in a WARC archive, for example. Perhaps scrapers will have to provide a list of URLs for such reports, rather than a single URL.

[divergentdave]

Here's a pathological corner case, a PDF file that links to more PDF files

http://www.epa.gov/oig/reports/2002/Models.pdf