Images requiring a Referer header are not fetched

[jinshuqishi2019]

Environment

• Operating System: debian 10
• node --version: v20.12.2
• npm --version: 10.7.0
• percollate --version:v4.1.1

Description

Hello!
When using percollate epub to generate EPUB files, I sometimes notice missing images or text. I found that some websites require the Referer parameter to be set for images to prevent hotlinking; otherwise, the images show a 403 error when downloaded. The missing text issue is due to the content being dynamically generated by JavaScript. Do you have any good solutions for these two situations? Thank you.

[danburzo]

Do you happen to have an example command for a page whose images have hotlink prevention? There may be something we can do about it, if we fetch them like the browser would, by respecting the Referrer Policy.

As for content generated with JavaScript, percollate does not run the original webpage in Puppeteer (Chromium), so you must fetch the page externally. For example, monolith suggests using chromium:

chromium --headless --incognito --dump-dom https://github.com | monolith - -I -b https://github.com -o github.html

You could do something similar with Percollate:

chromium --headless --incognito --dump-dom https://github.com | percollate --url https://github.com -

[jinshuqishi2019]

Example of missing images1:

percollate html https://sspai.com/post/88977

Perhaps the percollate can be used to add custom request headers.
If the Referer: https://sspai.com/ is added to the picture request header, it can be displayed normally.

Example of missing images2:
Referring to the instructions above, I tried to operate, but there were still problems. It would be better if there were more detailed explanations.

apt-get install chromium
chromium --headless --incognito --dump-dom --no-sandbox https://mp.weixin.qq.com/s/B_-_5k0JMiDoWWTAv7aSbw | percollate html https://mp.weixin.qq.com/s/B_-_5k0JMiDoWWTAv7aSbw

I can see the content, but still missing images

[danburzo]

Thanks for the test case. It seems that PDF generation has the same issue with images, because of the lack of Referer. I will look into what can be done.

As for the chromium command, the arguments to percollate are a bit obscure so let me unpack them a bit:

chromium --headless --incognito --dump-dom https://github.com | percollate --url https://github.com -

Notice the - argument at the very end: this makes Percollate read the HTML from the standard input instead of from an URL. This way it loads the result of the chromium command. Since we get the raw HTML, we must then provide the original URL manually, with the --url option, so that relative links (and other features) work correctly. See also this README section.

[danburzo]

Released a fix as part of [email protected]. For EPUBs, you can run the command as usual. For HTML and PDF output, you’ll need to use the --inline flag for images that require a Referer header.