-
Notifications
You must be signed in to change notification settings - Fork 696
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error reddit scraping #1005
Comments
Pushshift is effectively dead, so yeah, this is expected and can't work anymore. Pushshift was the only way to retrieve (a) useful search results since Reddit's own search is awful, (b) get all submissions in a subreddit since Reddit limits that to 1000 results, and (c) get all submissions/comments by a user due to the same limitation on Reddit. Potentially, PullPush could serve as a replacement, but since Reddit's API changes are rolling out this month, I'll wait for that to happen before making any changes. (If the Reddit API itself is sufficient for your purposes, I recommend using PRAW rather than snscrape.) |
Describe the bug
I have the following error when I try to scrape reddit :
snscrape.base.ScraperException: 4 requests to https://api.pushshift.io/reddit/search/submission?q=toto&limit=1000 failed, giving up.
I also tried with the python package and subreddit search but it doesn't work either.
I tried to do it from another device but the same result...
Any idea?
How to reproduce
run
snscrape -n 100 -vv reddit-search toto
Expected behaviour
Get data?
Screenshots and recordings
No response
Operating system
Kubuntu 22.04
Python version: output of
python3 --version
3.8.8
snscrape version: output of
snscrape --version
snscrape 0.7.0.20230622
Scraper
reddit-search
How are you using snscrape?
CLI (
snscrape ...
as a command, e.g. in a terminal)Backtrace
snscrape.base.ScraperException: 4 requests to https://api.pushshift.io/reddit/search/submission?q=toto&limit=1000 failed, giving up.
Log output
Dump of locals
I prefer to send it in private
Additional context
No response
The text was updated successfully, but these errors were encountered: