You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Unfortunately, every now and then requests still get dropped due to the IP blocking, despite the throttling.
Because the archives operation works on a periodic scheduled basis, this generally doesn't pose a problem, as those URLs will eventually be archived. But it is not graceful, and it'd probably be better if we had requests anticipate the IP throttling and wait a while before attempting it again, and only stopping for good if waiting fails 3 times in a row.
IP address blocking occurs for a period of 5 minutes at a time, per the linked documentation, so waiting for that length of time in the case of a NewConnectionError would be one solution. But there could be others.
The text was updated successfully, but these errors were encountered:
Because the Internet Archives temporarily blocks IP addresses which attempt to perform more than 15 archive requests in a single minute, the code has been throttled to reduce the risk of requests being dropped.
Unfortunately, every now and then requests still get dropped due to the IP blocking, despite the throttling.
Because the archives operation works on a periodic scheduled basis, this generally doesn't pose a problem, as those URLs will eventually be archived. But it is not graceful, and it'd probably be better if we had requests anticipate the IP throttling and wait a while before attempting it again, and only stopping for good if waiting fails 3 times in a row.
IP address blocking occurs for a period of 5 minutes at a time, per the linked documentation, so waiting for that length of time in the case of a
NewConnectionError
would be one solution. But there could be others.The text was updated successfully, but these errors were encountered: