Try alternative URLs for MNIST dataset download #589
Conversation
|
Instead of building your own redundancy scheme in code, why not simply point to a cloud-based storage location, such as the S3 bucket in your list (or alternatively any Dropbox/Gdrive/CDN)? These should have high enough availability to basically make this a non-issue (as opposed to yann.lecun.com, which seems to be hosted somewhere at NYU, perhaps just a single machine sitting under someone's desk) |
Well, firstly because we (@ebenolson, actually) pay for S3 and secondly because I'd prefer to include the official and probably most permanent location. Not sure if the costs for S3 would noticeable, but people use the example code as a basis for other experiments and so the link could spread quite a bit. It's the first time I've seen LeCun's website being down, by the way -- from my (limited) experience, it's highly available. That's why I propose to just leave it the way it is for now. As a compromise, we could just list the alternative download locations in a comment so they're easy to swap in (this wouldn't help with continuous integration, but at least make it easier for end users). |
| urlretrieve(source + filename, filename) | ||
| except URLError as exc: | ||
| if exc.args[-1].args == (-2, 'Name or service not known'): | ||
| sources.remove(source) # do not try this source again |
There was a problem hiding this comment.
Keep in mind sources is mutable. I think you might be doing this intentionally, but it's setting off all my Python gotcha alarms.
There was a problem hiding this comment.
Thought the comment would explain. But yes, it's on purpose. If the primary location is down when downloading the first file, I don't want it to be tried again for the next three downloads. I didn't call it ugly for no reason ;) We could move the sources list into the outer context to make it clearer.
Fixes #587. But it makes things a bit ugly... I tend towards holding back until download availability really becomes a problem.
Note: This will work on Python 2 and 3 on an English Ubuntu, but I don't know if it's portable to other languages or Windows, Mac. Would be interesting to know what Exception exactly
urlretrieve('http://foo.bar', 'foo')throws on other systems.