-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New Functionality: Input Keys from a File to Download #103
Comments
Or Just guide me how to do it, I will try to hack it |
Currently you can download multiple files based on the prefix. If this does not work for your scenario, more details on why not would be helpful. Nevertheless, there's already an enhancement request to make inputs of the -n and -f options available via a file. The current thinking on this (feedback is welcome) is to support something like this: |
Sry for the delay in Reply, Regarding your first question on why does prefix based download doesn't work, it does work for my use case because, am maintaining an Index (Database) of Blob keys where depending on the search criteria on top of the database a bunch of output keys will be emitted and for those keys data has to be downloaded. |
Any chance of this getting picked up ? |
HI,
Can we have this additional functionality where list of keys to be downloaded can be provided via a file through an additional argument and for simplicity this input file can be an other Blob as well.
Use case, we have millions of small files(images) which are required for training on a need basis, so it would be very handy for this use case.
Right-now am relying on Python multiprocessing for the same but I think Go would be much faster based on the experience using your module.
Thanks
The text was updated successfully, but these errors were encountered: