-
Notifications
You must be signed in to change notification settings - Fork 106
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to deal with large amounts of data? #61
Comments
Hi @fabioperez, unfortunately I built Xcessiv without deep learning/image processing in mind. It is certainly not feasible to use Xcessiv with the large amounts of data necessary for deep learning (which are usually passed in to a neural network as batches, anyway). Also, you give me too much credit. While I built Xcessiv, I certainly haven't had the time to maintain it in the last year, and I don't see that changing soon. :( |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I am trying to use Xcessiv to an image classification project (together with Keras or PyTorch). By reading the walkthrough, I found that I have to pass the entire dataset as a
(X, y)
tuple. This is unfeasible for a large image dataset. How can I outcome this problem?One solution that I thought about was to pass image paths as the
X
, and letfit
load the data lazily. Is this the best approach?PS: Thanks for creating and maintaining Xcessiv!
The text was updated successfully, but these errors were encountered: