-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Key error with file data/Imagenet64/metadata.pkl #9
Comments
After running "loadImagenet64.py", my .pkl files have:
"make_massive_tensor.py" cannot reshape them to (3, 64, 64) (<-- line 62 in "make_massive_tensor.py") If that can help:
|
The
Just a heads up: As for training on a mac, I haven't added mps support to the repo as in PyTorch 1.0, the MPS device had all types of weird issues. I'm not sure if that was fixed in PyTorch 2.0. For inference, this may be fine, but for training, I think you may run into multiple issues trying to get it to work properly. Hope this helps! Let me know if you run into any more issues. |
Hello! Thank you very much for your answer! — For Imagenet64_train_part1.zip in your — I changed a bit of code to get MPS device to work ok on Apple Silicon. In fact, I used MPS device declaration within PyTorch on previous projects and that worked. I do not fully understand as I am not a specialist. But it seems to be working like a charm now with PyToch 2 on MacOS 12.3+. — I will keep you posted with my tests and if I can make it work. I may ask for help from someone around me too. Thank you again, O. |
"One more thing": :) Do you know the shape of the tensors in your .pkl files for "img" and "label"? |
oh that makes sense. So the Kaggle dataset is probably in a different format from the ImageNet dataset which is why you're running into issues loading in the data. I haven't added support for the Kaggle dataset, but I'd imagine the process should be the same. Perhaps the Kaggle dataset uses the numpy version? Anyways, it'd be awesome if you could get the data working with Kaggle! Looking at the
|
Yes it does. But that is not the problem.
That makes sense here too! This is not what I have: I have a closer look at it! O. |
I think |
Hello!
I am trying to make to train a model myself using Imagenet64x64 for a test; on a MAC using "mps" device.
It took me a little while to see that after downloading Imagenet64x64, I have to
— use "loadImagenet64.py" to generate .pkl files in an "Imagenet64" folder.
— THEN use "make_massive_tensor.py" to make a large .pt file.
— THEN use "train.py" which will call "model_trainer.py"
Apparently (tell me if I am wrong):
— "loadImagenet64.py" needs "Imagenet64_train_part1.zip" and "Imagenet64_train_part2.zip".
Imagenet64x64 does not have these files. It rather has:
train_data_batch_1, train_data_batch_2, train_data_batch_3... etc
— I changed the code in "loadImagenet64.py" to make a series of img and label .pkl files within the "Imagenet64" folder.
— Then, when running "make_massive_tensor.py", I get the following errors:
— I probably did something wrong in "loadImagenet64.py" with the formatting of pickles. But I do not know where that is happening. Dict's keys seem fine: loadImagenet64 seems to replace 'data', 'mean', 'labels' found in Imagenet64 with 'data', 'mean', 'labels'.
=> Where did you get "Imagenet64_train_part1.zip" OR how did you make them?
=> How to deal with the shape and dict keys within those .pkl ?
Thank you for your help!!!
O.
The text was updated successfully, but these errors were encountered: