-
Notifications
You must be signed in to change notification settings - Fork 463
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
learn_bpe.py error #102
Comments
thanks for reporting this issue. @yimmon , this is related to parallel support you contributed; could you have a look? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I am running with a very big file: about 150M lines, disk size 60GB, --num-workers 10, and then :
'vocab += pickle.load(f)' in learn_bpe.py will report error: EOFError: Ran out of input.
tested on windows 10 os. I assume the 'tmp = tempfile.NamedTemporaryFile' introduce this?
anyone has such experience?
thx
The text was updated successfully, but these errors were encountered: