-
Notifications
You must be signed in to change notification settings - Fork 538
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multi-threading preprocess #76
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The preprocess (as well as converter) basically iterates and extracts features for each file independently. This can be easily multi-threaded with python on a multi-core machine or bigger clusters and get linear speedup. This can be beneficial especially when you want to experiment with preprocess hyperparameters. I can possibly do that (not sure when) if nobody has such plan yet.
The text was updated successfully, but these errors were encountered: