-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve filesystem scanning with parallelism #189
Labels
enhancement
New feature or request
Comments
baruchiro
pushed a commit
to nirmo/2ms
that referenced
this issue
Sep 28, 2023
Hi, please assign for me the issue. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
To scan with the
filesystem
plugin, we first walk over the whole directory tree, collect all the file names, and then loop over them.2ms/plugins/filesystem.go
Lines 64 to 92 in 8d71105
Instead, we can immediately handle every file we find, by sending it into a channel or launching a new gorutine.
(I'm not sure what will be the best approach. I think maybe launching a new routine for each file will be the most parallelism degree, but maybe to control all the routines and wait for them, we need to use a channel from the
walk
function to one goroutine that will trigger all the goroutines per file)The text was updated successfully, but these errors were encountered: