Improve performance of excluded files filter #5157
Conversation
Generated by 🚫 Danger |
SimplyDanny
force-pushed
the
excluded-performance
branch
from
e99f0a9
to
a253a0d
Compare
SimplyDanny
force-pushed
the
excluded-performance
branch
2 times, most recently
from
4094558
to
30d4908
Compare
|
Periphery had a similar performance issue not long ago, and I noticed there wasn't a solid glob to regex implementation, so I ported Python's fnmatch to Swift: https://github.com/ileitch/swift-filename-matcher. It might be useful here too. |
|
Last time we tried to speed this up, it caused some slight differences in the result of what was matched vs not, so please be super careful here. |
|
At the moment, I'm rather concerned here that normal runs without any excludes and includes seem to become much slower sometimes.
This is a very helpful tip. I don't want to invent a half-backed version myself. Thanks! |
SimplyDanny
force-pushed
the
excluded-performance
branch
3 times, most recently
from
bfd249c
to
6a0585a
Compare
SimplyDanny
force-pushed
the
excluded-performance
branch
from
a4e724b
to
a95044b
Compare
SimplyDanny
force-pushed
the
excluded-performance
branch
from
a95044b
to
ca66b0c
Compare
|
Any chance this can land? 🙏 |
SimplyDanny
force-pushed
the
excluded-performance
branch
from
ca66b0c
to
e5989f1
Compare
This is a critical change that needs thorough testing. Unfortunately, I'm lacking own projects with nifty @JaviSoto: In case this change took effect in your projects, I'd appreciate your feedback. |
The current algorithm is like "collect all included files and subtract all excluded files". Collecting all included and all excluded files relies on the file system. This can become slow when the patterns used to exclude files resolve to a large number of files. The new approach only collects all lintable files and checks them against the exclude patterns. This can be done by in-memory string-regex-match and does therefore not require file system accesses.
SimplyDanny
force-pushed
the
excluded-performance
branch
from
e5989f1
to
dad51eb
Compare
SimplyDanny
force-pushed
the
excluded-performance
branch
from
21c0add
to
61acd6d
Compare
The current algorithm is like "collect all included files and subtract all excluded files". Collecting all included and all excluded files relies on the file system. This can become slow when the patterns used to exclude files resolve to a large number of files.
The new approach only collects all lintable files and checks them against the exclude patterns. This can be done by in-memory string-regex-match and does therefore not require file system accesses.
The most critical part is the conversion of glob patterns to regular expressions. I might have missed cases.
Fixes #5018.