Skip to content

[RFC] Break down the static library into multiple files#300

Open
Algunenano wants to merge 1 commit intoashvardanian:mainfrom
Algunenano:static_library
Open

[RFC] Break down the static library into multiple files#300
Algunenano wants to merge 1 commit intoashvardanian:mainfrom
Algunenano:static_library

Conversation

@Algunenano
Copy link
Copy Markdown
Contributor

After updating ClickHouse to 4.6.0 we've noticed that stringzilla.c has become a bottleneck in the build, as it takes from ~90 (release) to ~150 (asan) seconds to build. The problem is that it build all the functions in a single translation unit, and compiling and specially optimizing it takes quite a bit of time.

Instead we could separate the functions into different files and build them independently and in parallel. Doing it this way reduces the build time to the slowest file (utf8_case.c, around 3 seconds with ASAN):

/mnt/ch/ClickHouse/contrib/StringZilla/c/compare.c

real    0m0.295s
user    0m0.275s
sys     0m0.019s
/mnt/ch/ClickHouse/contrib/StringZilla/c/find.c

real    0m0.708s
user    0m0.674s
sys     0m0.031s
/mnt/ch/ClickHouse/contrib/StringZilla/c/hash.c

real    0m1.023s
user    0m0.963s
sys     0m0.054s
/mnt/ch/ClickHouse/contrib/StringZilla/c/intersect.c

real    0m2.192s
user    0m2.130s
sys     0m0.049s
/mnt/ch/ClickHouse/contrib/StringZilla/c/memory.c

real    0m1.043s
user    0m1.002s
sys     0m0.035s
/mnt/ch/ClickHouse/contrib/StringZilla/c/sort.c

real    0m1.368s
user    0m1.321s
sys     0m0.040s
/mnt/ch/ClickHouse/contrib/StringZilla/c/stringzilla.c

real    0m0.209s
user    0m0.184s
sys     0m0.024s
/mnt/ch/ClickHouse/contrib/StringZilla/c/utf8.c

real    0m0.353s
user    0m0.326s
sys     0m0.025s
/mnt/ch/ClickHouse/contrib/StringZilla/c/utf8_case.c

real    0m3.025s
user    0m2.891s
sys     0m0.108s
/mnt/ch/ClickHouse/contrib/StringZilla/c/utf8_word.c

real    0m0.223s
user    0m0.195s

To implement this each file includes a single header and redefines the macros to expose the symbols in the translation unit (built SZ_DYNAMIC_DISPATCH=0) and in the headers we hide the implementation when building with SZ_DYNAMIC_DISPATCH=1, so stringzilla.c only sees the function definitions.

I had to fight with the LLM to get it to do what I needed, so for transparency here is its summary.

Note that I've only tested this with Linux and clang so far. It's likely that other platforms would need a different approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant