[ cpu_backend ] Add q4_0_4_8 GEMM + multithreading acceleration + q6_K bstp implemenation + cblas removal option #3350
+1,434
−144
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- ggml_gemm_q4_0_4x8_q8_0 - ggml_gemv_q4_0_4x8_q8_0 - ggml_repack_q4_0_to_q4_0_4_bl - By changing adaptable function params and non-static functions **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: skykongkong8 <[email protected]>
- Implement both openMP & bstp version of q4_0_4x8_q8_0 GEMM and GEMV - In the future _FP16 activation flow will be supported. Thus, add the function in a function template manner. **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: skykongkong8 <[email protected]>
- SIMD-optimized q40 GEMM kernel is q4048, since NEON register is 128 bit, not 256 bit like AVX2 or SVE **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: skykongkong8 <[email protected]>
- ARM : use repack_q4_0 function use __ggml_repack_q4_0_to_q4_0_4 kernel - x86 : use repack_q4_0 function use __ggml_repack_q4_0_to_q4_0_8 kernel - Fix unittest accordingly **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: skykongkong8 <[email protected]>
- This impl is fine-grained multithreading. - Should compare with coarse-grained multithreading version later on. **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: skykongkong8 <[email protected]>
skykongkong8
requested review from
myungjoo,
jijoongmoon,
again4you,
jaeyun-jung,
leemgs,
wooksong,
gichan-jang,
anyj0527,
lhs8928,
songgot,
jihochu,
DonghakPark,
SeoHyungjun,
baek2sm,
djeong20,
EunjuYang,
dkjung and
haehun
as code owners
skykongkong8
changed the title
- Change GGML_FP16_TO_FP32 -> GGML_COMPUTE_FP16_TO_FP32 - This patch resolves zero-value issues after lm head. - Remember : this patch should be eventually removed, since this bug occurs bevause ggml_init function is not properly called at model runtime. **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: skykongkong8 <[email protected]>
skykongkong8
changed the title
- In specific circumstances, it is better to build without cblas option. - trivial) Add option for ggml_interface.h to choose ggml related files or not. **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: skykongkong8 <[email protected]>
skykongkong8
force-pushed
the
pr/ggml/arm/q4048GEMM
branch
from
b4fbe30 to
d0c54da
Compare
**Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: skykongkong8 <[email protected]>
skykongkong8
force-pushed
the
pr/ggml/arm/q4048GEMM
branch
2 times, most recently
from
458c387 to
f06709c
Compare
**Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: skykongkong8 <[email protected]>
- q8_0 quant/dequant function to compare f16 f32 quant loss comparison **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: skykongkong8 <[email protected]>
skykongkong8
force-pushed
the
pr/ggml/arm/q4048GEMM
branch
from
f06709c to
17f5eb1
Compare
- quantize_q8_0 - dequantize_row_q8_0 - gemm_q4_0<float> and gemm_q4_0<_FP16> **Self evaluation:** 1. Build test: [X]Passed [ ]Failed [ ]Skipped 2. Run test: [X]Passed [ ]Failed [ ]Skipped Signed-off-by: skykongkong8 <[email protected]>
skykongkong8
force-pushed
the
pr/ggml/arm/q4048GEMM
branch
from
17f5eb1 to
358c9f7
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes:
Self evaluation:
Signed-off-by: skykongkong8 [email protected]