Skip to content

FlashMLA support #1159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 38 commits into
base: master
Choose a base branch
from
Open

FlashMLA support #1159

wants to merge 38 commits into from

Conversation

EricLBuehler
Copy link
Owner

@EricLBuehler EricLBuehler commented Feb 25, 2025

Support FlashMLA for improved throughput for MLA models (DeepSeek V2, V3/R1) on CUDA.

EricLBuehler/candle#74

https://github.com/deepseek-ai/FlashMLA

Copy link

github-actions bot commented Feb 25, 2025

Code Metrics Report
  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 C Header                2           34           29            0            5
 Dockerfile              1           41           22           10            9
 JSON                   12          105          104            0            1
 Makefile                1            6            5            0            1
 Python                 73         3126         2710           85          331
 Shell                   1           58           22           18           18
 Plain Text              3         3723            0         2413         1310
 TOML                   19          531          492            2           37
 YAML                    2           21           19            2            0
-------------------------------------------------------------------------------
 Jupyter Notebooks       4            0            0            0            0
 |- Markdown             2           77           32           31           14
 |- Python               2          205          178            1           26
 (Total)                            282          210           32           40
-------------------------------------------------------------------------------
 Markdown               50         4205            0         3196         1009
 |- BASH                 6          103          100            0            3
 |- JSON                 1           12           12            0            0
 |- Python               7          121          109            0           12
 |- Rust                17          586          495            0           91
 |- TOML                 2           75           63            0           12
 (Total)                           5102          779         3196         1127
-------------------------------------------------------------------------------
 Rust                  339       112404       100684         2173         9547
 |- Markdown           158         1808           25         1642          141
 (Total)                         114212       100709         3815         9688
===============================================================================
 Total                 507       124254       104087         7899        12268
===============================================================================
  

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant