Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add annotations to QKV layers #291

Open
seanmor5 opened this issue Nov 20, 2023 · 0 comments
Open

Add annotations to QKV layers #291

seanmor5 opened this issue Nov 20, 2023 · 0 comments
Labels
kind:chore Internal improvements note:upstream The issue must be tackled upstream

Comments

@seanmor5
Copy link
Contributor

Related to elixir-nx/axon#549

LoRA targets QKV nodes. Right now the only way to do it is by inferring based on layer name. We can provide more information by giving specific metadata (such as op_name) for which layers are QKV
@jonatanklosko jonatanklosko added note:upstream The issue must be tackled upstream kind:chore Internal improvements labels Feb 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind:chore Internal improvements note:upstream The issue must be tackled upstream
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@seanmor5 @jonatanklosko