LLaVa support

[SinanAkkoyun]

Feature request

Hello! It would be awesome to have LLaVa support (upload an image to the API and have it embed it via CLIP etc)
https://github.com/haotian-liu/LLaVA

text-generation-webui already has this built in, but we need to make use of the true model parallelization for faster inference of this repository.

Thank you very much! :)

Motivation

Current LLaVa inference code does not allow for true model parallelization. We (as many others) need very fast inference with LLaVa models.

Your contribution

I am not experienced enough to do any of the foundational work but I would love to help when the foundation is there, porting code from the LLaVa repository to here.

[Narsil]

Hi,

The first step would likely be implementing it like here: huggingface/transformers#22848 within transformers itself.

Then, this library has custom modeling code to speed things up, but it's very much hands-on work.
If you want to tackle it, you could look at llama for instance since the models should be relatively similar.
The 2 main things to consider are:

• Sharding: TP parallelism is important, that means replacing some layers with their sharded counterpart.
• Flash attention: This means changing everything around from transformers to remove padding and use specifically the flash attention kernels.

Also if I read correctly, this is a multimodal model. This is probably quite different from the current models supported by this repo which are highly targeted towards pure text generation. I am not sure if this repo will be a good fit (there are many assumption this repo relies on which might not be true for llava)

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.