[Question] How to load one more vision tower?

[zhlhlhlhl]

Question

I want to initialize one more vision tower in the llava_arch.py. I added
self.sam_model = SamModel.from_pretrained(model_args.sgtm_sam_model_path,torch_dtype=torch.bfloat16) in the
def initialize_vision_modules(self, model_args, fsdp=None), and I mimic the original code to use def get_vision_tower(self) in the class LlavaMetaForCausalLM(ABC).
However, I checked the self.sam_model that I want to initialize, and the parameters seem to be empty. How can another vision model in this part code be initialized correctly?