-
Notifications
You must be signed in to change notification settings - Fork 881
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CoCa RoBERTa Attention Map Size Issue #864
Comments
@sandeepmukh Then, I think this is needed in CocaModel to replace current vocab_size logic btw text and multimodal text towers
Also, the context_len used by tokenzier sources from text_cfg by default, so text_cfg and multimodal_cfg should have same context_len values in config (I think) to work best but I'm not 100% sure there. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi! I'm trying to train CoCa using the pretrained RoBERTa weights (has the casual masking issue #445 been addressed?), but I am running into an error with the Attention Maps sizes. Any help would be greatly appreciated :).
Below is the command I'm running:
However, this errors:
Inspecting the error, I tried to change the multi-modal context length to 77, which yields the following error:
The text was updated successfully, but these errors were encountered: