Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Good First Issue] Verify baichuan2-7b-chat with GenAI text_generation #328

Closed
wants to merge 5 commits into from
Closed

[Good First Issue] Verify baichuan2-7b-chat with GenAI text_generation #328

wants to merge 5 commits into from

Conversation

mengbingrock
Copy link
Contributor

Verified baichuan2-7b-chat with GenAI text_generation, added it to Github workflow and README.

This PR should be merged after a change to optimum-intel is made: 273

@mengbingrock mengbingrock mentioned this pull request Mar 26, 2024
Open
@pavel-esir pavel-esir self-requested a review March 26, 2024 09:17
@mengbingrock
Copy link
Contributor Author

Not sure why the CI failed, it could run locally:

convert_tokenizer ./Baichuan2-7B-Chat/pytorch/dldt/FP16/ --output ./Baichuan2-7B-Chat/pytorch/dldt/FP16/ --with-detokenizer --trust-remote-code
Loading Huggingface Tokenizer...
Converting Huggingface Tokenizer to OpenVINO...
Saved OpenVINO Tokenizer: Baichuan2-7B-Chat/pytorch/dldt/FP16/openvino_tokenizer.xml, Baichuan2-7B-Chat/pytorch/dldt/FP16/openvino_tokenizer.bin
Saved OpenVINO Detokenizer: Baichuan2-7B-Chat/pytorch/dldt/FP16/openvino_detokenizer.xml, Baichuan2-7B-Chat/pytorch/dldt/FP16/openvino_detokenizer.bin

@mengbingrock
Copy link
Contributor Author

mengbingrock commented Mar 26, 2024 •
edited
Loading

Not sure why the CI failed, it could run locally:

convert_tokenizer ./Baichuan2-7B-Chat/pytorch/dldt/FP16/ --output ./Baichuan2-7B-Chat/pytorch/dldt/FP16/ --with-detokenizer --trust-remote-code
Loading Huggingface Tokenizer...
Converting Huggingface Tokenizer to OpenVINO...
Saved OpenVINO Tokenizer: Baichuan2-7B-Chat/pytorch/dldt/FP16/openvino_tokenizer.xml, Baichuan2-7B-Chat/pytorch/dldt/FP16/openvino_tokenizer.bin
Saved OpenVINO Detokenizer: Baichuan2-7B-Chat/pytorch/dldt/FP16/openvino_detokenizer.xml, Baichuan2-7B-Chat/pytorch/dldt/FP16/openvino_detokenizer.bin

Just noticed that previous build step failed, so the folder ./Baichuan2-7B-Chat/pytorch/dldt/FP16/ is not existed:

OpenVINO Tokenizer version is not compatible with OpenVINO version. Installed OpenVINO version: 2024.1.0,OpenVINO Tokenizers requires 2024.0.0. OpenVINO Tokenizers models will not be added during export.
[ INFO ] openvino runtime version: 2024.1.0-14645-e6dc0865128
A new version of the following files was downloaded from https://huggingface.co/baichuan-inc/Baichuan2-7B-Chat:
- configuration_baichuan.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
Traceback (most recent call last):
  File "./llm_bench/python/convert.py", line 1411, in <module>
    main()
  File "./llm_bench/python/convert.py", line 1408, in main
    converter(args)
  File "./llm_bench/python/convert.py", line 1215, in convert_baichaun
    model = AutoModelForCausalLM.from_pretrained(args.model_id, trust_remote_code=True, **model_kwargs)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py", line 548, in from_pretrained
    model_class = get_class_from_dynamic_module(
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 488, in get_class_from_dynamic_module
    final_module = get_cached_module_file(
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 352, in get_cached_module_file
    get_cached_module_file(
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 314, in get_cached_module_file
    modules_needed = check_imports(resolved_module_file)
  File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/transformers/dynamic_module_utils.py", line 180, in check_imports
    raise ImportError(
ImportError: This modeling file requires the following packages that were not found in your environment: bitsandbytes. Run `pip install bitsandbytes`

Need to find a way to solve above issue

@mengbingrock
Copy link
Contributor Author

Hello @pavel-esir , I've added pip install bitsandbytes step before running this model. Could we try CI again to see if this time it works? Thanks a lot!

@mengbingrock
Copy link
Contributor Author

I've checked the new error in CI, this is expected. We need to wait for the change to intel-optimum to be merged. #273 (comment)

@p-wysocki p-wysocki linked an issue Apr 3, 2024 that may be closed by this pull request
[Good First Issue]: Verify baichuan2-7b-chat with GenAI text_generation #273
Open
@ilya-lavrenov ilya-lavrenov self-assigned this Apr 5, 2024
@ilya-lavrenov ilya-lavrenov mentioned this pull request Apr 8, 2024
Merged
ilya-lavrenov added a commit that referenced this pull request Apr 9, 2024
@ilya-lavrenov @eaidova
Update optimum-intel for llm_bench (#350)
c21b149 
To catch up huggingface/optimum-intel#628
Required for #328

---------

Co-authored-by: Ekaterina Aidova <[email protected]>
ilya-lavrenov
ilya-lavrenov reviewed Apr 9, 2024
View reviewed changes
.github/workflows/causal_lm_cpp.yml
tokenized = tokenizer('69', return_tensors='pt')
for beam in transformers.AutoModelForCausalLM.from_pretrained('baichuan-inc/Baichuan2-7B-Chat',trust_remote_code=True).generate(**tokenized, num_beam_groups=3, num_beams=15, num_return_sequences=15, diversity_penalty=1.0, max_new_tokens=20, early_stopping=False, length_penalty=1.0, no_repeat_ngram_size=9**9, do_sample=False):
ref = tokenizer.decode(beam[tokenized['input_ids'].numel():], skip_special_tokens=True)
idx = predictions.find(ref)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now, we have error on this line:

RuntimeError: Missing ref='69696969696969696969' from predictions

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, same happened locally. I'm not sure if it is the problem. Could it due to the random nature of generating? I'm studying into it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image
maybe we need to address these warnings? let's try to fully override GenerationConfig for the model

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great suggestion, these warning could be the clue. I'll try to understand these parameters and make change.

@andrei-kochin
Copy link
Collaborator

@mengbingrock are you still working on this? if so could you please resolve conflicts with the latest master?

@mengbingrock
Copy link
Contributor Author

@mengbingrock are you still working on this? if so could you please resolve conflicts with the latest master?

I could continue working in this, first I'll resolve the conflicts with the latest master.

@Wovchena
Copy link
Collaborator

@mengbingrock any updates here?

@ilya-lavrenov ilya-lavrenov removed their assignment Aug 2, 2024
@mlukasze mlukasze closed this Sep 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ilya-lavrenov ilya-lavrenov ilya-lavrenov left review comments

@pavel-esir pavel-esir Awaiting requested review from pavel-esir

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Good First Issue]: Verify baichuan2-7b-chat with GenAI text_generation
5 participants
@mengbingrock @andrei-kochin @Wovchena @ilya-lavrenov @mlukasze