How could I set the stop sequence for inference like in CodeLlama 70B? #666
Replies: 1 comment 6 replies
-
|
This is more of an application implementation issue than an mlx issue. I have implemented this in one of my projects. You can take a look here. |
Beta Was this translation helpful? Give feedback.
-
|
You may need to double-check if you are using the base model for code completion instead of using the chat model. The chat model is not normally trained for auto-complete tasks. |
Beta Was this translation helpful? Give feedback.
-
|
I'm using model from here: https://huggingface.co/mlx-community/CodeLlama-70b-Instruct-hf-4bit-MLX Thanks for the mlx-llm-server, I can run curl to get response from api as the README in github project. I tested both the following 3 projects, none of them could work properly with mlx-llm-server exposed OpenAI compatible API. Do you use a webui front-end to work with mlx-llm-server? Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
There's a issue with llama tokenizer, may cause the problem, but I have test the integration with chat-ui using another models it was working fine. I am currently working on add openai like api support into mlx-lm, hopefully it will get merged soon. So you can use that instead. FYI |
Beta Was this translation helpful? Give feedback.
-
|
That's cool. I couldn't wait to try the merged version with openai like api support. I'll try chat-ui. |
Beta Was this translation helpful? Give feedback.
-
|
I've tried chat-ui, The mlx-llm-server exposed API works with huggingface chat-ui. Great job, thanks mzbac! |
Beta Was this translation helpful? Give feedback.
-
When running inference with CodeLlama 70B, I need to specify the stop sequence in llama.cpp or in ollama.
When I run CodeLlama 70B 4bit MLX, it outputs lots of EOT and could not stop. I am not sure if it is caused by stop sequences settings. How could I set the stop sequence in MLX?
Beta Was this translation helpful? Give feedback.
All reactions