Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added custom model inference. #437

Open
wants to merge 34 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
5909d4a
Added first version of custom model.
JoelNiklaus Dec 11, 2024
a2d6b63
Merge branch 'main' into add-custom-model
JoelNiklaus Dec 11, 2024
2283c89
Merge branch 'main' into add-custom-model
JoelNiklaus Dec 11, 2024
9563fab
Merge branch 'main' into add-custom-model
JoelNiklaus Dec 12, 2024
319d482
Merge branch 'main' into add-custom-model
clefourrier Dec 12, 2024
464edfe
Merge branch 'main' into add-custom-model
clefourrier Dec 12, 2024
6096042
Moved custom model config.
JoelNiklaus Dec 12, 2024
a7e1fe5
Added warning.
JoelNiklaus Dec 12, 2024
24b8bd3
Added custom model example for google translate.
JoelNiklaus Dec 12, 2024
c177a8e
Added documentation for custom model config.
JoelNiklaus Dec 12, 2024
d712cdb
Added docs.
JoelNiklaus Dec 12, 2024
7553147
Merge branch 'main' into add-custom-model
JoelNiklaus Dec 12, 2024
b41949c
Fixed path error.
JoelNiklaus Dec 12, 2024
aaaadb0
Fixed doc error.
JoelNiklaus Dec 12, 2024
c85065f
Added requirements file for google translate.
JoelNiklaus Dec 12, 2024
f1103da
Moved model loading function to reduce merge conflicts with litellm i…
JoelNiklaus Dec 12, 2024
71f871e
Added diskcache and get source and target language from the task name.
JoelNiklaus Dec 12, 2024
d1af518
Fixed problem with removing languages in the context.
JoelNiklaus Dec 12, 2024
2511158
Added retry logic.
JoelNiklaus Dec 13, 2024
7d5f76d
Merge branch 'main' into add-custom-model
JoelNiklaus Dec 16, 2024
743a284
Update google-translate requirements.
JoelNiklaus Dec 16, 2024
1a37f71
Added another example for a custom model.
JoelNiklaus Dec 17, 2024
2f27645
Made local mt model example more general to support madlad400 as well.
JoelNiklaus Dec 17, 2024
a4d4fee
Merge branch 'main' into add-custom-model
JoelNiklaus Dec 17, 2024
bd08781
Merge branch 'main' into add-custom-model
clefourrier Dec 18, 2024
b7106e4
Make sure generation can happen on the GPU.
JoelNiklaus Dec 18, 2024
a7d176c
Fixed issue with src and tgt lang for seamless model.
JoelNiklaus Dec 19, 2024
f1ba65c
Added cleanup to free the GPU memory again.
JoelNiklaus Dec 19, 2024
ace6e59
Fix dependency issues by switching to deep-translator.
JoelNiklaus Dec 22, 2024
cfd7254
Made inference code more robust against empty responses.
JoelNiklaus Dec 22, 2024
3ddc104
Merge branch 'main' into add-custom-model
JoelNiklaus Dec 23, 2024
f6df2a3
Merge branch 'main' into add-custom-model
clefourrier Jan 2, 2025
348e427
Merge branch 'main' into add-custom-model
JoelNiklaus Jan 7, 2025
a63f4b3
Merge branch 'main' into add-custom-model
JoelNiklaus Jan 11, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Moved model loading function to reduce merge conflicts with litellm i…
…nference.
JoelNiklaus committed Dec 12, 2024
commit f1103da9f425dfdd80835823a1ff5d763425929d
18 changes: 9 additions & 9 deletions src/lighteval/models/model_loader.py
Original file line number Diff line number Diff line change
@@ -113,6 +113,15 @@ def load_model_with_tgi(config: TGIModelConfig):
return model


def load_openai_model(config: OpenAIModelConfig, env_config: EnvConfig):
if not is_openai_available():
raise ImportError()

model = OpenAIClient(config, env_config)

return model


def load_custom_model(config: CustomModelConfig, env_config: EnvConfig):
logger.warning(f"Executing custom model code loaded from {config.model_definition_file_path}.")

@@ -142,15 +151,6 @@ def load_custom_model(config: CustomModelConfig, env_config: EnvConfig):
return model


def load_openai_model(config: OpenAIModelConfig, env_config: EnvConfig):
if not is_openai_available():
raise ImportError()

model = OpenAIClient(config, env_config)

return model


def load_model_with_inference_endpoints(config: InferenceEndpointModelConfig, env_config: EnvConfig):
logger.info("Spin up model using inference endpoint.")
model = InferenceEndpointModel(config=config, env_config=env_config)