feat: CompletionProvider API #379

FFFiend · 2023-07-12T02:09:38Z

What kind of change does this PR introduce?
Added a rough sketch of what the infrastructure surrounding the models we use might look like. Addresses #252 , #69 and #368 and may also be tangentially/directly relevant to other issues

Checklist

My code follows the style guidelines of OpenAdapt
I have performed a self-review of my code
If applicable, I have added tests to prove my fix is functional/effective
I have linted my code locally prior to submission
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation (e.g. README.md, requirements.txt)
New and existing unit tests pass locally with my changes

How can your code be run and tested?

Other information

0dm · 2023-07-14T20:13:32Z

openadapt/ml_models/ml_models.py

+PRIORITY = ["CPU", "GPU", "HOSTED"]
+from loguru import logger


Suggested change

PRIORITY = ["CPU", "GPU", "HOSTED"]

from loguru import logger

from loguru import logger

PRIORITY = ["CPU", "GPU", "HOSTED"]

abrichr · 2023-07-18T00:18:32Z

Thanks for getting started on this @FFFiend !

What do you think about:

class Modality:
    TEXT = "TEXT"
    IMAGE = "IMAGE"

MODALITY_BY_DB_MODEL = {
    models.ActionEvent: Modality.TEXT,
    models.Screenshot: Modality.IMAGE,
    models.WindowEvent: Modality.TEXT,
}

ML_MODELS = [
    {
        "name": "gpt-4",
        "provider": openai.ml.openai,
        # "tune_func": openadapt.ml.openai.gpt4.tune,
        # "infer_func": openadapt.ml.openai.gpt4.infer,
        "modalities": openadapt.ml.openai.gpt4.MODALITIES,  #  gpt4/__init__.py: MODALITIES =  [Modality.TEXT],
    },
     {
        "name": "BLIP2",
        "provider": openai.ml.huggingface,
        # "tune_func": openadapt.ml.huggingface.blip2.tune,
        # "infer_func": openadapt.ml.huggingface.blip2.infer,
        "modalities": openadapt.ml.huggingface.blip2.MODALITIES,  #  huggingface/__init__.py: MODALITIES =  [Modality.TEXT, Modality.IMAGE],
    },
    ...
]

    
def select_ml_model(recording_ids):
    recording_data = get_data_by_modality(recording_ids)
    for ml_model in ML_MODELS:
        # evaluate the performance of the given models autoregressively on an entire recording
        # (optionally with start_timestamp/end_timestamp parameters for evaluating on a section of a recording)

abrichr · 2023-07-18T01:10:02Z

@FFFiend please see:

https://www.youtube.com/watch?v=DxInMGIvkp0

See also: https://mldsai.slack.com/archives/C050869D3K4/p1689640271500029

https://twitter.com/Yampeleg/status/1668337702440165376

FFFiend · 2023-08-30T17:52:47Z

@FFFiend it looks like this PR includes several unrelated changes (e.g. alembic/versions/5139d7df38f6_add_recording_task_description.py).

Can you please remove all unrelated code? 🙏

@abrichr removed all unrelated code, however privacy provider tests (fixed in #488 ) are failing, so just waiting on that to get merged 👍

KrishPatel13 · 2023-08-30T18:59:02Z

@FFFiend #488 is merged now. Please update the branch then continue :-)

FFFiend · 2023-08-30T19:18:01Z

@abrichr Ready to merge.

abrichr · 2023-08-30T21:16:05Z

openadapt/ml/base_provider.py

+class CompletionProviderFactory:
+ """CompletionProviderFactory class."""
+
+ def get_for_modality(modality: Modality) -> list[CompletionProvider]:


Please add @staticmethod

abrichr · 2023-08-30T21:16:33Z

openadapt/ml/exceptions.py

+ """To be raised when a model is not supported by the infrastructure."""
+
+ def __init__(self, model_name: str) -> None:
+ """Init method for the ModelNotImplementedError class."""


Please remove

abrichr · 2023-08-30T21:17:47Z

openadapt/ml/exceptions.py

+ super().__init__(f"{model_name} is not currently supported")
+
+
+class GpuNotAvailableError(BaseException):


Please rename to GPUNotAvailableError

abrichr · 2023-08-30T21:18:15Z

openadapt/ml/huggingface_models/__init__.py

@@ -0,0 +1 @@
+"""HuggingFace ml package."""


Can we please name the submodule openadapt.ml.huggingface?

abrichr · 2023-08-30T21:25:29Z

openadapt/ml/huggingface_models/generic_provider.py

+ Capability.INFERENCE,
+ ]
+ Modalities: list[Modality] = [Modality.TEXT]
+ Availabilities: list[Availability] = [Availability.HOSTED]


Should this be Availabilities: list[Availability] = [Availability.LOCAL_CPU, Availability.LOCAL_GPU]?

We would need a separate provider for HOSTED models.

Can you please rename this LocalHuggingFaceProvider, and add a TODO above the class to implement one or more HostedHuggingFaceProvider to support Inference API and Inference Endpoints at https://huggingface.co/docs/huggingface_hub/guides/inference?

abrichr · 2023-08-30T21:27:02Z

openadapt/ml/huggingface_models/generic_provider.py

+ prompt: str,
+ model_path: str,
+ task_description: str,
+ use_pipeline: bool = True,


Please remove if unused 🙏

abrichr · 2023-08-30T21:28:44Z

openadapt/ml/huggingface_models/generic_provider.py

+ self,
+ prompt: str,
+ model_path: str,
+ task_description: str,


Please rename to huggingface_task to differentiate between https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/models.py#L46

abrichr · 2023-08-30T21:30:27Z

openadapt/ml/open_ai/gpt/gptprovider.py

+ def finetune(self, prompt_completion_pair: list[dict[str]]) -> None:
+ """Fine-tune the GPT model."""
+ # waiting on FineTuning PR which is currently
+ # a work in progress.


Please modify this comment to say:

TODO: implement once fine tuning is merged

abrichr · 2023-08-30T21:31:40Z

openadapt/ml/open_ai/gpt/gptprovider.py

+ valid_inference_models = openai.Model.list()["data"]
+ for model_dict in valid_inference_models:
+ if model_dict["id"] == gpt_model_name:
+ if gpt_model_name == "gpt-4" or gpt_model_name == "gpt-3.5-turbo":


if gpt_model_name in CHAT_MODEL_NAMES:

Ideally we would not have to define CHAT_MODEL_NAMES from scratch but load it from the openai library.

abrichr · 2023-08-30T21:32:38Z

openadapt/ml_utils/modal.py

@@ -0,0 +1,47 @@
+"""Conda Image creation tool."""


Can you please move this to openadapt/ml/utils/modal.py?

abrichr · 2023-08-30T21:33:18Z

tests/openadapt/ml/test_ml.py

+from openadapt.ml.huggingface_models.generic_provider import GenericHuggingFaceProvider
+from openadapt.ml.open_ai.gpt.gptprovider import GPTCompletionProvider
+
+test_gpt_provider = GPTCompletionProvider()


Can you please instantiate these inside of a @pytest.fixture?

FFFiend · 2023-08-31T20:18:09Z

openadapt/ml/base_provider.py

+ @staticmethod
+ def get_for_modality(modality: Modality) -> list[CompletionProvider]:
+ """Gets all available CompletionProviders with the given modality."""
+ completion_providers = CompletionProvider.get_children()


@abrichr On adding self to the CompletionProvider class, one would have to pass in parameters to the CompletionProvider that adhere to its attributes types here inside the factory (else it throws the following error):

E pydantic.error_wrappers.ValidationError: 4 validation errors for CompletionProvider E Name E field required (type=value_error.missing) E Capabilities E field required (type=value_error.missing) E Modalities E field required (type=value_error.missing) E Availabilities E field required (type=value_error.missing) pydantic/main.py:341: ValidationError

so we would be generating a CompletionProvider object with specific parameters. Not sure if that's adheres to: https://www.digitalocean.com/community/tutorials/factory-design-pattern-in-java

abrichr · 2023-12-13T06:43:35Z

This is now stale and needs to be updated to support images.

Related: #527

@FFFiend do you have any bandwidth? 🙏

abrichr · 2023-12-14T04:48:09Z

@FFFiend would love your thoughts on #534 🙏 😄

FFFiend and others added 20 commits June 23, 2023 01:24

added layout replaystrat mixin (partially complete)

e4b7490

fixed return value

3ea5cf1

removed extra space

ed74db9

removed unintended methods

b9453c8

reordered imports

0d3000d

ordered imports

009dd8d

changed type contract of path list to str

f8f2c90

Merge branch 'MLDSAI:main' into main

4e59ccc

fixed formatting

d17e35c

Merge branch 'MLDSAI:main' into main

bdd2ad5

Merge branch 'OpenAdaptAI:main' into main

7903161

added test images and complete test suite.

7b9c9b2

added failure case handling for unsupported doc types (gui and such)

ff85ce9

fixed file paths

775c3d9

Merge branch 'OpenAdaptAI:main' into main

f4ff09b

removed layoutlm file

a1c56c8

Merge branch 'OpenAdaptAI:main' into main

6de8c69

Merge branch 'OpenAdaptAI:main' into main

dc18646

created folders, started on model api file

836c0b0

added init file

8eaa432

OpenAdaptAI deleted a comment from cr-gpt bot Jul 12, 2023

FFFiend added 2 commits July 11, 2023 22:17

added check model availability function

7465107

changed name of folder to ml_models

b6ef7d6

OpenAdaptAI deleted a comment from cr-gpt bot Jul 12, 2023

FFFiend added 2 commits July 12, 2023 14:56

got rid of layout files

fd1af9f

changed to correct syntax

96373f1

0dm reviewed Jul 14, 2023

View reviewed changes

FFFiend and others added 4 commits August 30, 2023 19:31

Merge branch 'OpenAdaptAI:main' into models_branch

91aa924

fixed pytest in alembic

d578480

tesT_events fix

0bf1556

changed folder name

4861d07

FFFiend and others added 3 commits August 30, 2023 14:30

added openai api key

1230073

added skip on openai test if no key found

738d0b4

Merge branch 'OpenAdaptAI:main' into models_branch

aa0daa7

Merge branch 'OpenAdaptAI:main' into models_branch

1002f34

FFFiend requested a review from abrichr August 30, 2023 19:10

abrichr reviewed Aug 30, 2023

View reviewed changes

added changes

64f204e

FFFiend commented Aug 31, 2023

View reviewed changes

FFFiend requested a review from abrichr August 31, 2023 20:26

removed exception

b008fc9

LaPetiteSouris mentioned this pull request Sep 14, 2023

feat: UI structured representation #495

Draft

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: CompletionProvider API #379

feat: CompletionProvider API #379

FFFiend commented Jul 12, 2023 •

edited

0dm Jul 14, 2023

abrichr commented Jul 18, 2023 •

edited

abrichr commented Jul 18, 2023 •

edited

FFFiend commented Aug 30, 2023 •

edited

KrishPatel13 commented Aug 30, 2023

FFFiend commented Aug 30, 2023

abrichr Aug 30, 2023

abrichr Aug 30, 2023

abrichr Aug 30, 2023

abrichr Aug 30, 2023

abrichr Aug 30, 2023

abrichr Aug 30, 2023 •

edited

abrichr Aug 30, 2023

abrichr Aug 30, 2023

abrichr Aug 30, 2023 •

edited

abrichr Aug 30, 2023

abrichr Aug 30, 2023

FFFiend Aug 31, 2023 •

edited

abrichr commented Dec 13, 2023

abrichr commented Dec 14, 2023

		PRIORITY = ["CPU", "GPU", "HOSTED"]
		from loguru import logger

		super().__init__(f"{model_name} is not currently supported")


		class GpuNotAvailableError(BaseException):

feat: CompletionProvider API #379

Are you sure you want to change the base?

feat: CompletionProvider API #379

Conversation

FFFiend commented Jul 12, 2023 • edited

Choose a reason for hiding this comment

abrichr commented Jul 18, 2023 • edited

abrichr commented Jul 18, 2023 • edited

FFFiend commented Aug 30, 2023 • edited

KrishPatel13 commented Aug 30, 2023

FFFiend commented Aug 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abrichr Aug 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abrichr Aug 30, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FFFiend Aug 31, 2023 • edited

Choose a reason for hiding this comment

abrichr commented Dec 13, 2023

abrichr commented Dec 14, 2023

FFFiend commented Jul 12, 2023 •

edited

abrichr commented Jul 18, 2023 •

edited

abrichr commented Jul 18, 2023 •

edited

FFFiend commented Aug 30, 2023 •

edited

abrichr Aug 30, 2023 •

edited

abrichr Aug 30, 2023 •

edited

FFFiend Aug 31, 2023 •

edited