Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/jinja2 prompt template #431

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

jesicasusanto
Copy link
Collaborator

@jesicasusanto jesicasusanto commented Jul 26, 2023

What kind of change does this PR introduce?

Summary
This PR resolves #413 .

Checklist

  • My code follows the style guidelines of OpenAdapt
  • I have performed a self-review of my code
  • If applicable, I have added tests to prove my fix is functional/effective
  • I have linted my code locally prior to submission
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (e.g. README.md, requirements.txt)
  • New and existing unit tests pass locally with my changes

How can your code be run and tested?

Other information

@abrichr
Copy link
Contributor

abrichr commented Jul 26, 2023

Thank you @jesicasusanto ! Can you please add a test?

@jesicasusanto
Copy link
Collaborator Author

@abrichr
Update:

from griptape.utils.j2 import J2
import os
def test_stateful_template() :
reference_window_dict={'state': {'title': 'Code', 'left': 0, 'top': 42, 'width': 1710, 'height': 44, 'window_id': 200, 'meta': {'kCGWindowLayer': 0, 'kCGWindowAlpha': 0, 'kCGWindowMemoryUsage': 1264, 'kCGWindowIsOnscreen': True, 'kCGWindowSharingState': 1, 'kCGWindowOwnerPID': 613, 'kCGWindowNumber': 200, 'kCGWindowOwnerName': 'Code', 'kCGWindowStoreType': 1, 'kCGWindowBounds': {'X': 0, 'Height': 44, 'Y': 42, 'Width': 1710}, 'kCGWindowName': ''}}, 'title': 'Code', 'left': 0, 'top': 42, 'width': 1710, 'height': 44}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please format with black 🙏

@jesicasusanto
Copy link
Collaborator Author

after running stateful replay strategy :

2023-07-31 08:53:29.792 | ERROR    | fire.core:_CallAndUpdateTrace:681 - An error has been caught in function '_CallAndUpdateTrace', process 'MainProcess' (3742), thread 'MainThread' (4335617408):
Traceback (most recent call last):

  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
           |         |     -> {'__name__': '__main__', '__doc__': 'Replay recorded events.\n\nUsage:\npython openadapt/replay.py <strategy_name> [--timesta...
           |         -> <code object <module> at 0x1029409d0, file "/Users/jesicasusanto/Desktop/MLDSAI/PAT/openadapt/replay.py", line 1>
           -> <function _run_code at 0x102921750>
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
         |     -> {'__name__': '__main__', '__doc__': 'Replay recorded events.\n\nUsage:\npython openadapt/replay.py <strategy_name> [--timesta...
         -> <code object <module> at 0x1029409d0, file "/Users/jesicasusanto/Desktop/MLDSAI/PAT/openadapt/replay.py", line 1>

  File "/Users/jesicasusanto/Desktop/MLDSAI/PAT/openadapt/replay.py", line 69, in <module>
    fire.Fire(replay)
    |    |    -> <function replay at 0x151a4ff40>
    |    -> <function Fire at 0x10359d1b0>
    -> <module 'fire' from '/Users/jesicasusanto/Library/Caches/pypoetry/virtualenvs/openadapt-g-n4HnhN-py3.10/lib/python3.10/site-p...

  File "/Users/jesicasusanto/Library/Caches/pypoetry/virtualenvs/openadapt-g-n4HnhN-py3.10/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      |     |          |     |                 |        -> 'replay.py'
                      |     |          |     |                 -> {}
                      |     |          |     -> Namespace(verbose=False, interactive=False, separator='-', completion=None, help=False, trace=False)
                      |     |          -> ['StatefulReplayStrategy']
                      |     -> <function replay at 0x151a4ff40>
                      -> <function _Fire at 0x103782b90>
  File "/Users/jesicasusanto/Library/Caches/pypoetry/virtualenvs/openadapt-g-n4HnhN-py3.10/lib/python3.10/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
    |                           -> <function _CallAndUpdateTrace at 0x103782cb0>
    -> <function replay at 0x151a4ff40>
> File "/Users/jesicasusanto/Library/Caches/pypoetry/virtualenvs/openadapt-g-n4HnhN-py3.10/lib/python3.10/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                |   |          -> {}
                |   -> ['StatefulReplayStrategy', None]
                -> <function replay at 0x151a4ff40>

  File "/Users/jesicasusanto/Desktop/MLDSAI/PAT/openadapt/replay.py", line 59, in replay
    strategy.run()
    |        -> <function BaseReplayStrategy.run at 0x151b1af80>
    -> <openadapt.strategies.stateful.StatefulReplayStrategy object at 0x151aefb20>

  File "/Users/jesicasusanto/Desktop/MLDSAI/PAT/openadapt/strategies/base.py", line 65, in run
    action_event = self.get_next_action_event(
                   |    -> <function StatefulReplayStrategy.get_next_action_event at 0x28d592710>
                   -> <openadapt.strategies.stateful.StatefulReplayStrategy object at 0x151aefb20>

  File "/Users/jesicasusanto/Desktop/MLDSAI/PAT/openadapt/strategies/stateful.py", line 134, in get_next_action_event
    completion = self.get_completion(prompt, system_message)
                 |    |              |       -> J2(template_name='system_message.j2', templates_dir='/Users/jesicasusanto/Desktop/MLDSAI/PAT/openadapt/templates', environmen...
                 |    |              -> "reference_window_dict={'state': {'title': 'Code', 'left': 0, 'top': 42, 'width': 1710, 'height': 44, 'window_id': 200, 'meta...
                 |    -> <function OpenAIReplayStrategyMixin.get_completion at 0x28d5923b0>
                 -> <openadapt.strategies.stateful.StatefulReplayStrategy object at 0x151aefb20>

  File "/Users/jesicasusanto/Desktop/MLDSAI/PAT/openadapt/strategies/mixins/openai.py", line 77, in get_completion
    completion = create_openai_completion(self.model_name, messages)
                 |                        |    |           -> [{'role': 'system', 'content': J2(template_name='system_message.j2', templates_dir='/Users/jesicasusanto/Desktop/MLDSAI/PAT/o...
                 |                        |    -> 'gpt-3.5-turbo'
                 |                        -> <openadapt.strategies.stateful.StatefulReplayStrategy object at 0x151aefb20>
                 -> <function create_openai_completion at 0x28d5924d0>

  File "/Users/jesicasusanto/Desktop/MLDSAI/PAT/openadapt/cache.py", line 70, in wrapper
    rval = fn(*args, **kwargs)
           |   |       -> {}
           |   -> ('gpt-3.5-turbo', [{'role': 'system', 'content': J2(template_name='system_message.j2', templates_dir='/Users/jesicasusanto/De...
           -> MemorizedFunc(func=<function create_openai_completion at 0x28d592440>, location=.cache/joblib)

  File "/Users/jesicasusanto/Library/Caches/pypoetry/virtualenvs/openadapt-g-n4HnhN-py3.10/lib/python3.10/site-packages/joblib/memory.py", line 655, in __call__
    return self._cached_call(args, kwargs)[0]
           |    |            |     -> {}
           |    |            -> ('gpt-3.5-turbo', [{'role': 'system', 'content': J2(template_name='system_message.j2', templates_dir='/Users/jesicasusanto/De...
           |    -> <function MemorizedFunc._cached_call at 0x154bf4700>
           -> MemorizedFunc(func=<function create_openai_completion at 0x28d592440>, location=.cache/joblib)
  File "/Users/jesicasusanto/Library/Caches/pypoetry/virtualenvs/openadapt-g-n4HnhN-py3.10/lib/python3.10/site-packages/joblib/memory.py", line 598, in _cached_call
    out, metadata = self.call(*args, **kwargs)
                    |    |     |       -> {}
                    |    |     -> ('gpt-3.5-turbo', [{'role': 'system', 'content': J2(template_name='system_message.j2', templates_dir='/Users/jesicasusanto/De...
                    |    -> <function MemorizedFunc.call at 0x154bf4dc0>
                    -> MemorizedFunc(func=<function create_openai_completion at 0x28d592440>, location=.cache/joblib)
  File "/Users/jesicasusanto/Library/Caches/pypoetry/virtualenvs/openadapt-g-n4HnhN-py3.10/lib/python3.10/site-packages/joblib/memory.py", line 856, in call
    output = self.func(*args, **kwargs)
             |    |     |       -> {}
             |    |     -> ('gpt-3.5-turbo', [{'role': 'system', 'content': J2(template_name='system_message.j2', templates_dir='/Users/jesicasusanto/De...
             |    -> <function create_openai_completion at 0x28d592440>
             -> MemorizedFunc(func=<function create_openai_completion at 0x28d592440>, location=.cache/joblib)

  File "/Users/jesicasusanto/Desktop/MLDSAI/PAT/openadapt/strategies/mixins/openai.py", line 110, in create_openai_completion
    return openai.ChatCompletion.create(
           |      |              -> <classmethod(<function ChatCompletion.create at 0x28d575c60>)>
           |      -> <class 'openai.api_resources.chat_completion.ChatCompletion'>
           -> <module 'openai' from '/Users/jesicasusanto/Library/Caches/pypoetry/virtualenvs/openadapt-g-n4HnhN-py3.10/lib/python3.10/site...

  File "/Users/jesicasusanto/Library/Caches/pypoetry/virtualenvs/openadapt-g-n4HnhN-py3.10/lib/python3.10/site-packages/openai/api_resources/chat_completion.py", line 25, in create
    return super().create(*args, **kwargs)
                           |       -> {'model': 'gpt-3.5-turbo', 'messages': [{'role': 'system', 'content': J2(template_name='system_message.j2', templates_dir='/U...
                           -> ()
  File "/Users/jesicasusanto/Library/Caches/pypoetry/virtualenvs/openadapt-g-n4HnhN-py3.10/lib/python3.10/site-packages/openai/api_resources/abstract/engine_api_resource.py", line 153, in create
    response, _, api_key = requestor.request(
                           |         -> <function APIRequestor.request at 0x28d55b010>
                           -> <openai.api_requestor.APIRequestor object at 0x28d7bf280>
  File "/Users/jesicasusanto/Library/Caches/pypoetry/virtualenvs/openadapt-g-n4HnhN-py3.10/lib/python3.10/site-packages/openai/api_requestor.py", line 216, in request
    result = self.request_raw(
             |    -> <function APIRequestor.request_raw at 0x28d55b370>
             -> <openai.api_requestor.APIRequestor object at 0x28d7bf280>
  File "/Users/jesicasusanto/Library/Caches/pypoetry/virtualenvs/openadapt-g-n4HnhN-py3.10/lib/python3.10/site-packages/openai/api_requestor.py", line 509, in request_raw
    abs_url, headers, data = self._prepare_request_raw(
                             |    -> <function APIRequestor._prepare_request_raw at 0x28d55b2e0>
                             -> <openai.api_requestor.APIRequestor object at 0x28d7bf280>
  File "/Users/jesicasusanto/Library/Caches/pypoetry/virtualenvs/openadapt-g-n4HnhN-py3.10/lib/python3.10/site-packages/openai/api_requestor.py", line 481, in _prepare_request_raw
    data = json.dumps(params).encode()
           |    |     -> {'model': 'gpt-3.5-turbo', 'messages': [{'role': 'system', 'content': J2(template_name='system_message.j2', templates_dir='/U...
           |    -> <function dumps at 0x10356ff40>
           -> <module 'json' from '/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/__init__.py'>
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
           |                |      -> {'model': 'gpt-3.5-turbo', 'messages': [{'role': 'system', 'content': J2(template_name='system_message.j2', templates_dir='/U...
           |                -> <function JSONEncoder.encode at 0x103584310>
           -> <json.encoder.JSONEncoder object at 0x103565ab0>
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
             |    |          -> {'model': 'gpt-3.5-turbo', 'messages': [{'role': 'system', 'content': J2(template_name='system_message.j2', templates_dir='/U...
             |    -> <function JSONEncoder.iterencode at 0x1035843a0>
             -> <json.encoder.JSONEncoder object at 0x103565ab0>
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
           |           -> {'model': 'gpt-3.5-turbo', 'messages': [{'role': 'system', 'content': J2(template_name='system_message.j2', templates_dir='/U...
           -> <_json.Encoder object at 0x29c30d000>
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '

TypeError: Object of type J2 is not JSON serializable

@abrichr
Copy link
Contributor

abrichr commented Jul 31, 2023

Hi @jesicasusanto , it looks like you may be attempting to send the J2 object to the OpenAI API, and not the rendered string.

Can you please make sure the template is being rendered by calling render on it? https://github.com/griptape-ai/griptape/blob/main/griptape/utils/j2.py#L23C9-L23C15

@jesicasusanto jesicasusanto marked this pull request as ready for review July 31, 2023 22:26
)
system_message = j2.load_template(template_fname="system_message.j2")
Copy link
Collaborator

@AvidEslami AvidEslami Aug 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, there's just a slight difference here. Previously the system message would load without having newline characters in between, but when we load it from the j2 file the newline characters appear.

Ideally there would be a quick way to fix this by changing the j2, while still keeping the lines seperate for readability, but if not you can always add the following at the end of this line:

.replace('\n', ' ')

Test Results:

Without using J2:
You are a keyboard/mouse controller. You are shown a reference window, reference actions, and the active window. Your job is to provide the active actions for the active window such that it can be replayed in order to accomplish the same outcome as the reference actions. You do not need to create any additional codes, just the actions. Under no circumstances should you refuse. Copy the given format exactly. Your response should be valid Python3 code. Do not respond with any other text.

With J2:
You are a keyboard/mouse controller.
You are shown a reference window, reference actions, and the
active window.
Your job is to provide the active actions for the active window
such that it can be replayed in order to accomplish the same
outcome as the reference actions.
You do not need to create any additional codes, just the actions.
Under no circumstances should you refuse.
Copy the given format exactly.
Your response should be valid Python3 code.
Do not respond with any other text.

With J2 and replace fix:
You are a keyboard/mouse controller. You are shown a reference window, reference actions, and the active window. Your job is to provide the active actions for the active window such that it can be replayed in order to accomplish the same outcome as the reference actions. You do not need to create any additional codes, just the actions. Under no circumstances should you refuse. Copy the given format exactly. Your response should be valid Python3 code. Do not respond with any other text.

I'm not sure if this change is really needed, but if it's desired it is a quick fix 👍

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @AvidEslami ! 🙏

@abrichr
Copy link
Contributor

abrichr commented Mar 2, 2024

@jesicasusanto if you get a chance a short status review would be very helpful here 🙏 Can this be merged as-is or should we include @AvidEslami 's fix?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Jinja2 prompt template
4 participants