feat: UI structured representation #495
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce?
Introduce
DOMElement
class, which will represent fully a screen state regardless of OS type. ThisDOMElement
is the atomic unit to represent any UI and serves as the lowest level of representation. It is the adapter which "translate" the window state into an uniform representational formatSummary
Refer to RFC on UI Representation, the job of prediction of a single action to take on a single screen can be translated into the following steps:
Backward wise, the evaluation process used in model training/tuning/RFLHF can be decomposed into several steps.
Given a window state and action taken as reference pair, then a current window state a predicted action as inference pair:
Represent the reference window state in UI Tree
This
DOMElement
is the low level mapping of everything into a single unified UI, which can later be translated into prompt language, regardless of UI state or OS type.Checklist
How can your code be run and tested?
Other information
Next step:
griptape AI is a very good candidate for this
With implementation of CompletionProvider, in general
CompletionProvider
should take the generic prompt generated or the pipeline generated to run based on specific provider.UITranslator
class, which takesDOMElement
and translate the element into operational prompt to ask. This is the "secret glue" which translates the representational of the UI to LLM language.NOTE The main effort is to uniform, standardize the way UI is interacted with first. Then to uniform and standardize the way the models are interacted with. I.e: to translate and represent the UI into LLMs' language.